Evaluators: Whose Needle Are You Trying to Move?

Foundation staff and boards often look to answer a deceptively simple question, “Are we moving the needle?” But there’s a bigger question at stake: whose needle are we talking about?

I recently attended the American Evaluation Association’s Evaluation 2018 Conference: Speaking Truth to Power. I was inspired by presentations that explored the inherent power-dynamics between foundations and nonprofits and those who went even further — delving into how these dynamics impact communities and social movements. This led me to wonder, is it enough to use best practices, community data, and nonprofit feedback to accomplish the work? Are we really engaging the right stakeholders? Ultimately, are we even trying to move the right needles?

The Equitable Evaluation Initiative’s website (“Why EE,” n.d.) states that the current paradigm of evaluation, “looks for generalizable and scaled data and findings that often feel disconnected and not reflective of the values of the nonprofit/community partner.” As one example, how often have we reported back to clients or community members about the data we ask nonprofits to gather? How often do we ask their perceptions about our findings? I, like many in the sector, started working to make conditions so that everyone can set and achieve their goals on a level field. Evaluation helps us see how close we are to achieving those conditions. But, are we even looking at the right conditions?

When we look for changes in community, we typically track changes in behaviors and conditions (e.g. increased household income, increased test scores, increased graduation rates, increased percentage of people with health insurance, etc.). These changes cannot occur without changes in beliefs and attitudes from those we engage — community members.

When I say community members, I am not only talking about those individuals that lead nonprofit and for-profit organizations. Nor am I talking only about government representatives, or about your board members. I am talking about all community members. The “right” stakeholders are the people that want to make or sustain change in their community. I would suggest that while there are many factors that go into influencing change, the majority of our efforts should focus on what community members want to achieve, and how they want to go about driving that change.

In Community Engagement Matters (Now More Than Ever), Melody Barnes and Paul Schmitz (2016) warn practitioners against data-driven and evidence-based approaches stating, “…leaders must avoid the temptation to act in a top-down manner. Instead, they should design and implement programs in ways that engage community members directly in the work of social change.” (And don’t miss Paul’s recent #FieldFocus blog on how the dynamics of power are changing in our field!)

I would take this a step further and say that you must engage community members around the evaluation process to not only make social change, but also to make sure you are measuring what is truly important about that change. Ask them what they think the problems in their communities are, and which ones they would like to see you work on.

How can foundations and evaluators engage community? Evaluation consists of four main phases: planning, implementation, analysis, and reporting.

Planning: When planning a grantmaking evaluation, evaluators will often work with foundation staff to plan strategies based on desired outcomes. This is typically what evaluators use to define the evaluation questions. Why not engage community to define the intended outcomes and strategies they think would improve the outcomes they want to change? What do community members want to know more about?

Implementation: During the implementation of the evaluation, often foundations hire evaluators to collect, record, and continually analyze the data. This is a great time to engage community. Can the evaluators train youth to collect some data? Do community members want to see and have access to the aggregated data? Who owns the data (this could be its own blog post)? What data do community members want to know more about?

Analysis: Evaluators then take the data that they collected and analyze it to answer the evaluation questions. Community members can add context to this analysis. How can community members have access to and learn how to analyze data on their community? Can this be more accessible to community?

Reporting: What happens to the evaluation reports? Often these reports go to the funders and are not shared beyond the board and foundation. How can we change that? How can community members have access to the reports? Do community members want to give the reports to the foundation? How can this be used to continually guide the further work of the foundation?

These questions do not necessarily have simple answers, but they are well worth exploring. Therefore, my challenge/question to you is, what will you do next? Nonprofits, how will you advocate for the communities you serve around evaluation with your funders? Foundations, how do you engage with your community around evaluation? How do you think you can use evaluation to not only lift the voices of the community, but also to affect change?

Whose Needle Are You Trying to Move?

Why Do Foundations Struggle With Evaluation?

New Frameworks for Evaluating Impact

Data and the Sector: Using Data to Drive Effective Philanthropy

References