Evaluating Community Change It is difficult work!

Community Change Initiatives Arose as an ambitious strategy in the 1990s to address the needs of residents of poor communities Adopted a comprehensive approach to neighborhood change and worked according to community building principles that value resident engagement and community capacity building Community Change Initiatives

“Most community change initiatives and community change efforts have not produced the degree of community transformation envisioned by their designers.

“One area of progress in evaluating community change is in “more realistic expectations for measuring impact,” with a growing understanding that establishing causality is difficult.” Anne Kubisch Director of the Roundtable on Community Change The Aspen Institute

The Harlem Children’s Zone brought third-grade test scores up to the New York state average Population of West Philadelphia enjoys safety, decent housing, a good school, and commercial space 16 neighborhoods in Chicago have high-quality comprehensive plans Miner Country, S.D., has put into place renewable energy facilities that are revitalizing the local economy Success Stories

A CASE STUDY: The Neighborhood Improvement Initiative From 1996 to 2006, The William and Flora Hewlett Foundation gave $20 million to initiative designed to improve lives of residents in three San Francisco Bay area communities. Helped some people, but its impact did not reflect the investments of time and the $17.5 million — which leveraged more than $27.4 million in public and private resources — that were expended.

Hewlett did not develop a detailed theory of change that specified all the connections between each of NII’s six goals, the strategies to achieve the goals, and the assumptions underlying the connections.

"It reminds us what happens when a foundation and its partners fail to achieve clarity about goals, strategies, and how to measure progress. …  

 “When this happens, we should seize the opportunity to understand the causes in order to improve our own performance and benefit others working in the field.” Paul Brest, President The William and Flora Hewlett Foundation

Evaluations not intended to justify the existence of a project, i. e Evaluations not intended to justify the existence of a project, i.e., to prove that it works ✕

Evaluations serve as a source of important information to promote project development and growth

Useful CCI Evaluations: Inform decisions Clarify options Identify strengths and weaknesses Provide information on program improvements, policies, and key contextual factors affecting the program

COMMUNITY CHANGE INITIATIVES (CCIs) 5 RULES FOR EVALUATING COMMUNITY CHANGE INITIATIVES (CCIs)

Evaluations are not experiments but part of the community change process Controlling all the factors at play in a community is impossible - Too many contextual variables Must include constant assessment, adaptation of strategy, and accommodation of interventions to new and changing conditions Rigorous measurement tools and methods implemented throughout the life the program Not an afterthought to satisfy donor

Evaluations need a strong focus on the processes of community change Focusing solely on establishing causal relationships will not contribute meaningful and useful information Document the factors that influence change, not just what the change results in on the other side Causality relationship between an event (the cause) and a second event (the effect), where the second event is understood as a consequence of the first

Evaluations need to measure ongoing progress towards achieving outcomes and results Most successful when they have been able to define explicit results and outcomes that multiple agents and community constituencies can share Measure program or single agency-level outcomes and neighborhood-wide or population-level outcomes Not simply the regular measurement of performance but an instrument of accountability and feedback capable of motivating, informing, and reinforcing positive change

Need to understand, document, and explain the multiple theories of change at work over time Explicitly define upfront the theories of achieving broad community scale and reaching an improved and sustainable state of transformation Single theory insufficient. Use multiple theories from a wide range of disciplines

Prioritize real-time learning and the community’s capacity to understand and use data from evaluations Building self-evaluation, learning, and accountability capacity and infrastructure among and inside organizations, systems, and the community is a necessary task of the evaluation Community and evaluators need to build relationships and capacity to work together to make sure that effective community transformation is informed and reinforced by strong and relevant data collection, analysis, and reporting on process and outcomes

Deciding what to measure, how to measure it, how to collect the data, and how to interpret the results is a challenging process that involves many trade-offs of rigor, practicality, and resources.