1 Why Impact Evaluation Partially adapted from “Theory of Change in Program Evaluation”, J-PAL 2011

2 Why Impact Evaluation? “Micro-credit has been changing people's lives and revitalizing communities” UN, 2005, Year of micro-credit “More than 200 poor….killed themselves in late 2010, according to media reports compiled by the government of the south Indian state. The state blamed microfinance companies--which give small loans intended to lift up the very poor--for fueling a frenzy of over indebtedness and then pressuring borrowers so relentlessly that some took their own lives.” Wall Street Journal, February 2012

3 Why Impact Evaluations? Surprisingly little hard evidence on the social programs that actually work Decisions often based more on emotion than evidence Accountability (for the implementing agency) -Did we do what we said we were going to do? -Did we have a positive impact on people’s lives? Lesson learning (for the entire world) –Particular programs do or do not work –What is the most effective route to achieve a certain outcome? –Similarities in strategies that are successful, for example, in changing behavior, even across fields?

4 What makes a good evaluation? Answers an important question in an unbiased and definitive way To do that you need a model/theory of change (ToC) –Who is the target? –What are their needs? –What is the program seeking to change? –What is the precise program or part of program being evaluated?

What is Theory of Change Theory of Change defines all building blocks required to bring about a given long-term goal. This set of connected building blocks is represented as a pathway of change or as a roadmap Theory of Change describes the intervention(s) that bring about the outcomes depicted in the pathway of a change map. The pathway of change reveals the linkages of outcomes and the often complex web of activity that is required to bring about change. 5

Requires an articulation of assumptions Inclusion of stakeholders – staff, participants, constituents, funders Requires clarity on long-term goals and formulation of actions to achieve goals Requires measurable indicators of success and change What is Theory of Change (cont.)

Formulating a Theory of Change 1.Determine your long-term goals 2.“Backwards mapping” 3.Identify the holes 4.Filling the holes 5.Measure your outcomes and impact 7 Source:

Formulating a Theory of Change (ex.) 1.Determine your long-term goals –Think BIG! But be reasonable… 2.“Backwards mapping” –Connect the criteria necessary to achieve those goals 3.Identify the holes –Which steps have been met? Which bridges must be built? 4.Filling the holes –Does your program build the right bridges? 5.Measure your outcomes and impact –Determine your outcomes, develop indicators, collect data 8 Source:

Types of Change Examples of types of change: Skills Status, functioning Behavior Attitudes and perceptions Structures and processes 9

ComponentDescription InputsResources that go into a programme (funding, staffing, equipment, curriculum, materials etc. ActivitiesWhat we do as a part of the programme (deliver, teach, train, etc.) OutputsTangible products or services produced as a result of the activities - usually can be counted OutcomesBehavioural changes that result from the project outputs (preventive health habits, usage etc.) ImpactsLong-term changes that result from an accumulation of outcomes – goals of the programme Components of theory of change

11 Example: Provision of textbooks Needs Logical Framework Long- term Goal InputOutputOutcomeImpact Poor children in Busia district Kenya with low learning, low incomes, few books, can’t take books home, hard to learn. Higher Income

12 Example: Provision of textbooks Needs Logical Framework Long- term Goal InputOutputOutcomeImpact Poor children in Busia district Kenya with low learning, low incomes, few books, can’t take books home, hard to learn. NGO purchases books. Books delivered to and used by schools. Children use books and are able to study better. Higher test scores Higher Income

More complex models 13 Ex: delaying the age of first birth –Economic constraints –Lack of access to family planning –Family pressure / Group effects –Ignorance of the dangers for maternal / infant health What are adequate programs to address these issues? –Incentives – for whom? –Symbolic reward – to whom? –Information – for whom? Think about constraints, targets, spillovers Ill-designed programs may have no model at all

Steps in Evaluation Research Identifyin g a focus of Interest or Problem 2. Collectin g data 3. Analyzing data/gener ating hypothesis 4. Plannin g Action Steps 5. Impleme n-ting Action steps 6. Collecting data to monitor change 7. Analysis and Evaluatio n