Howard White International Initiative for Impact Evaluation (3ie)

An impact evaluation seeks to attribute all, or part, of the observed change in outcomes to a specific intervention.

LevelIndicators InputsResources: Funds and personnel ActivitiesTeacher training School improvements Decentralized management and local school management committees OutputsTrained teachers Better school facilities Functioning school management committees Intermediate outcomesHigher school enrolments at all levels Teacher and parent satisfaction Better managed schools Final outcomesImproved learning outcomes ImpactHigher productivity and earnings Empowerment

 Focus on final welfare outcomes, e.g.  Infant mortality  Income poverty  Security  Usually long-term, but not necessarily so (but then sustainability is an issue)

 Projects (or specific interventions)  Individual projects are the ‘back bone’ of impact analysis  But even then may only be able to do rigorous impact analysis of some components  Programmes  Sector wide programs can be conceived of as supporting a range of interventions, many of which can be subject to rigorous impact evaluation.  Policies  In general different approaches are required, such as CGEs – these are not being discussed today

Pick a named intervention for an impact evaluation and make a short list of indicators (using the log frame) for evaluation of this intervention

BeforeAfter Project (treatment)66 Control

But we don’t know if they were similar before… though there are ways of doing this BeforeAfter Project (treatment)66 Control55

Sometimes this can work … but usually not BeforeAfter Project (treatment)4066 Control

BeforeAfter Project (treatment)4066 Control4455

 Ex ante design preferred to ex post: impact evaluation design is much stronger if baseline data are available (but may still be possible even if they are not)  Means collecting data before intervention starts, and can be affecting the design of the intervention  But can sometimes use secondary data, that is an existing survey

1. Confounding factors 2. Selection effects 3. Spillovers and contagion 4. Impact heterogeneity 5. Ensuring policy relevance

 Other things happen – so before versus after rarely sufficient  So get a control group… but different things may happen there  So collect data on more than just outcome and impact indicators  And collect baseline data  But …

 Program placement and self-selection  Program beneficiaries have particular characteristics correlated with outcomes – so impact estimates are biased  Need to use experimental or quasi- experimental methods to cope with this; this is what has been meant by rigorous impact evaluation  But it is just one facet of impact evaluation design  Other things can also bias impact estimates

 Experimental (randomized):  Limited application, but there are applications and it is a powerful approach  Many concerns (e.g. budget and ethics) and not valid  Quasi-experimental design (regression based):  Propensity score matching is most common  Regression discontinuity  Interrupted time series  Regression modelling of outcomes

 Spillover – positive and negative impacts on non-beneficiaries  Contagion – similar interventions in control areas  Need to collect data on these aspects and may need to revise evaluation design

WHAT ARE THE MAJOR CONFOUNDING FACTORS FOR YOUR OUTCOME AND IMPACT INDICATORS? HOW MIGHT SELECTION BIAS, SPILLOVER AND CONTAGION AFFECT THE EVALUATION OF THE INTERVENTION YOU HAVE SELECTED ?

 Impact varies by intervention (design), beneficiary and context  ‘Averages’ can be misleading  Strong implications for evaluation design

 Is the impact of X and Y, bigger, equal to or less than the impacts of doing X and Y separately?  For example, hygiene promotion and sanitation facilities  Evidence suggestions they are substitutes- either one reduces incidence child diarrhea by 40-50%, but not by more if the two are combined

 Irreparable damage to physical and cognitive development results from nutritional deprivation in the first two years of life  Hence interventions to infants have greater long-run impact on many outcomes than do those aimed at older children (such as school feeding programs)

Expected impact Good yearLow Average-bad yearHigh Very bad yearNone

What sort of differences in impact would you expect for your intervention with respect to intervention (design), context and beneficiary?

 Process  Stakeholder engagement  Packaging messages  Design  Theory-based approach  Mixed methods  Capture all costs and benefits, including cross- sectoral effects  Cost effectiveness and CBA

 Make explicit underlying theory about how inputs lead to intended outcomes and impacts  Documents every step in causal chain  Draws on multiple data sources and approaches  Stresses context of why or why not working

AssumptionFindings Provide nutritional counselling to care givers Mothers are not decision makers, especially if they live with their mother-in-law Women know about sessions and attend 90% participation, lower in more conservative areas Malnourished and growth faltering children correctly identified No – community nutrition practitioners cannot interpret growth charts Women acquire knowledgeThose attending training do so And knowledge is turned into practice No there is a substantial knowledge- practice gap Supplementary feeding is additional food for intended beneficiary No, considerable evidence of substitution and leakage Adopted changes are sufficient to improve intended outcomes Only sometimes (not for pregnant women)

 Need to collect survey data at the unit of intervention (child, firm etc)  Will need also facility/project data  Need data across the log frame and for confounding factors – and for your instrumental variables (lack of valid instruments is the major obstacle to performing IE)  Designing data collection instruments takes time and should be iterated with qualitative data

StudyData sources Rural electrification3 rural electrification surveys 11 DHS 2 LSMS India irrigation and rural livelihoods Own survey District-level government data Census data Bangladesh maternal and child health and nutrition DHS Project data + national nutrition survey Ghana basic education1988/89 GLSS (LSMS) Own follow up survey Kenya agricultural extension2 previous rural surveys Own follow up survey

OUTLINE YOUR PROPOSED EVALUATION DESIGN (TIMING OF DATA COLLECTION, IDENTIFICATION OF CONTROL, IF ANY) WHAT DATA SOURCES WOULD YOU USE FOR YOUR PROPOSED EVALUATION?

VISIT