Impact Evaluation: Presentation to DAC Evaluation Working Group, Paris, June 2, 2005 Howard White Operations Evaluation Department World Bank

What is impact evaluation? Has had different meanings Sector or country-wide evaluation Long-run effects Establishing the counterfactual Focus on final welfare outcomes OED adopt a combination of the last two: ‘a counterfactual-based analysis of how the intervention has affected welfare outcomes’

Impact evaluation in the development context Been associated with poverty, gender and environment effects, using mixed methods with bias toward qualitative But growing tide of quantitative impact evaluation. Driven by: New techniques and technology Results-based agenda (including MDGs)

Impact evaluation at the World Bank As with all evaluation, much work takes place outside of OED For impact there is a new initiative (DIME) promoting greater use of impact evaluation. These studies are: All prospective Attempt to promote randomization OED’s own program is adopting a range of non- experimental approaches, firmly grounded by context

The existing OED program Impact evaluation not new to OED Over 80 studies classified as impact, and others not so classified adopt different approaches to measuring impact Current program under OED-DFID partnership supporting three studies: Ghana basic education Bangladesh maternal and child health India rural poverty

Ghana: Method and approach Main data collection was survey to follow up GLSS2 from 1988 education module Combined with time spent in the field in three visits Background analysis: budget and political economy (context) Multivariate analysis of determinants of educational outcomes. Link those determinants to donor-financed activities Work in collaboration with MOEYS and GSS

Ghana: findings Enrolments are rising Learning outcomes are improving Better school infrastructure is part of explanation Hence Bank-financed school investments lay behind a substantial part of these improvements in education outcomes But growing dichotomization of public sector (partly rectified in new Bank-supported program)

Bangladesh: method and approach Initial meta-analysis Using existing data sets (mainly DHS) Own analysis and commissioned studies Multivariate analysis of determinants of mortality and nutrition (Oaxaca decomposition) For BINP using propensity score matching combining two datasets (problem of poor quality and contaminated control) Holistic approach Work with IMED, and local research company

Bangladesh findings Role of publicly-supported interventions in successful reduction of fertility and mortality Immunization most cost-effective, but so are Training TBAs Female secondary stipends Strong cross-sectoral effects (need not imply multi-sectoral programs) BINP: theory-based approach identifies weak links that help explain poor outcomes

India rural poverty Looking at two interventions AP Irrigation II and III AP Rural Livelihoods Project Three rounds of surveys ( ) Using innovative data collection instruments for ‘immeasurables’

Do you want to do your own impact studies? Pros Demonstrate results Perceived as high quality products, appear to find audience in country teams Cons Expensive Technically demanding So when to use Periodic validation Pilots

What do you need to do your own impact study? Some opportunism in selecting cases Sufficient scale intervention to justify cost Lengthy lead time, especially if collecting own data (18 months from start to finish is best you can hope for, months is more realistic) Appropriate skills mix Promote prospective evaluation

Data collection for impact evaluation ApproachDescription Project-specific surveyOwn data collection in project and control areas Synchronized surveyCarry out project survey at same time as related national-level survey Piggy-backingUtilize an existing national survey, possibly paying for (1) over-sampling project areas, and (2) adding a project-related module

How OED is planning to take its program forward Partnership has helped consolidate commitment to continuing impact evaluation ‘Study a year’ is built into work program One prospective study being agreed (possibly Karnataka health) Open to discussion of expanded program