Knowing if the RBF mechanism is working Incorporating Rigorous Impact Evaluation into your HRBF program Sebastian Martinez World Bank

Overview The priest and the taxi driver Why evaluate? What is an Impact Evaluation? Asking the right questions Supporting your impact evaluation

Why Evaluate? Generate evidence on what works  Close the knowledge gap in HRBF  Effective use of scarce resources Improve program design overtime  Manage-by-Results Information key to sustainability  National Budget – scale up program  Public opinion – transparency and accountability Institutional requirements  PAD guidelines: “Baseline data collected”  ICR guidelines (new, 2006) : “Evaluation of the borrower’s own performance…[and] of the performance of the Bank”; and “development of the borrower’s independent evaluation capacity, including capacity for impact evaluation”

Political Economy Cultural shift  From retrospective evaluation  Look back and (try to) judge  To prospective evaluation  Decide what we need to learn  Experiment with alternatives  Measure and inform  Adopt better alternatives overtime Change in incentives  Rewards for improving programs  Rewards for generating knowledge  Separating job performance from knowledge generation

Overview The priest and the taxi driver Why evaluate? What is an Impact Evaluation? Asking the right questions Supporting your impact evaluation

What makes an Impact Evaluation? Process Evaluation / Monitoring: Impact Evaluation: Is program being implemented? Is program targeting the right population? Are outcomes moving the right direction? Descriptive analysis What is the effect of the program on outcomes? Which alternative features of the program are most effective? Causal analysis IE is NOT a substitute for good monitoring systems & administrative data. For a comprehensive evaluation of our programs, we need both!

Evaluation problem: need a counterfactual Compare Y before and after intervention Impact A-B ? What would have happened? Impact  Boom = A-D  Recession = A-C Y AfterBefore A B C? T0T0 D? Time O Treatment T1T1

Estimating Impact of the Program on an outcome Y OBSERVE outcome under treatment ESTIMATE counterfactual  Use comparison or control group α= (Y | P=1)-(Y | P=0) IMPACT = outcome with treatment - counterfactual

The perfect “Clone” 6 Vistits Impact = = 2 Visits BeneficiaryControl 4 Visits

In reality, use statistics Average Y = 6 visits Impact = = 2 Visits BeneficiaryControl Average Y = 4 visits

Prospective Impact Evaluation Designs Opportunities to generate good controls come from operational rules The majority of programs cannot assign benefits to all the entire eligible population  Budget limitations: Eligible beneficiaries that receive benefits are treatments Eligible beneficiaries that do not receive benefits are controls  Logistical limitations: Those that go first are treatments Those that go later are controls  Eligibility rules Many times important operational decisions benefit from empirical evidence  Test different program components/models

Overview The priest and the taxi driver Why evaluate? What is an Impact Evaluation? Asking the right questions Supporting your impact evaluation

HRBF Impact Evaluation Questions Answer the important policy questions Is the program working?  What is the effect of the supply side intervention on outcomes?  What is the effect of the demand side interventions?  Do supply and demand incentives together work better than the sum of its parts?  What is the cost-effectiveness of interventions?  Are the impacts sustainable? Could the program work better?  Opening up the black box How much? In cash/in kind? How frequently? How long?

IE Cycle 1. Identify policy questions 2. Design evaluation 3. Prepare data collection instruments 4. Collect baseline 5. Implement program in treatment areas 6. Collect follow up 8. Improve program and implement new 7. Analyze and feedback

Overview The priest and the taxi driver Why evaluate? What is an Impact Evaluation? Asking the right questions Supporting your impact evaluation

Impact Evaluation Ingredients Theory (logical framework)  Formulate questions  Indicators Measuring Impact  Identification Strategy Data Technical Support Resources

Technical support for your evaluation Country Specific IE Teams  PIs – evaluation design  Coordinators – day to day management Survey firms  Collect data IE Support Network  Health Systems Measurement Facility Surveys Exit Polls  Health Outcome Measurement Maternal and Child Heath  Data Collection Sampling and Power Data quality control: field procedures  Costing

IE Support Network - a Resource Center Support for country programs  Technical advisory group for sampling, power calculations, costing, data capturing program, field work procedures & quality control for survey firm Harmonize measurement so results are comparable across sites  Rigorous evaluation methodologies  Common set of basic survey instruments (HH, Facility, Community) & measurement protocols Tailored to country context Summarize and disseminate lessons learned What other technical support? Let me know……

Next Steps Harmonization of measurement and methods First IE support network meeting VCs with country teams  July 08 Cross-country HRBF IE seminar  November 08 Support country programs with evaluation design Intensive 2-3 day in-country IE workshops  On Demand Remote support via VC/teleconference  On Demand

Final Messages Plan the impact evaluation into the program early  Tailored policy questions Answer the key questions  Better methods Precise estimates  Timely feedback and program changes Improve results on the ground Remember, when planning your evaluation:  Include a control group  Collect baseline data