Operational Aspects of Impact Evaluation Arianna Legovini Lead Specialist Africa Impact Evaluation Initiative

Implementing impact evaluation Shift of paradigm Institutional setting Key steps Team capacity & Bank support Evaluation questions Choosing methodology Data, Analysis & Dissemination Budget & Financing Timeline

Paradigms Program is a set of activities defined at time zero designed to deliver expected results. Program will either deliver or not Program is menu of alternatives, with a strategy to find out which are best Activities might change overtime. Project will deliver more in the future than it might do at the beginning

Retro- & Pro-spective Old: retrospective Look back and judge New: prospective Decide what need to learn Experiment with alternatives Measure and inform Adopt best alternatives

Prospective evaluation can Measure effectiveness of operation Measure effectiveness of alternatives modes of delivery packages pricing schemes Provide rigorous evidence to modify project features overtime (managing for results) Inform future project designs

“Infrastructure of evaluation” Not a one shot study, but… Institutional framework linking evaluation to policy cycle + An analytical framework and data system for sequential learning

Institutional framework PM/Presidency: Communicate to constituencies Treasury/ Finance: Allocate budget Line ministries: Deliver programs and negotiate budget Cost-effectiveness of different programs Effects of government program Cost-effectiveness of alternatives and effect of sector programs

Institutional framework M&E staff IE specialists Local researchers Statisticians Data collectors Policy-makers Program managers

Key steps 2. Design evaluation 3. Prepare data collection instruments 1. Identify policy questions 3. Prepare data collection instruments 4. Collect baseline 5. Implement program in treatment areas 6. Collect follow up 8. Improve program and implement new 7. Analyze & feedback

Build team capacity Provide training opportunity for team and operational staff from the outset Keep training team and implementing agencies overtime Keep team involved in the work: learning by doing is best Bank support: Cross country training, coordination, network & knowledge sharing Cross country technical assistance (measurement, instruments, analytical tools) Country-specific technical assistance (project team, research team and field coordinator)

Keep it useful! Answer the questions that are most important to improve program Use evaluation discussion to question your program Use baseline to adjust targeting and learn more about the factors that explain the problems you face Time pilots to give you quick answers on behavioral change (e.g. take up rates) and adjust program in the early stages Give enough exposure time to ensure you can measure real results Use evaluation results to improve program Get better results

Operational questions Question design-choices of operation Develop causal chain Ask whether equally likely alternatives should be considered Think of what choices were made on hunches rather than solid evidence Identify program features that are being changed and question underlying rationale

Dynamic learning / managing for results Use random trials to test alternatives Focus on short term outcomes e.g. take up rates, use, adoption Follow up data collection and analysis 6-12 months after exposure Measure impact of alternative treatments on ST outcomes and identify “best” Change program to adopt best alternative Start over

Policy questions Is the operation achieving its objectives? Does the operation have other desirable or undesirable effects? Is it cost-effective?

Fiscal accountability Use most rigorous method of evaluation Focus on higher level outcomes e.g. educational achievement, health status, income Follow up data collection and analysis at yearly or bi yearly intervals (1-10 years horizon) Measure impact of operation on stated objectives and range of other outcomes Compare with results from other programs Allocate budget toward more effective programs and away from least effective programs

Example: Malaria Government target for under five child mortality = 50 ST-Operational question: will we achieve target? Experiment with alterative communication campaigns (radio, community, door-to-door) to increase use of nets for children under five Measure use of nets for each type of communication campaign Evaluate within the rain season 2-4 months Adopt communication campaigns for the program LT-Policy questions: will net use secure expected gains in mortality? Compare mortality over time in phase 1 communities with phase 2 communities Longer horizon (until control is available)

Choosing the Methodology Methodological issues Identify control group Outcome indicators Available data and required data Sampling and sample sizes Choose the most robust strategy that fits the operational context Complement operations Do not deny benefits for evaluation purpose

Using opportunities to find Control Groups Limited roll out capacity Use phase in and give equal chance to people to be in phase I or II (I treatments, II controls) Limited budget Establish clear eligibility criteria: regression discontinuity Give eligible population equal chance to participate: random assignment Universal eligibility Vary treatment Encourage participation

Collecting Data Measure indicators at baseline in both control and treatment groups Follow-up surveys with enough exposure time to detect changes How will you collect data? Available in administrative data? Can be added? Collected through existing surveys? Do we need special surveys?

Administrative data Typical content Lists of beneficiaries Distribution of benefits Expenditures Outputs Information used by impact evaluation to verify: Beneficiaries, Benefits, Timing Data can be used for IE analysis if it covers: Control population Outcome indicators

Survey data Who do you survey? Households? Communities? Firms? Facilities? Content: information needed for analysis and covariates Researchers with survey experts design the questionnaires Harmonize basic questionnaires across evaluations and add evaluation-specific modules Develop set of common outcome indicators to compare results across different evaluations

Sample Based on power calculations Larger the size the rarer the outcome the larger the standard deviation of the outcome indicator the smaller the effect size you want to detect the larger the number of subpopulations you want to analyze (gender, regions, income groups)

Manage trade-offs Method Experimental Non-experimental Data Least costly Best estimates Not always feasible Non-experimental More expensive Good second best estimates Always feasible Data Administrative data (expanded to control) Most cost effective Piggy back on ongoing data collection Cost effective Coordination intensive Collect special surveys Tailored and timely Expensive

Qualitative and Quantitative are complementary-not substitutes Gain insights Develop hypotheses Explain quantitative Quantitative Measure Generalize Replicate Mixed Qual/Quant Qualitative informs & enriches quantitative Quantitative tests qualitative

Monitor Intervention Evaluation requires intensive monitoring to: Ensure intervention is rolled out as planned Ensure design of evaluation is respected This is to avoid contamination of evaluation results All people involved in the program should be informed Train implementers: they need to know why Hire supervisor: avoid or at least record deviations from plan Strengthen reporting systems

Analysis and Dissemination Analyze baseline and each follow-up Disseminate and discuss on an on-going basis Tailor product to different audiences Politicians- one liners Policy makers- one pagers Press- one pagers Program managers- non-technical summaries Researchers – academic papers, data access

Budget Financing Data collection BIG BUDGET ITEM Consultants Travel Staff time Country: Establish which funds are already available as part of project funds and MAP allocations Prepare and submit funding proposal (IE concept note with detailed budget) Bank TTL: Include IE budget in project/MAP financing Initiate product IE-code and obtain BB supervision funds Financing

Timeline (18-24 months) Preparation 3-6 months Design and institutional approval Funding and Procurement Questionnaire development Training Pilot testing Baseline data collection Field Work (1-2 months), Data input (1-2 months) Exposure period Follow-up data collection Analysis and dissemination Program and Policy Feedback loop

Operational messages Plan evaluation while planning project Build an explicit team Influence roll-out to obtain control groups Strengthen monitoring systems Develop budget and financing plan Schedule evaluation to match policy cycles Early results buy interest