Implementation Issues Program roll-out This presentation draws heavily on previous presentations by Dan Levy, Rachel Glennerster, Arianna Legovini, Paul Gertler, and Sebastian Martinez. Africa Impact Evaluation Initiative World Bank

So you’ve design a great impact evaluation… Now it’s time to implement the program. But problems in implementation can invalidate the evaluation. Given the costs of the evaluation, should avoid any second best solutions, or evaluate questions that are not of priority relevance. Key steps The best remedy is prevention – do we really have a program design which is politically and administratively feasible, in the short and long run? Monitor the roll-out carefully – demand high standards of compliance with the original design (small details make a difference); e.g.: accessibility of communities Make adjustments immediately as needed

1. Delays Problem: Program expected to roll out in January but it doesn’t roll out until August; Delay in funding Solution: Move follow-up survey; Delay delivery of program; Collect light baseline; Look at alternative questions (plan B) Concerns Seasonality effects: If the baseline measures the end of school year and the follow-up measures just after a holiday, student performance may not be comparable School calendar Do we have a relevant to the baseline (close to the launch of the program)? Examples: Eritrea Malaria and ECD, Nigeria Malaria Other examples?

2. Timing of benefits and Multiple Treatments Treatment 1: Grants only Treatment 2: Grants + School management training Programs should be rolled out at same time. Suppose benefits roll out to Group 1 in June but to Group 2 in December, with a follow-up survey in January Now measured effect is different treatment + different time with treatment. Can’t measure effects of long term exposure to CCT in Mexico. Other examples?

3. Contamination Problem: Comparison group receives treatment Textbooks roll out to all schools (not by group) Trained teachers transfer to control schools Parents in neighboring non-treated schools learn about innovations in treated schools Some contamination is almost always bound to happen when there are alternative programs available, or NGOs are operating in the same area Solutions If contamination is small, can still estimate effect of being assigned to treatment group – will estimate different parameter, which may still be of interest. If control schools received benefits but less / or different, then measure impact of additional (redefine question – but make sure it is still relevant). If contamination is random… [deworming externalities]. David’s example. Other examples?

4. Heterogeneous treatment Sometimes the quality of the intervention is not the same across all recipients. This may happen because it is difficult to implement an intervention with uniform quality. If quality differences are not randomly assigned this may require some reinterpretation of results, even if they are observed. At the end of the day we have to focus on the question of interest and direct the evaluation towards that question: what type of quality can we deliver in practice?

5. Access to Treatment and Control Areas Because of new constraints, or constraints that were not planned for, some of the communities in the study are not fully accessible, either for treatment, for data collection, or both. If randomization is compromised then the whole study is compromised. Carefully plan for it (and to avoid it), and make an effort to stick with original design.