1. Implementation Issues Program roll-out
This presentation draws heavily on previous presentations by Dan Levy, Rachel Glennerster, Arianna Legovini, Paul Gertler, and Sebastian Martinez.
Africa Impact Evaluation Initiative
World Bank

2. So you’ve design a great impact evaluation…
Now it’s time to implement the program.
But problems in implementation can invalidate the evaluation.
Given the costs of the evaluation, should avoid any second best solutions, or evaluate questions that are not of priority relevance.
Key steps
The best remedy is prevention – do we really have a program design which is politically and administratively feasible, in the short and long run?
Monitor the roll-out carefully – demand high standards of compliance with the original design (small details make a difference); e.g.: accessibility of communities
Make adjustments immediately as needed

3. 1. Delays
Problem: Program expected to roll out in January but it doesn’t roll out until August; Delay in funding
Solution: Move follow-up survey; Delay delivery of program; Collect light baseline; Look at alternative questions (plan B)
Concerns
Seasonality effects: If the baseline measures the end of school year and the follow-up measures just after a holiday, student performance may not be comparable
School calendar
Do we have a relevant to the baseline (close to the launch of the program)?
Examples: Eritrea Malaria and ECD, Nigeria Malaria
Other examples?

4. 2. Timing of benefits and Multiple Treatments
Treatment 1: Grants only
Treatment 2: Grants + School management training
Programs should be rolled out at same time.
Suppose benefits roll out to Group 1 in June but to Group 2 in December, with a follow-up survey in January
Now measured effect is different treatment + different time with treatment.
Can’t measure effects of long term exposure to CCT in Mexico.
Other examples?

5. 3. Contamination Problem: Comparison group receives treatment
Textbooks roll out to all schools (not by group)
Trained teachers transfer to control schools
Parents in neighboring non-treated schools learn about innovations in treated schools
Some contamination is almost always bound to happen when there are alternative programs available, or NGOs are operating in the same area
Solutions
If contamination is small, can still estimate effect of being assigned to treatment group – will estimate different parameter, which may still be of interest.
If control schools received benefits but less / or different, then measure impact of additional (redefine question – but make sure it is still relevant).
If contamination is random… [deworming externalities].
David’s example. Other examples?

6. 4. Heterogeneous treatment
Sometimes the quality of the intervention is not the same across all recipients. This may happen because it is difficult to implement an intervention with uniform quality.
If quality differences are not randomly assigned this may require some reinterpretation of results, even if they are observed.
At the end of the day we have to focus on the question of interest and direct the evaluation towards that question: what type of quality can we deliver in practice?

7. 5. Access to Treatment and Control Areas
Because of new constraints, or constraints that were not planned for, some of the communities in the study are not fully accessible, either for treatment, for data collection, or both.
If randomization is compromised then the whole study is compromised.
Carefully plan for it (and to avoid it), and make an effort to stick with original design.