1. Steps in Implementing an Impact Evaluation
This presentation draws heavily on previous presentations by Dan Levy, Rachel Glennerster, Arianna Legovini, Paul Gertler, and Sebastian Martinez.
Aïchatou Hassane
Africa Impact Evaluation Initiative
World Bank

2. Step 1: Identify priorities
Examine sector plan
examples
Poverty Reduction Strategy Paper
Education Long-term Strategic and Financial Framework
Identify highest priorities for learning
Education
Teacher incentives and management
Learning materials
School-based management (SBM)
If most resources going into SBM, then that’s the opportunity to learn
New policies can also present an opportunity for learning ( Rwanda)

3. How will the school-based management program roll out?
Step 2: Understand roll-out of intervention and opportunities for impact evaluation
How will the school-based management program roll out?
Piloted in a random sample of schools?
Rolled out nationwide?
Rolled out in schools satisfying certain clear crtiteria?
Each roll-out strategy yields distinct opportunities for impact evaluation

4. How will the different incentives for contract teachers be implemented
Step 2+: Understand roll-out of intervention and opportunities for impact evaluation
How will the different incentives for contract teachers be implemented
Piloted in a random sample of districts/schools?
Rolled out in certain districts?
Pick a nationwide representative sample
Each roll-out strategy yields distinct opportunities for impact evaluation

5. 3. Appropriate design Keep in mind
the needs of the intervention: Target needy schools
the evaluation: Take advantage of opportunities for random assignment
School grants example: 1,000 schools to receive grants over 3 years
Randomly assign 300 to each of Phases 1-3
Identify 500 neediest and assign to Phases 1 and 2
Rwanda: 3,000 contract teachers to be hired by year
Representative randomly selected sample of schools with contract teachers.
These 7 steps present a very simplified description of the process. Idea is to give a complete picture on how this works in a typical experiment.

6. 3+. More on design Determine scale: at scale or small pilot? At scale
Nationally representative
More costly to implement
Better information about national effectiveness
Small pilot (e.g., in two districts)
Easier to implement
Not as informative

7. 4. Random assignment
Randomly assign […] to treatment and control groups
Can randomly assign at individual, school, clinic, or community level
School grants: at school level
Treatment package: at individual level
Contract teachers: at school level
Trade-off: higher level means bigger sample

8. 5. Collect baseline data
Baseline data not strictly necessary: Randomization implies treatment and control are similar
but
Allows you to verify that treatment and control appear balanced
Provides valuable data for impact analysis
Did the program mostly benefit patients who were poor at baseline? Or high school performers at baseline?
Allows analysis of targeting efficiency
Take advantage of on-going data collection

9. 5+ Baseline questionnaires
Include areas essential to impact evaluation
Ultimate outcomes we care most about
Intermediate outcomes we expect to change first
Take advantage of opportunity to collect essential sector data
Gambia: corporal punishment
Rwanda: double shift teaching (location and teacher), bonus for teachers paid by parents
Focus group on contract teachers for more information.
Who collects it?
Bureau of Statistics: Integrate with existing data
Ministry concerned: i.e Rwanda: Ministry of Education
Private agency: Sometimes easier quality monitoring

10. 6. Check for balance
Do treatment and control groups look similar at baseline?
If not, all is not lost!
Even in absence of perfect balance, can use baseline data to adjust analysis
Poverty
Female-headed hh
Number of children in hh
Formal sector job
Treatment
70%
64%
3.1
20%
Control
68%
66%
2.9
18%

11. 7. Roll out intervention
Monitor to roll-out to ensure evaluation is not compromised
What if the benefits are accidentally rolled out to everyone, all at once?
Example: New chalkboards go to all schools
Evaluation is compromised: Needed to monitor!
What if all the control households receive some other benefit?
Example: NGO targets control schools to receive lunches. (WFP does over some schools in Rwanda)
Changes evaluation. PTA training at district level

12. 7+ Gather info on roll-out
In reality, who receives which benefits when?
Could affect the impacts measured
Does the intervention involve something other than initially planned?
Example: Learn that those giving resources to clinics gave detailed guidance on clinic management
Program impact now includes the guidance

13. 8. Follow-up data
Collect follow-up data for both the treatment and control groups
Appropriate intervals
Consider how long it should take for outcomes to change
Sub-sample at six months? Intermediate changes
One year
Provide initial outcomes
Adjust program if needed
Two years: Changes in longer term outcomes?
After end of program: Do effects endure?
School feeding in Kenya
What happens once teachers have obtained permanent position or have not obtained permanent position
School feeding refers to

14. 9. Estimate program impacts
Randomization: Simply compare average outcomes for treatment and comparison
Other methods: Make statistical assumptions to estimate impact of program

15. 10. Are they big enough to matter?
Are the effects statistically significant?
Basic statistical test tells whether differences are due to the program or to noisy data
Are they policy significant?
If the anti-HIV media campaign costs a million dollars and has positive effect but it’s tiny, may not be worthwhile

16. 11. Disseminate!
If no one knows about it, it won’t make a difference to policy!
Make sure the information gets into the right policy discussions
Ownership to government, capacity building
Forums
Workshop
Report
Policy brief

17. 12. Iterate
Re-examine sector priorities: Identify next learning opportunity
Or suppose the effects aren’t as big as you hoped
Test variations (ex: different teacher or clinic officer training)
Test other interventions to affect same outcomes (ex: better equipment or school materials)
Test, test, test!

18. Thank you