Africa Program for Education Impact Evaluation Dakar, Senegal December 15-19, 2008 Experimental Methods Muna Meky Economist Africa Impact Evaluation Initiative

2 Motivation Objective in evaluation is to estimate the CAUSAL effect of intervention X on outcome Y –What is the effect of a housing upgrade on household income? For causal inference, we need to understand exactly how benefits are distributed –Assigned / targeted –Take-up

3 Causation versus Correlation Correlation is NOT causation –Necessary but not sufficient condition –Correlation: X and Y are related Change in X is related to a change in Y And…. A change in Y is related to a change in X –Example: age and income –Causation – if we change X how much does Y change A change in X is related to a change in Y Not necessarily the other way around

4 Causation versus Correlation Three criteria for causation: –Independent variable precedes the dependent variable. –Independent variable is related to the dependent variable. –There are no third variables that could explain why the independent variable is related to the dependent variable.

5 Statistical analysis: Typically involves inferring the causal relationship between X and Y from observational data –Many challenges & complex statistics –We never know if we’re measuring the true impact Impact Evaluation: –Retrospectively: same challenges as statistical analysis –Prospectively: we generate the data ourselves through the program’s design  evaluation design makes things much easier! Statistical Analysis & Impact Evaluation

6 How to assess impact What is the effect of a housing upgrade on household income? Ideally, compare same individual with & without programs at same point in time What’s the problem? The need for a good counterfactual –What are the requirements?

7 Case study: housing upgrade Informal settlement of 15,000 households Goal: upgrade housing of residents Evaluation question: What is the impact of upgrading housing on household income? on employment? Counterfeit counterfactuals

8 Gold standard: Experimental design Only method that ensures balance in unobserved (and observed) characteristics  Only difference is treatment Equal chance of assignment into treatment and control for everyone With large sample, all characteristics average out Experimental design = Randomized evaluation

9 “Random” What does the term “random” mean here? –Equal chance of participation for everyone How could one really randomize in the case of housing upgrading? Options –Lottery –Lottery among the qualified –Phase-in –Encouragement –Randomize across treatments

10 Kinds of randomization Random selection: external validity –Ensure that the results in the sample represent the results in the population –What does this program tell us that we can apply to the whole country? Random assignment: internal validity –Ensure that the observed effect on the outcome is due to the treatment rather than other factors –Does not inform scale-up without assumptions Example: Housing upgrade in Western Cape vs Sample from across country

11 Randomization External Validity (sample) Internal Validity (identification) External vs Internal

12 Example of Randomization What is the impact of providing free books to students on test scores? Randomly assign a group of school children to either: - Treatment Group – receives free books - Control Group – does not receive free books

13 Randomization Random Assignment

14 How Do You Randomize? 1) At what level? –Individual –Group School Community District

15 When would you use randomization? Universe of eligible individuals typically larger than available resources at a single point in time –Fair and transparent way to assign benefits –Gives an equal chance to everyone in the sample Good times to randomize: –Pilot programs –Programs with budget/capacity constraints –Phase in programs

16 Basic Setup of an Experimental Evaluation Potential Participants Evaluation Sample Random Assignment Treatment Group Control Group ParticipantsNo-Shows Based on Orr (1999) All informal settlement dwellers Communities that might participate or a targeted sub- group Select those you want to work with right now

17 Examples…

18 Beyond simple random assignment Assigning to multiple treatment groups –Treatment 1, Treatment 2, Control –Upgrade housing in situ, relocation to better housing, control –What do we learn? Assigning to units other than individuals or households –Health Centers (bed net distribution) –Schools (teacher absenteeism project) –Local Governments (corruption project) –Villages (Community-driven development projects)

19 Unit of randomization Individual or household randomization is lowest cost option Randomizing at higher levels requires much bigger samples: within-group correlation Political challenges to unequal treatment within a community –But look for creative solutions: e.g., uniforms in Kenya Some programs can only be implemented at a higher level –e.g., strengthening school committees

20 Efficacy & Effectiveness Efficacy –Proof of Concept –Pilot under ideal conditions Effectiveness –At scale –Normal circumstances & capabilities –Lower or higher impact? –Higher or lower costs?

21 Advantages of experiments Clear causal impact Relative to other studies –Much easier to analyze –Cheaper! (smaller sample sizes) –Easier to convey –More convincing to policymakers –Not methodologically controversial

22 What if randomization isn’t possible? It probably is… Budget constraints: randomize among the needy Roll-out capacity: randomize who receives first Randomly promote the program to some

23 When is it really not possible? The treatment has already been assigned and announced and no possibility for expansion of treatment The program is over (retrospective) Universal eligibility and universal access –Example: free education, exchange rate regime