Nigeria Impact Evaluation Community of Practice Abuja, Nigeria, April 2, 2014 Measuring Program Impacts Through Randomization David Evans (World Bank)

Objective 2  Evaluate the causal impact of a program or an intervention on some outcome  Examples  How much did free distribution of bednets decrease malaria incidence?  Which of two supply chain models was most effective at eliminating drug shortages?

Counterfactual Criteria 3 Treated & comparison groups…  Have identical average characteristics (observed & unobserved)  The only difference is the treatment  Therefore the only reason for any difference in outcomes is the treatment  Key question: What would participant look like if she hadn’t received the program?

Perfect Experiment 4 1. Identify target beneficiaries 2. Clone them! Identical on the outside (observable) Identical on the inside (unobservable) Chief Ahun We’re both middle- aged chiefs We both love to take up new health interventions! Chief Batun

Perfect Experiment 5 Give the intervention to one set of clones Ahun Batun

Perfect Experiment 6 Observe some time later Because the groups are identical (inside & out), the difference is due to the bednets! Ahun Batun

Back to Reality 7 What would Batun look like if he didn’t receive the bednet? Room For Improvement Control Groups  Before – After  Participants – Non Participants ???

RFI: Before-After BEFORE BEDNETS 6 malaria episodes in 6 months AFTER BEDNETS 2 malaria episodes in 6 months What else might be going on besides the bednets? Seasonal differences Rising incomes: Households invest in other measures Too many other factors! Impact of bednets = ???

RFI: Before-After  Important to monitor before-after  Insufficient to show impact of program  Too many factors changing over time  Example of cash transfers in Nicaragua!  Counterfactual: What would have happened in the absence of the project, with everything else the same

RFI: Participants vs Non-Participants 10  Compare recipients of a program to  People who were not eligible for the program  People who chose not to enroll in the program Impact of clinic births? What else might explain the difference?

RFI: Participants vs Non-Participants 11 Observable differences  Income  Education Unobservable differences  Heard rumor about hospitals  Neighbor available to care for other children Mercy Patience

RFI: Participants vs Non-Participants 12 How much of difference is because of clinic? Impact of clinic births = ??? Impact of clinic births Other factors!

Selection bias 13  People who choose to join the program are different!  If we cannot account completely for those differences in our data…  We never can  How do you capture attitudes toward health systems? Initiative?  …then our comparison will not show the true impact of the program

What should we do? Gold standard: Randomized experimental design

Randomized Experimental Design 15  Randomly assign potential beneficiaries to be in the treatment or comparison group  Treatment and comparison have the same characteristics (observed and unobserved), on average, so…  Any difference in outcomes is due to treatment

Why Randomization Works 16  Randomization with two doesn’t work!  But differences average out in a big sample  On average, same number of Ahuns and Patiences  Observable AND unobservable  Result: Measure true impact of program Comparison Treatment Comparison Treatment

Random Sample or Assignment? RANDOM SAMPLE  Select randomly who to gather data on  Gives unbiased average of the group NOT of impact  If take random sample of group: Half women, half men – Sample should be about ½ women, ½ men RANDOM ASSIGNMENT  Randomly assign who gets the program  Gives unbiased estimate of program impact  Why?  Treatment & comparison are IDENTICAL (on average) 17 T T C C

LET’S RANDOMIZE! Identify the eligible participants What is the impact of receiving a new car on body-mass index?

LET’S RANDOMIZE! Generate a random number for each one

LET’S RANDOMIZE! Generate a random number for each one

LET’S RANDOMIZE! Re-order based on the random number

LET’S RANDOMIZE! Assign the first ten to receive cars I really wanted a car!

LET’S RANDOMIZE! Check for balance across treatment & control Treatment # Drs: 4 Control # Drs: 6 Total observations 21 Not a very big sample! How close? 2/3

LET’S RANDOMIZE! 24 What if we had 500 observations? Treatment # Drs: 114 Control # Drs: 116 How close? 98/100

Is there more? 25  That’s a simple way to randomize  Works with BIG samples  You can help randomization by stratifying  Randomize within each sub-group  Ensure that each group is equally represented

Stratifying Identify the characteristic(s) to stratify on

Stratifying Sort on those characteristics

Stratifying Generate those random numbers

Stratifying Sort on those characteristics

Stratifying Assign treatment within each sub-group Result: Equal doctors in each group

Can we really randomize? 31  Randomization does not mean denying people the benefits of the project  Usually existing constraints in project roll-out allow randomization  Randomization often the fairest way to allocate treatment

Use Staggered Roll-out 32 Roll-out to 200 clinics Roll-out to 200 more clinics Roll-out to 400 more clinics Jan 2014 July 2014Jan 2015 Randomize the order in which clinics receive program Compare Jan 2014 group to Jan 2015 group at end of first year

Some groups must get the program 33

Vary treatment INTENSITY OF TREATMENT Malaria Information Campaign 100 villages Malaria Information Campaign + SMS Reminders 100 villages NATURE OF TREATMENT Radio campaign 100 villages Newspaper campaign 100 villages 34

Randomization is often the fairest 35 Watch the movie! [link]link Randomization of an early child development program in Côte d’Ivoire

What if randomization is impossible? 36  Think again: It often is possible on some level, and it’s the best way to get a clear measure of impact  Always begin the IE with imagining what the ideal would look like  With a national policy  Use randomization to test implementation

Key takeaway #1 The single best way to evaluate the unbiased average impact of an activity is by randomizing treatment. 37

Key takeaway #2 It is more ethical to test programs rigorously before universally implementing them than it is to use scarce public resources to implement a universal program with uncertain benefits. 38

Key takeaway #3 Randomization is more flexible than you think:  It does not require withholding of benefits.  It can take advantage of necessary staggered roll-out.  It can test different reforms or packages of services across groups at the same time (so all receive at least some package). 39

Let’s randomize 40

Thank you! 41

BONUS SLIDES

43

LET’S RANDOMIZE! 44 What if we had 100 observations? Treatment # Drs: 26 Control # Drs: 22 Total observations 21 Not a very big sample!