1. LAURA RALSTON, ECONOMIST, CCSD FINDING TRUE PROGRAM IMPACTS THROUGH RANDOMIZATION

2. SESSION OVERVIEW 1.Background 2.What is a randomized experiment? 3.Why randomize? 4.Key Takeaways Materials used from MIT Open Courseware http://ocw.mit.edu JPAL Executive Training: Evaluating Social Programs 2011 Chris Blattman “Can swords be turned into ploughshares? Experimental effects of an agricultural program on employment, lawlessness, and armed recruitment” with Jeannie Annan

3. IMPACT: WHAT IS IT?

6. HOW TO MEASURE IMPACT? Impact is defined as the comparison between: 1.The outcome some time after the program has been introduced 2.The outcome at that same point in time had the program not been introduced (the counterfactual )

7. COUNTERFACTUAL The counterfactual represents the state of the world that program participants would have experienced in the absence of the program (i.e., had they not participated in the program) Problem : counterfactual cannot be observed Solution : We need to “mimic” or construct the counterfactual

8. IMPACT EVALUATION METHODS Randomized Experiments Also known as: Random Assignment Studies Randomized Field Trials Social Experiments Randomized Controlled Trials (RCTs) Randomized Controlled Experiments

9. IMPACT EVALUATION METHODS Non- or Quasi –Experimental Methods Includes: Pre-Post Simple Difference Difference-in-Difference Multivariate Regression Statistical Matching Instrumental Variables Regression Discontinuity More on these tomorrow

10. 2.What is a randomized experiment? SESSION OVERVIEW

11. THE BASICS Start with simple case: Take a sample of program applicants Randomly assign them to either: Treatment Group – is offered treatment Control Group – not allowed to receive treatment (during the evaluation period) Note: Randomization does not mean denying people the benefits of the program Usually existing constraints in project roll-out allow randomization Randomization is often the fairest way to allocate treatment

12. KEY ADVANTAGE OF EXPERIMENTS Because members of the groups (treatment and control) are randomly selected, they do not systematically differ at the start of the experiment, Any difference that subsequently arises between them can be attributed to the program rather than to other factors.

13. EXAMPLE: “WOMEN AS POLICYMAKERS” TREATMENT VS. CONTROL VILLAGES AT BASELINE Standard Errors in parentheses. Statistics for West Bengal, India. Source: Chattopadhyay and Duflo (2004)

14. RANDOMIZATION EXAMPLE CORE INTERVENTION: CASH PLUS SKILLS TRAINING IMPLEMENTED BY AVSI UGANDA 2009-11 Target: 15 poorest, most marginalized rural women in villages of 80 - 300 households; Nominated by community and screened by NGO Age 27, Work 15 hours/week, earn <$10/month in cash Research questions: What limits the growth of self-employment and income among the poorest and marginalized? Does more work and income “empower” them?

15. SAMPLE: 120 VILLAGES IN 6 SUBCOUNTIES SELECTED BASED ON BEING UNDERSERVED VILLAGES REPRESENT 25% OF SUBCOUNTY POPULATION

16. BUCKET RANDOMIZATION BY VILLAGE TO TREATMENT OR WAITLIST FIRST 60 VILLAGES RECEIVE IMMEDIATE TREATMENT (PHASE 1) 60 RECEIVE DELAYED TREATMENT 18 MONTHS LATER (PHASE 2)

17. SOME VARIATIONS ON THE BASICS Assigning to multiple treatment groups Assigning of units other than individuals or households: Health Centers Schools Local Governments Villages

18. KEY STEPS IN CONDUCTING AN EXPERIMENT 1. Design the study carefully: what is the objective of your impact evaluation? What do you most want to test? 2. Randomly assign people to treatment or control 3.Collect baseline data 4. Verify that assignment looks random 5. Monitor process so that integrity of experiment is not compromised

19. KEY STEPS IN CONDUCTING AN EXPERIMENT (CONT.) 1. Collect follow-up data for both the treatment and control groups 2. Estimate program impacts by comparing mean outcomes of treatment group vs. mean outcomes of control group 3.Assess whether program impacts are statistically significant and practically (size) significant.

20. 3.Why randomize? SESSION OVERVIEW

21. WHY RANDOMIZE? – CONCEPTUAL ARGUMENT If properly designed and conducted, randomized experiments provide the most credible method to estimate the impact of a program

22. WHY “ MOST CREDIBLE ”? Because members of the groups (treatment and control) do not differ systematically at the outset of the experiment, Any differences that subsequently arises between them can be attributed to the program rather than to other factors.

23. EXAMPLE – CAN EMPLOYMENT PROGRAMS REDUCE LAWLESSNESS AND REBELLION? A FIELD EXPERIMENT WITH HIGH-RISK YOUTH IN LIBERIA (BLATTMAN 2014) Knowledge Gaps: 1.Little experimental evidence of employment or incomes on crime or violence (Freeman 1999, Blattman and Miguel 2010) Exceptions are with low-risk populations (Blattman et al 2013, 2014) US experiments test adolescent schooling, neighborhoods 2.Few experimental job programs generate jobs Demobilization and reintegration (Kingma and Muggah 2009) Vocational and business training (Card et al 2010, Attanasio et al 2011, McKenzie & Woodruff 2012) Cash to microenterprises target the already employed (de Mel et al 2008, Fafchamps et al 2012) 3.Where there is evidence, theoretical mechanism unclear Adult education: Opportunity cost, socialization, or peer effects? Income-conflict correlation: Opportunity cost or grievance?

24. INTERVENTION Offer high-risk young people in hotspots: 1.4-month residential training program Highly practice-based agricultural skills 2.“Life skills” and counseling Handling conflict, dealing with trauma and PTSD, career counseling Mentoring by former ex-combatants 3.Assistance returning to a community Leader permission, land access, transport 4.Package of agricultural inputs $125 in tools and materials in two stages Choice between vegetable farming, animals $50 cash (Sinoe site only)

25. AIMS 1.Increase farm incomes and activity 2.Shift occupational incentives away from illicit resource extraction 3.Socialize into peacetime, non- violent life 4.Reduce risk of mercenary recruitment

26. RANDOMIZED EXPERIMENT Suppose we evaluated this program using a randomized experiment Question 1: What would this entail? How would we do it? Question 2: What would be the advantage of using this method to evaluate the impact of the program?

27. METHODS TO ESTIMATE IMPACTS Let’s look at different ways of estimating the impacts using the data from young people who were enrolled in this program: 1.Pre – Post (Before vs. After) 2.Simple difference 3.Difference-in-difference 4.Other non-experimental methods 5.Randomized Experiment

28. PRE-POST (BEFORE VS. AFTER) Look at average change in: Involvement in crime (drug selling, illicit extraction, stealing) Hours per week worked in legal activities (raising animals, farming) Question: under what conditions can these differences be interpreted as an impact of the program? Crime RateHours worked Average Pre Program47%31 Average Post Program41%37 Difference-6%+6

29. WHAT WOULD HAVE HAPPENED IN ABSENCE OF THE PROGRAM?

30. SIMPLE DIFFERENCE Compare crime rates of… Young men who got program with those that didn’t

31. SIMPLE DIFFERENCE Look at average difference in: Involvement in crime (drug selling, illicit extraction, stealing) Hours per week worked in legal activities (raising animals, farming) Question: under what conditions can these differences be interpreted as an impact of the program? Crime RateHours worked Average Outside of Program47%28 Average Inside of Program41%37 Difference-6%+9

32. WHAT WOULD HAVE HAPPENED IN ABSENCE OF THE PROGRAM? Impact = 9 hrs ?

33. DIFFERENCE-IN-DIFFERENCES (MORE ON THIS TOMORROW) Compare decrease in crime rates of… Young men who got program with those that didn’t

34. DIFFERENCE-IN-DIFFERENCES Look at average difference in: Involvement in crime (drug selling, illicit extraction, stealing) Hours per week worked in legal activities (raising animals, farming) Question: under what conditions can these differences be interpreted as an impact of the program? Pre-Crime Rate Post-Crime Rate Difference Average Outside Program47%52%+5% Average Inside Program47%41%-6% Difference11%

35. OTHER METHODS There are more sophisticated non-experimental methods to estimate program impacts: Regression Matching Instrumental Variables Regression Discontinuity But all these methods rely on being able to mimic the counterfactual under certain assumptions Problem: Assumptions are not testable

36. IMPACT OF PROGRAM - SUMMARY MethodImpact on Crime Rates Impact on Hrs worked 1. Pre-post-6%+6* 2. Simple Difference-10%*+9 3. Difference-in-Difference-8%+11* 4. Regression-4%+5 5. Randomized Experiment-5%*+5.5* *: Statistically significant at the 5% level Bottom Line: which method we use matters!

38. 4.Key Takeaways SESSION OVERVIEW

39. KEY TAKEAWAY #1 The single best way to evaluate the true average impact of a program is by randomizing treatment

40. KEY TAKEAWAY #2 Randomization is more flexible than you think: It does not require withholding of benefits It can take advantage of necessary staggered roll- out It can test different reforms or packages across groups at the same time

41. EXAMPLE OF ROLL-OUT RANDOMIZATION 1800 clients in 120 villages 60 villages: Training, grant and follow-up 30 villages: Intensify group formation and cooperation 30 villages: No added services “Phase 1” 300 clients: No follow-up visits 300 clients: “Accountability & advice”: 3-5 follow-up visits 300 clients: Accountability: 1-2 follow-up visits 60 villages: Training and grant 18 months later “Phase 2”

42. KEY TAKEAWAY #3 It is more ethical to test programs rigorously before universally implementing them than it is to use scarce public resources to implement a universal program with uncertain benefits.

43. Thank you !