1. Randomized Evaluations: Applications Kenny Ajayi September 22, 2008 Global Poverty and Impact Evaluation

2. Review  Causation and Correlation (5 factors)  X → Y  Y → X  X → Y and Y → X  Z → X and Z → Y  X ‽ Y

3. Review  Causation and Correlation (5 factors)  X → Ycausality  Y → Xreverse causality  X → Y and Y → Xsimultaneity  Z → X and Z → Yomitted variables / confounding  X ‽ Y spurious correlation  Reduced-form relationship between X and Y

4. Review  Exogenous

5. Review  Exogenous  Endogenous

6. Review  Exogenous  Endogenous  External Validity

7. Review  Exogenous  Endogenous  External Validity  Natural Experiment

8. Practical Applications  Noncompliance  Cost Benefit Analysis  Theory and Mechanisms

9. Kling, Liebman and Katz (2007)  Do neighborhoods affect their residents?

10. Do neighborhoods affect their residents?  Positively

11. Do neighborhoods affect their residents?  Positively  Social connections, role models, security, community resources

12. Do neighborhoods affect their residents?  Positively  Social connections, role models, security, community resources  Negatively

13. Do neighborhoods affect their residents?  Positively  Social connections, role models, security, community resources  Negatively  Discrimination, competition with advantaged peers

14. Do neighborhoods affect their residents?  Positively  Social connections, role models, security, community resources  Negatively  Discrimination, competition with advantaged peers  Not at all

15. Do neighborhoods affect their residents?  Positively  Social connections, role models, security, community resources  Negatively  Discrimination, competition with advantaged peers  Not at all  Only family influences, genetic factors, individual human capital investments or broader non-neighborhood environment matter

16. Do neighborhoods affect their residents?  What is Y?

17. Do neighborhoods affect their residents?  What is Y?  Adults  Self sufficiency  Physical & mental health  Youth  Education (reading/math test scores)  Physical & mental health  Risky behavior

18. Kling, Liebman and Katz (2007)  Do neighborhoods affect their residents?  X → Ycausality  Y → Xreverse causality  X → Y and Y → Xsimultaneity  Z → X and Z → Yomitted variables / confounding  X ‽ Y spurious correlation

19. Neighborhood Effects  Natural Experiment

20. Neighborhood Effects  Natural Experiment  Hurricane Katrina  Can we randomly assign people to live in different neighborhoods?

21. Neighborhood Effects  Natural Experiment  Hurricane Katrina  Can we randomly assign people to live in different neighborhoods?  Sort of…

22. Moving to Opportunity  Solicit volunteers to participate  Randomly assign each volunteer an option  Control  Section 8  Section 8 with restriction (low poverty neighborhood)  Volunteers decide whether to move  Compliers: use voucher  Non-compliers: don’t move

23. Moving to Opportunity Public housing residents VolunteerSection 8Move Don’t move Control Restricted Section 8 Don’t move Move Don’t volunteer

24. Non-random Participation  Why is this OK?  Volunteers presumably care about their neighborhood environment, so should be the target when thinking about uptake for the use of housing vouchers BUT  Results might not generalize to other populations with different characteristics

25. Treatment and Compliance  With perfect compliance:  Pr(X=1│Z=1)=1  Pr(X=1│Z=0)=0  With imperfect compliance:  1>Pr(X=1│Z=1)>Pr(X=1│Z=0)>0 Where X - actual treatment Z - assigned treatment

26. Some Econometrics  Intention-to-treat (ITT) Effects  Differences between treatment and control group means

27. Some Econometrics  Intention-to-treat (ITT) Effects  Differences between treatment and control group means ITT = E(Y|Z = 1) – E(Y|Z = 0) Y - outcome Z - assigned treatment

28. Some Econometrics  Intention-to-treat (ITT) Effects  Writing this relationship in mathematical notation Y = Zπ 1 + Wβ 1 + ε 1 Individual wellbeing (Y) depends on whether or not the individual is assigned to treatment (Z), baseline characteristics (W), and whether or not the individual got lucky (ε) π 1 - effect of being assigned to treatment group = (effect of actually moving) x (compliance rate)

29. Some Econometrics  Effect of Treatment on the Treated (TOT)  Differences between treatment and control group means Differences in compliance for treatment and control groups

30. Some Econometrics  Effect of Treatment on the Treated (TOT)  Differences between treatment and control group means Differences in compliance for treatment and control groups  TOT = “Wald Estimator” = E(Y|Z = 1) – E(Y|Z = 0) E(X|Z = 1) – E(X|Z = 0) Y - outcome Z - assigned treatment X - actual treatment (i.e. compliance)

31. Some More Econometrics  Estimate TOT using instrumental variables  Use offer of a MTO voucher as an instrument for MTO voucher use

32. Some More Econometrics  Estimate TOT using instrumental variables  Use offer of a MTO voucher as an instrument for MTO voucher use. In mathematical notation: (1)X = Z ρ d + W β d + ε d 1. Predict whether people use voucher (X), given their assigned treatment (Z) and observable characteristics (W)

33. Some More Econometrics  Estimate TOT using instrumental variables  Use offer of a MTO voucher as an instrument for MTO voucher use. In mathematical notation: (1)X = Z ρ d + W β d + ε d (2)Y = Xγ 2 + W β 2 + ε 2 1. Predict whether people use voucher (X), given their assigned treatment (Z) and observable characteristics (W) 2. Estimate how much individual wellbeing (Y) depends on predicted compliance (X) and observable characteristics (W)

34. Some Econometrics  Effect of Treatment on the Treated (TOT)  Writing the reduced-form relationship in mathematical notation Y = Xγ 2 + Wβ 2 + ε 2 Individual wellbeing (Y) depends on whether or not the individual complies with treatment (X), baseline characteristics (W), and whether or not the individual got lucky (ε) Note:γ 2 - effect of actually moving (i.e. compliance) = effect of being assigned to treatment group compliance rate

35. Moving to Opportunity Public housing residents VolunteerSection 8Move Don’t move Control Restricted Section 8 Don’t move Move Don’t volunteer

36. Moving to Opportunity Public housing residents VolunteerSection 8Move Don’t move Control Restricted Section 8 Don’t move Move Don’t volunteer INTENTION TO TREAT

37. Moving to Opportunity Public housing residents VolunteerSection 8Move Don’t move Control Restricted Section 8 Don’t move Move Don’t volunteer EFFECT OF TREATMENT ON THE TREATED

38. Results  Positive effects  Teenage girls  Adult mental health  Negative effects  Teenage boys  No effects  Adult economic self-sufficiency & physical health  Younger children  Increasing effects for lower poverty rates

39. Limitations  Can’t fully separate relocation effect from neighborhood effect  Can’t observe spillovers into receiving neighborhoods

40. Some Thoughts  We can still estimate impacts in cases where we don’t have full compliance  Policies sometimes target people who are different from the general population (i.e. those more likely to take up a program) BUT  We have to be careful about generalizing these results for other populations ( external validity )

41. Practical Applications  Noncompliance  Cost Benefit Analysis  Theory and Mechanisms

42. Duflo, Kremer & Robinson (2008)  Why don’t Kenyan farmers use fertilizer?

43. Duflo, Kremer & Robinson (2008)  Why don’t Kenyan farmers use fertilizer?  Low returns  High cost  Preference for traditional farming methods  Lack of information

44. Is Fertilizer Use Profitable?  What is Y (outcome)?  Crop yield  Rate of return over the season  What is X (observed)?  Fertilizer  Hybrid seed

45. Is Fertilizer Use Profitable?  What is Z (unobserved)?

46. Is Fertilizer Use Profitable?  What is Z (unobserved)?  Farmer experience/ability  Farmer resources  Weather  Soil quality  Market conditions

47. Correlation or Causation? Fertilizer Use (X) and Crop Yield (Y)

48. Correlation or Causation? Fertilizer Use (X) and Crop Yield (Y)  X → Ycausality  Y → Xreverse causality  X → Y and Y → Xsimultaneity  Z → X and Z → Yomitted variables / confounding  X ‽ Y spurious correlation

49. Fertilizer Experiment  How could we test the effect of fertilizer use on crop yield using randomization?

50. Fertilizer Experiment  Randomly select participating farmers  Measure 3 adjacent equal-sized plots on each farm  Randomly assign farming package for each plot  Top dressing fertilizer (T 1 )  Fertilizer and hybrid seed (T 2 )  Traditional seed (C)  Weigh each plot’s yield at harvest

51. Fertilizer Experiment  Calculate rate of return over the season: value of treatment -comparison -value of plot output plot outputinput RR = ----------------------------------------------------------- - 100 value of input

52. Results

55. Some Thoughts  More comprehensive package increased yield BUT  Lower cost package had highest rate of return  Lab results don’t always apply in the field  Think about the COSTS and BENEFITS

56. Practical Applications  Noncompliance  Cost Benefit Analysis  Theory and Mechanisms

57. Karlan and Zinman (2007)  What is the optimal loan contract?

58. Objectives  Profitability  Gross Revenue  Repayment  Targeted Access  Poor households  Female microentrepreneurs  Sustainability

59. Theory of Change Loan Profits contract  Mechanisms  How do we expect a change to affect the outcome?  What assumptions are we making?  Can we test if they are true?

60. Loan Offer Experiment  What is Y?  Take-up  Loan size  Default  What is X?  Interest rate  Maturity  How sensitive are borrowers to interest rates and maturities?

61. Loan Offer Experiment

62. Results  Downward sloping demand curves  Loan size more responsive to loan maturity than to interest rate

63. Some Limitations  Again, results apply only to people who had previously borrowed from the lender.  Elasticities apply to direct mail solicitations and might not hold for other loan offer methods

64. Final Thoughts  Randomized evaluation can have many applications beyond merely looking at reduced-form relationships between X and Y