1. N ON -E XPERIMENTAL M ETHODS Shwetlena Sabarwal (thanks to Markus Goldstein for the slides)

2. O BJECTIVE Find a plausible counterfactual Every non-experimental method is associated with an assumption The stronger the assumption the weaker the estimate TEST ASSUMPTIONS Reality check

3. P ROGRAM TO EVALUATE  Hopetown HIV/AIDS Program (2008-2012)  Objectives  Reduce HIV transmission  Intervention: Peer education  Target group: Youth 15-24  Outcome Indicator: Pregnancy rate (proxy for unprotected sex)

4. I. B EFORE - AFTER IDENTIFICATION STRATEGY Counterfactual: Rate of pregnancy observed before program started EFFECT = After minus Before

5. I. B EFORE - AFTER IDENTIFICATION STRATEGY Counterfactual: Rate of pregnancy observed before program started EFFECT = After minus Before Year Number of areas Teen pregnancy rate (per 1000) 20087062.90 20127066.37 Difference+3.47

6. C OUNTERFACTUAL ASSUMPTION : No change over time Effect = +3.47 Intervention Question: what else might have happened in 2008-2012 to affect teen pregnancy?

7. Number of areas Teen pregnancy (per 1000) 200420082012 7054.9662.9066.37 TEST ASSUMPTION with prior data REJECT counterfactual hypothesis of no change over time

8. II. N ON -P ARTICIPANT I DENTIFICATION S TRATEGY Counterfactual: Rate of pregnancy among non-participants Teen pregnancy rate (per 1000) in 2012 Participants66.37 Non-participants57.50 Difference+8.87

9. C OUNTERFACTUAL ASSUMPTION : Without intervention participants have same pregnancy rate as non-participants Effect = +8.87 Participants Non-participants Question: how might participants differ from non- participants?

10. T EST ASSUMPTION WITH PRE - PROGRAM DATA ? REJECT counterfactual hypothesis of same pregnancy rates

11. III. D IFFERENCE - IN -D IFFERENCE IDENTIFICATION STRATEGY Counterfactual: 1.Non-participant rate of pregnancy, purging pre-program differences in participants/nonparticipants 2.“Before” rate of pregnancy, purging before-after change for nonparticipants 1 and 2 are equivalent

12. Average rate of teen pregnancy in 20082012 Difference (2008-2012) Participants (P)62.9066.373.47 Non-participants (NP)46.3757.5011.13 Difference (P=NP)16.538.87 -7.66 III. D IFFERENCE - IN -D IFFERENCE IDENTIFICATION STRATEGY

13. 57.50 - 46.37 = 11.13 66.37 – 62.90 = 3.47 Non-participants Participants Effect = 3.47 – 11.13 = - 7.66 III. D IFFERENCE - IN -D IFFERENCE (1) Nonparticipants purging before/after

14. After Before Effect = 8.87 – 16.53 = - 7.66 66.37 – 57.50 = 8.87 62.90 – 46.37 = 16.53 III. D IFFERENCE - IN -D IFFERENCE (2) Before purging participation

15. 74.0 16.5 III. D IFFERENCE - IN -D IFFERENCE

16. 74.0 -7.6 III. D IFFERENCE - IN -D IFFERENCE

17. C OUNTERFACTUAL ASSUMPTION : Question: why might participants’ trends differ from that of nonparticipants? Without intervention participants and nonparticipants’ pregnancy rates follow same trends

18. T EST ASSUMPTION WITH PRE - PROGRAM DATA Average rate of teen pregnancy in 20042008 Difference (2004-2008) Participants (P)54.9662.907.94 Non-participants (NP) 39.9646.376.41 Difference (P=NP)15.0016.53 +1.53 ? REJECT counterfactual hypothesis of same trends

19. IV. M ATCHING WITH D IFFERENCE - IN - D IFFERENCE IDENTIFICATION STRATEGY Counterfactual: Comparison group is constructed by pairing each program participant with a “similar” nonparticipant Minimize differences in the vector of observed characteristics between participant and nonparticipant Parametrically (propensity score matching) Nonparametrically

20. C OUNTERFACTUAL ASSUMPTION Question: how might participant differ from matched nonparticipant? Unobserved characteristics do not affect outcomes of interest

21. 73.36 66.37 Matched nonparticipant Participant Effect = - 7.01 C OUNTERFACTUAL ASSUMPTION

22. T EST ASSUMPTION WITH EXPERIMENTAL DATA REJECT counterfactual hypothesis Meta-analysis of studies using experimental data to estimate bias in impact estimates using matching: unobservables matter! direction of bias is unpredictable!

23. V. R EGRESSION DISCONTINUITY IDENTIFICATION STRATEGY Applicability: When strict quantitative criteria determine eligibility Counterfactual: Nonparticipants just below the eligibility cutoff are the comparison for participants just above the eligibility cutoff

24. C OUNTERFACTUAL ASSUMPTION Question: Is the distribution around the cutoff smooth? Then, assumption is reasonable However, can only estimate impact around the cutoff, not for the whole program Nonparticipants just below the eligibility cutoff are the same as participants just above the eligibility cutoff

25. E XAMPLE : E FFECT OF SCHOOL INPUTS ON TEST SCORES Target transfer to poorest schools Construct poverty index from 1 to 100 Schools with a score <=50 are in Schools with a score >50 are out Inputs transfer to poor schools Measure outcomes (i.e. test scores) before and after transfer

27. Non-Poor Poor

29. Treatment Effect

30. S UMMARY Gold standard is randomization – no assumptions needed, always precise estimates Non-experimental requires assumptions – can you live with them? We did not cover: – Encouragement design – Instrumental variables – Pipeline comparisons