1. New research methods for the evaluation of policy changes Sanjay Basu, MD, PhD basus@stanford.edu O LD P ROBLEMS, N EW S OLUTIONS

2. Three new methods to discuss If you have individual-level data, but an imperfect control group: Near-far matching If you have population-level data, but an imperfect control group: Synthetic control analysis If you have either type of data, and want to estimate disparities: Distributional decomposition

3. Example: Is the school meal program “worsening child health”? In a recent analysis:  “Rural children in the meals program had a significantly higher probability of being stunted than those not in the program, even after controlling for income differences.” (IIPS, 2014)

4. Our typical solutions Propensity score matching Problem: unobserved confounders

5. Our typical solutions An instrumental variable  Problem of “weak” instruments

6. Method: near-far matching Baiocchi, et al., Health Serv Outcomes Res Method 2012

7. Re-analysis of food programs Re-analysis of India school mean program and stunting  Without matching: OR = 1.28 (1.12,1.44)  With propensity score matching: OR= 0.96 (0.80, 1.12)  With near-far matching: OR = 0.84 (0.70,0.98) For a worked example, see: Lorch et al, Pediatrics, 2012

8. The ‘individualistic fallacy’ Some of our policies are designed to focus on a population-level outcome, not just an individual- level one And many of our most interesting policies are ‘case studies’ of one group performing a policy, with no natural ‘control group’

9. Typical solution: difference-in-differences analysis

10. Example of synthetic control For worked example, see Abadie et al., Am J Pol Sci, 2014 In Stata: ssc install synth In R: install.packages("Synth")

11. Decomposition As compared to standard regression

12. Distributional decomposition For proof and worked example, see Basu et al., Am J Epi, 2015 In Stata: download distdecomp package from sdr.stanford.edu

13. References Near-far matching  Mike Baiocchi, http://web.stanford.edu/~baiocchi/http://web.stanford.edu/~baiocchi/  See: Baiocchi, et al., Health Serv Outcomes Res Method 2012 Synthetic control  Jens Hainmueller, http://web.stanford.edu/~jhain/http://web.stanford.edu/~jhain/  See: Abadie et al., Am J Pol Sci, 2014 Distributional decomposition  Sanjay Basu, http://web.stanford.edu/~basus/http://web.stanford.edu/~basus/  See: Basu et al., AJE (in press), 2015

14. Additional slides

15. Regression discontinuity Advantages: only post-policy data needed Disadvantages: people can ‘cheat’ only informs the margins