New research methods for the evaluation of policy changes Sanjay Basu, MD, PhD O LD P ROBLEMS, N EW S OLUTIONS

Three new methods to discuss If you have individual-level data, but an imperfect control group: Near-far matching If you have population-level data, but an imperfect control group: Synthetic control analysis If you have either type of data, and want to estimate disparities: Distributional decomposition

Example: Is the school meal program “worsening child health”? In a recent analysis:  “Rural children in the meals program had a significantly higher probability of being stunted than those not in the program, even after controlling for income differences.” (IIPS, 2014)

Our typical solutions Propensity score matching Problem: unobserved confounders

Our typical solutions An instrumental variable  Problem of “weak” instruments

Method: near-far matching Baiocchi, et al., Health Serv Outcomes Res Method 2012

Re-analysis of food programs Re-analysis of India school mean program and stunting  Without matching: OR = 1.28 (1.12,1.44)  With propensity score matching: OR= 0.96 (0.80, 1.12)  With near-far matching: OR = 0.84 (0.70,0.98) For a worked example, see: Lorch et al, Pediatrics, 2012

The ‘individualistic fallacy’ Some of our policies are designed to focus on a population-level outcome, not just an individual- level one And many of our most interesting policies are ‘case studies’ of one group performing a policy, with no natural ‘control group’

Typical solution: difference-in-differences analysis

Example of synthetic control For worked example, see Abadie et al., Am J Pol Sci, 2014 In Stata: ssc install synth In R: install.packages("Synth")

Decomposition As compared to standard regression

Distributional decomposition For proof and worked example, see Basu et al., Am J Epi, 2015 In Stata: download distdecomp package from sdr.stanford.edu

References Near-far matching  Mike Baiocchi,  See: Baiocchi, et al., Health Serv Outcomes Res Method 2012 Synthetic control  Jens Hainmueller,  See: Abadie et al., Am J Pol Sci, 2014 Distributional decomposition  Sanjay Basu,  See: Basu et al., AJE (in press), 2015

Additional slides

Regression discontinuity Advantages: only post-policy data needed Disadvantages: people can ‘cheat’ only informs the margins