Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics, Yale Faculty Research Fellow, NBER Visiting Associate Professor, SIEPR October 2015

Experiments Address Internal Validity, But What About External Validity? Explosion in use of experiments Are they externally valid? Quandary: what to do if the results from randomized experiments do not conform to results of seemingly convincing natural experiments? Policy-relevant example: Oregon Health Insurance Experiment vs. Massachusetts Health Reform My results show that there may not be a conflict after all…

Oregon Health Insurance Experiment (OHIE) Shows ER Visits Increased Do these results generalize to other settings? Massachusetts Health Reform showed – ER visits decreased Miller (2012), Smulowitz et al. (2011) – ER visits stayed the same Chen, Scheffler, Chandra (2011) – Admissions from ER (proxy for ER visits) decreased Kolstad & Kowalski (2012)

I Apply MTE Methods to Examine External Validity of the OHIE Prior literature – Bjorklund and Moffitt (1987) – Heckman Vytlacil (1999, 2005, 2007) – Brinch, Mogstad, Wiswall (2015) Extended Heckman Vytlacil to allow discrete instrument MTEs in experiments (binary instruments) are now possible To my knowledge, these methods have not been applied to experiments (Brinch et al. application is to quantity-quality of children in IV framework) Key insight: unless everyone assigned to lottery gets intervention (and everyone else does not), then external validity will be affected by selection

I find that MTE of Insurance on ER is positive for some types and negative for others in Oregon Massachusetts individuals induced to gain health insurance appear more similar to the Oregon individuals that decreased ER utilization Individuals who entered a lottery for health insurance coverage in Oregon likely had a higher desire to use the ER than individuals who gained coverage after Massachusetts mandate MTE methods and related diff-in-diff test of external validity will be a valuable addition to the standard toolkit for analysis of experiments

Outline Model Marginal Treatment Effects New Tests of Internal and External Validity Bringing MTE to OHIE Results Bringing Oregon MTE to Massachusetts Treatment Effect Heterogeneity Experimental Design for External Validity Conclusions

Selection into treatment given lottery is key model element

Outline Model Marginal Treatment Effects New Tests of Internal and External Validity Bringing MTE to OHIE Results Bringing Oregon MTE to Massachusetts Treatment Effect Heterogeneity Experimental Design for External Validity Conclusions

MTE is treatment effect for individuals indifferent to treatment Can recover LATE from MTE:

The closed form expression of the linear MTE and the LATE

Can identify linear MTE from standard LATE assumptions, nonlinear MTE requires additional separability Assumption 1 sufficient for linear MTE – Standard LATE conditional independence and monotonicity – Key is additional moment: separation of compliers from always takers and never takers Assumption 2 yields nonlinear MTE – Additive separability between observed and unobserved heterogeneity often imposed in applied work

Outline Model Marginal Treatment Effects New Tests of Internal and External Validity Bringing MTE to OHIE Results Bringing Oregon MTE to Massachusetts Treatment Effect Heterogeneity Experimental Design for External Validity Conclusions

Compliers, Always Takers, Never Takers Standard formulas for average characteristics Standard formulas Always takers: D=1 regardless of Z=1 or Z=0 Compliers: D=1 if Z=1 but D=0 if Z=0 Never takers:D=0 regardless of Z=1 or Z=0 Defiers: D=1 if Z=0 but D=0 if Z=1 (not present by assumption)

New Test of Internal Validity

Test of External Validity

Bootstrapped standard error of the slope = To obtain the bootstrapped standard errors, we bootstrap for 200 replications and use the standard deviation of the replications as an estimate of the standard errors.

Bootstrapped standard error of the slope = To obtain the bootstrapped standard errors, we bootstrap for 200 replications and use the standard deviation of the replications as an estimate of the standard errors.

The Geometry of the LATE and MTE Always takers and treated compliers Never takers and control compliers

The Geometry of the LATE and MTE Always takers and treated compliers Never takers and control compliers

The Geometry of the LATE and MTE

Outline Model Marginal Treatment Effects New Tests of Internal and External Validity Bringing MTE to OHIE Results Bringing Oregon MTE to Massachusetts Treatment Effect Heterogeneity Experimental Design for External Validity Conclusions

Bringing MTE to OHIE Control for lottery entrants matters Otherwise, replicate well with covariates Results for number of visits similarnumber of visits

Outline Model Marginal Treatment Effects New Tests of Internal and External Validity Bringing MTE to OHIE Results Bringing Oregon MTE to Massachusetts Treatment Effect Heterogeneity Experimental Design for External Validity Conclusions

MTE Results

MTE Weights

Weighted MTE

Outline Model Marginal Treatment Effects New Tests of Internal and External Validity Bringing MTE to OHIE Results Bringing Oregon MTE to Massachusetts Treatment Effect Heterogeneity Experimental Design for External Validity Conclusions

Bringing Oregon to Massachusetts

Propensity Scores in MA

Propensity Scores in MA Compliers Only, Rescaled

Massachusetts Weights

Oregon MTE/Massachusetts Weights Negative ER Impacts Plausible in MA

Outline Model Marginal Treatment Effects New Tests of Internal and External Validity Bringing MTE to OHIE Results Bringing Oregon MTE to Massachusetts Treatment Effect Heterogeneity Experimental Design for External Validity Conclusions

Treatment Effect Heterogeneity Binary Subgroup Analysis Point estimates all positive except non-English speakers

Treatment Effect Heterogeneity Multivariate Subgroup Analysis Many negative point estimates

Who Are the People with Reductions in ER Utilization? 38

Outline Model Marginal Treatment Effects New Tests of Internal and External Validity Bringing MTE to OHIE Results Bringing Oregon MTE to Massachusetts Treatment Effect Heterogeneity Experimental Design for External Validity Conclusions

Experimental Design for External Validity Collect data on treatment (endogenous variable) Collect a wide array of covariates – Use standard definitions Only stratify on covariates that can be obtained elsewhere Collect data on experimental subjects randomized out of treatment who get treatment via other means Design experiment so that always takers and never takers are possible

Lessons for Experimentalists Going to great lengths to encourage “attendance” could decrease external validity – In the limiting case, if full attendance, cannot estimate MTE Going to great lengths to encourage “attendance” could decrease estimated effect size

Outline Model Marginal Treatment Effects New Tests of Internal and External Validity Bringing MTE to OHIE Results Bringing Oregon MTE to Massachusetts Treatment Effect Heterogeneity Experimental Design for External Validity Conclusions

I find that MTE of Insurance on ER is positive for some types and negative for others in Oregon Massachusetts individuals induced to gain health insurance appear more similar to the Oregon individuals that decreased ER utilization Individuals who entered a lottery for health insurance coverage in Oregon likely had a higher desire to use the ER than individuals who gained coverage after Massachusetts mandate MTE methods and related diff-in-diff test of external validity will be a valuable addition to the standard toolkit for analysis of experiments

Appendix Slides

Appendix: Bringing OHIE to MTE Number of ER visits Return to similar results for any visitsany visits

Appendix: Formulas for Complier Characteristics Treated Compliers Control Compliers Return to results with complier characteristicswith complier characteristics

Appendix with editable slides/formulas

MTE is treatment effect for individuals indifferent to treatment Can recover LATE from MTE:

The closed form expression of the linear MTE and the LATE

The Geometry of the LATE and MTE Number of Emergency Room Visits E(Y|D=0, Z=0) E(Y|D=0, Z=1) E(Y|D=1, Z=0) E(Y|D=1, Z=1) P(D=1, Z=0) P(D=1, Z=1) p

The Geometry of the LATE and MTE Number of Emergency Room Visits E(Y|D=0, Z=0) E(Y|D=0, Z=1) E(Y|D=1, Z=0) E(Y|D=1, Z=1) P(D=1, Z=0) P(D=1, Z=1) p

The Geometry of the LATE and MTE Number of Emergency Room Visits E(Y|D=0, Z=0) E(Y|D=0, Z=1) E(Y|D=1, Z=0) E(Y|D=1, Z=1) P(D=1, Z=0) P(D=1, Z=1) p

The Geometry of the LATE and MTE LATE Number of Emergency Room Visits E(Y|D=0, Z=0) E(Y|D=0, Z=1) E(Y|D=1, Z=0) E(Y|D=1, Z=1) P(D=1, Z=0) P(D=1, Z=1) p