1. Marginal Treatment Effects and the External Validity of the Oregon Health Insurance Experiment Amanda Kowalski Associate Professor, Department of Economics, Yale Faculty Research Fellow, NBER Visiting Associate Professor, SIEPR October 2015

2. Experiments Address Internal Validity, But What About External Validity? Explosion in use of experiments Are they externally valid? Quandary: what to do if the results from randomized experiments do not conform to results of seemingly convincing natural experiments? Policy-relevant example: Oregon Health Insurance Experiment vs. Massachusetts Health Reform My results show that there may not be a conflict after all…

3. Oregon Health Insurance Experiment (OHIE) Shows ER Visits Increased Do these results generalize to other settings? Massachusetts Health Reform showed – ER visits decreased Miller (2012), Smulowitz et al. (2011) – ER visits stayed the same Chen, Scheffler, Chandra (2011) – Admissions from ER (proxy for ER visits) decreased Kolstad & Kowalski (2012)

4. I Apply MTE Methods to Examine External Validity of the OHIE Prior literature – Bjorklund and Moffitt (1987) – Heckman Vytlacil (1999, 2005, 2007) – Brinch, Mogstad, Wiswall (2015) Extended Heckman Vytlacil to allow discrete instrument MTEs in experiments (binary instruments) are now possible To my knowledge, these methods have not been applied to experiments (Brinch et al. application is to quantity-quality of children in IV framework) Key insight: unless everyone assigned to lottery gets intervention (and everyone else does not), then external validity will be affected by selection

5. I find that MTE of Insurance on ER is positive for some types and negative for others in Oregon Massachusetts individuals induced to gain health insurance appear more similar to the Oregon individuals that decreased ER utilization Individuals who entered a lottery for health insurance coverage in Oregon likely had a higher desire to use the ER than individuals who gained coverage after Massachusetts mandate MTE methods and related diff-in-diff test of external validity will be a valuable addition to the standard toolkit for analysis of experiments

6. Outline Model Marginal Treatment Effects New Tests of Internal and External Validity Bringing MTE to OHIE Results Bringing Oregon MTE to Massachusetts Treatment Effect Heterogeneity Experimental Design for External Validity Conclusions

7. Selection into treatment given lottery is key model element

8. Outline Model Marginal Treatment Effects New Tests of Internal and External Validity Bringing MTE to OHIE Results Bringing Oregon MTE to Massachusetts Treatment Effect Heterogeneity Experimental Design for External Validity Conclusions

9. MTE is treatment effect for individuals indifferent to treatment Can recover LATE from MTE:

10. The closed form expression of the linear MTE and the LATE

11. Can identify linear MTE from standard LATE assumptions, nonlinear MTE requires additional separability Assumption 1 sufficient for linear MTE – Standard LATE conditional independence and monotonicity – Key is additional moment: separation of compliers from always takers and never takers Assumption 2 yields nonlinear MTE – Additive separability between observed and unobserved heterogeneity often imposed in applied work

12. Outline Model Marginal Treatment Effects New Tests of Internal and External Validity Bringing MTE to OHIE Results Bringing Oregon MTE to Massachusetts Treatment Effect Heterogeneity Experimental Design for External Validity Conclusions

13. Compliers, Always Takers, Never Takers Standard formulas for average characteristics Standard formulas Always takers: D=1 regardless of Z=1 or Z=0 Compliers: D=1 if Z=1 but D=0 if Z=0 Never takers:D=0 regardless of Z=1 or Z=0 Defiers: D=1 if Z=0 but D=0 if Z=1 (not present by assumption)

14. New Test of Internal Validity

15. Test of External Validity

16. Bootstrapped standard error of the slope = 0.104 To obtain the bootstrapped standard errors, we bootstrap for 200 replications and use the standard deviation of the replications as an estimate of the standard errors.

17. Bootstrapped standard error of the slope = 0.572 To obtain the bootstrapped standard errors, we bootstrap for 200 replications and use the standard deviation of the replications as an estimate of the standard errors.

18. The Geometry of the LATE and MTE Always takers and treated compliers Never takers and control compliers

19. The Geometry of the LATE and MTE Always takers and treated compliers Never takers and control compliers

20. The Geometry of the LATE and MTE

22. Outline Model Marginal Treatment Effects New Tests of Internal and External Validity Bringing MTE to OHIE Results Bringing Oregon MTE to Massachusetts Treatment Effect Heterogeneity Experimental Design for External Validity Conclusions

23. Bringing MTE to OHIE Control for lottery entrants matters Otherwise, replicate well with covariates Results for number of visits similarnumber of visits

24. Outline Model Marginal Treatment Effects New Tests of Internal and External Validity Bringing MTE to OHIE Results Bringing Oregon MTE to Massachusetts Treatment Effect Heterogeneity Experimental Design for External Validity Conclusions

25. MTE Results

27. MTE Weights

28. Weighted MTE

29. Outline Model Marginal Treatment Effects New Tests of Internal and External Validity Bringing MTE to OHIE Results Bringing Oregon MTE to Massachusetts Treatment Effect Heterogeneity Experimental Design for External Validity Conclusions

30. Bringing Oregon to Massachusetts

31. Propensity Scores in MA

32. Propensity Scores in MA Compliers Only, Rescaled

33. Massachusetts Weights

34. Oregon MTE/Massachusetts Weights Negative ER Impacts Plausible in MA

35. Outline Model Marginal Treatment Effects New Tests of Internal and External Validity Bringing MTE to OHIE Results Bringing Oregon MTE to Massachusetts Treatment Effect Heterogeneity Experimental Design for External Validity Conclusions

36. Treatment Effect Heterogeneity Binary Subgroup Analysis Point estimates all positive except non-English speakers

37. Treatment Effect Heterogeneity Multivariate Subgroup Analysis Many negative point estimates

38. Who Are the People with Reductions in ER Utilization? 38

39. Outline Model Marginal Treatment Effects New Tests of Internal and External Validity Bringing MTE to OHIE Results Bringing Oregon MTE to Massachusetts Treatment Effect Heterogeneity Experimental Design for External Validity Conclusions

40. Experimental Design for External Validity Collect data on treatment (endogenous variable) Collect a wide array of covariates – Use standard definitions Only stratify on covariates that can be obtained elsewhere Collect data on experimental subjects randomized out of treatment who get treatment via other means Design experiment so that always takers and never takers are possible

41. Lessons for Experimentalists Going to great lengths to encourage “attendance” could decrease external validity – In the limiting case, if full attendance, cannot estimate MTE Going to great lengths to encourage “attendance” could decrease estimated effect size

42. Outline Model Marginal Treatment Effects New Tests of Internal and External Validity Bringing MTE to OHIE Results Bringing Oregon MTE to Massachusetts Treatment Effect Heterogeneity Experimental Design for External Validity Conclusions

43. I find that MTE of Insurance on ER is positive for some types and negative for others in Oregon Massachusetts individuals induced to gain health insurance appear more similar to the Oregon individuals that decreased ER utilization Individuals who entered a lottery for health insurance coverage in Oregon likely had a higher desire to use the ER than individuals who gained coverage after Massachusetts mandate MTE methods and related diff-in-diff test of external validity will be a valuable addition to the standard toolkit for analysis of experiments

44. Appendix Slides

45. Appendix: Bringing OHIE to MTE Number of ER visits Return to similar results for any visitsany visits

46. Appendix: Formulas for Complier Characteristics Treated Compliers Control Compliers Return to results with complier characteristicswith complier characteristics

47. Appendix with editable slides/formulas

48. MTE is treatment effect for individuals indifferent to treatment Can recover LATE from MTE:

49. The closed form expression of the linear MTE and the LATE

50. The Geometry of the LATE and MTE 00.10.2 0.30.40.50.60.70.80.9 1 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 Number of Emergency Room Visits E(Y|D=0, Z=0) E(Y|D=0, Z=1) E(Y|D=1, Z=0) E(Y|D=1, Z=1) P(D=1, Z=0) P(D=1, Z=1) p

51. The Geometry of the LATE and MTE 00.10.2 0.30.40.50.60.70.80.9 1 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 Number of Emergency Room Visits E(Y|D=0, Z=0) E(Y|D=0, Z=1) E(Y|D=1, Z=0) E(Y|D=1, Z=1) P(D=1, Z=0) P(D=1, Z=1) p

52. The Geometry of the LATE and MTE 00.10.2 0.30.40.50.60.70.80.9 1 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 Number of Emergency Room Visits E(Y|D=0, Z=0) E(Y|D=0, Z=1) E(Y|D=1, Z=0) E(Y|D=1, Z=1) P(D=1, Z=0) P(D=1, Z=1) p

53. The Geometry of the LATE and MTE 00.10.2 0.30.40.50.60.70.80.9 1 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 LATE Number of Emergency Room Visits E(Y|D=0, Z=0) E(Y|D=0, Z=1) E(Y|D=1, Z=0) E(Y|D=1, Z=1) P(D=1, Z=0) P(D=1, Z=1) p