Econometrics with Observational Data: Research Design Todd Wagner

Research Design Goal: evaluate behaviors and identify causation Goal: evaluate behaviors and identify causation –Policy X caused effect Y –Medication A resulted in B hospitalizations Unit of analysis can be individual or organizational Unit of analysis can be individual or organizational

Research Methods Random assignment? Intent to Treat? Yes

Research Methods Random assignment? Intent to Treat? Yes Basic RCT Analysis Yes No On Treatment

On Treatment RCT comparing drug A to drug B RCT comparing drug A to drug B Adherence for drugs Adherence for drugs –A is 70% –B is 40% What does a comparison of A versus B tell us? What does a comparison of A versus B tell us?

Research Methods Random assignment? Intent to Treat? Is there a control group? Yes No

Research Methods Is there random assignment Randomized Trial Is there a control group Quasi-experimental Design Descriptive Study

Research Methods Is there random assignment Randomized Trial Is there a control group Quasi-experimental Design Descriptive Study

Quasi-Experimental Designs Difference-in-differences Difference-in-differences Regression discontinuity Regression discontinuity Switching replications Switching replications Non-equivalent dependent variables Non-equivalent dependent variables Most common In health

Difference-in-Differences AKA: DD, D in D, or Diff in Diff AKA: DD, D in D, or Diff in Diff Differences across time and arms Differences across time and arms –Usually two arms: treatments, controls –In theory can be used with 3+ arms

Methods for Identifying Controls Inherent matching: Find similar individuals not getting treatment to serve as controls (e.g., twins) Inherent matching: Find similar individuals not getting treatment to serve as controls (e.g., twins) Statistical: use statistical techniques to identify best comparison groups Statistical: use statistical techniques to identify best comparison groups Location: use other geographic sites, states or regions as controls Location: use other geographic sites, states or regions as controls

Unit of Analysis D in D works for different units of analysis D in D works for different units of analysis –Person–people followed over time –Site– sites followed over time –State– states followed over time May need to make some analytical changes depending on unit of analysis May need to make some analytical changes depending on unit of analysis

Diff in Diff example Gruber, Adams and Newhouse (1997) Gruber, Adams and Newhouse (1997) Tennessee increased Medicaid fees for primary care services (goal encourage office care; decrease hospital-based ambulatory care) Tennessee increased Medicaid fees for primary care services (goal encourage office care; decrease hospital-based ambulatory care) What is the effect of this policy change? What is the effect of this policy change?

Research Designs Difference-in-differences Difference-in-differences Regression discontinuity Regression discontinuity Switching replications Switching replications Nonequivalent dependent variables Nonequivalent dependent variables

Regression Discontinuity Participants are assigned to program or comparison groups solely on the basis of an observed measure (education test or means test) Participants are assigned to program or comparison groups solely on the basis of an observed measure (education test or means test) Appropriate when we wish to target a program or treatment to those who most need or deserve it Appropriate when we wish to target a program or treatment to those who most need or deserve it

Regression Discontinuity Partial coverage (not everyone gets the treatment) Partial coverage (not everyone gets the treatment) Requires the selection mechanism to be fully known Requires the selection mechanism to be fully known Selection mechanism must be consistently applied to all persons Selection mechanism must be consistently applied to all persons

RD Design Graphically Source: Urban Institute Threshold MUST be known and consistently applied Test for significance

Research Designs Difference-in-differences Difference-in-differences Regression discontinuity Regression discontinuity Switching replications Switching replications Nonequivalent dependent variables Nonequivalent dependent variables

Switching Replications Has two groups and three waves of measurement Has two groups and three waves of measurement AKA waitlist control group AKA waitlist control group This design is sometimes used in randomized trials This design is sometimes used in randomized trials

Example from Pap Smear Study treat 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% > 12 Months since Initial Pap Cumulative % Followed Up InterventionControl Immediate treatment delayed treatment

Research Designs Difference-in-differences Difference-in-differences Regression discontinuity Regression discontinuity Switching replications Switching replications Nonequivalent dependent variables Nonequivalent dependent variables

Non-Equivalent DVs Analyze dependent variable that should not be affected by the intervention Analyze dependent variable that should not be affected by the intervention Example: Intervention is designed to affect quality of diabetes care, but could also see if intervention affected quality of asthma care Example: Intervention is designed to affect quality of diabetes care, but could also see if intervention affected quality of asthma care

Notes on the Analysis of DD data

Analytical Methods Plot or graph unadjusted data Plot or graph unadjusted data Graduate to more complex models Graduate to more complex models Address, if possible, model limitations Address, if possible, model limitations

DD Raw Data Standard deviations in parentheses +DD = (Exp followup - Exp baseline )-(Control followup - Control baseline ) † unadjusted estimates Baseline 1-Year Follow-Up Exp. Control Exp Control DD Utilization Entry (% yes) 84.5% 86.1% 88.9% 86.8% 3.7 (36.2) (34.6) (31.4) (33.9) No. of visits (0-16) (4.28) (4.36) (4.00) (4.07)

Diff n Diff Model Y =  +   G +   T +   GT+  X +  Y=outcome G = group (0=control, 1=treatment) T= time (0=baseline, 1=follow-up) X = characteristics of person, place, etc.  = error term

Program Effect  f  3 = 0 then the program has no effect  f  3 = 0 then the program has no effect Limited statistical power. Testing interactions increases risk of type 2 error. Limited statistical power. Testing interactions increases risk of type 2 error. Outcome =  +  1 G +  2 T +  3 GT +  X + 

avgcost sta3n exp yr_d year Note: Data Listed in Stata Organizing the Dataset

Identification How do you obtain an unbiased estimate of  3 ? How do you obtain an unbiased estimate of  3 ? For an unbiased estimate of GT, G must not be correlated with  that is, G must be exogenous For an unbiased estimate of GT, G must not be correlated with  that is, G must be exogenous Outcome =  +  1 G +  2 T +  3 GT +  X + 

Identification G may be endogenous G may be endogenous Selection bias Selection bias –Selection bias is type of endogeneity –Caused by non-random assignment –Outcome and G (group) affect each other -- causality runs both ways –Impact:  3 is biased Outcome =  +  1 G +  2 T +  3 GT +  X + 

Wagner, T. H., & Chen, S. (2005). An economic evaluation of inpatient residential treatment programs in the department of veterans affairs. Med Care Res Rev, 62(2), Example: VA Residential Treatment

Residential Treatment Programs RTPs provide mental health and substance use treatment RTPs provide mental health and substance use treatment RTPs were designed to RTPs were designed to –treat eligible veterans in a less- intensive and more self-reliant setting. –to provide cost-effective care that “promotes independence and fosters responsibility.”

Objectives 1. Did the RTPs save money? 2. Were savings a “one-time” event or do they continue to accrue?

Design Choice Selection mechanism is not observed– can’t use regression discontinuity Selection mechanism is not observed– can’t use regression discontinuity We know who adopted RTP and when– DD is feasible We know who adopted RTP and when– DD is feasible

Methods Built a longitudinal dataset for for all VA medical centers Built a longitudinal dataset for for all VA medical centers Tracked approved RTP programs (N=43) Tracked approved RTP programs (N=43) We merged data from the PTF and CDR to track We merged data from the PTF and CDR to track –Total MH inpatient days (PTF) and dollars (CDR) –Total SA inpatient days (PTF) and dollars (CDR)

Outcomes Department-level costs Department-level costs –Average cost per MH day –Average cost per SA day –Total MH/SA department costs Sensitivity analysis Sensitivity analysis –Outpatient MH/SA costs –FTE

Multivariate models Fixed-effects models 1 Fixed-effects models 1 –DV: Department-level costs –Controlled for medical center size –Inflation adjusted to 1999 using CPI –Year dummies –Wage index 1 Random effects were similar; Hausman tests were not significant. Fixed effects were more conservative.

Results: Mental Health Average cost savings of $81 per day (p<0.01). Average cost savings of $81 per day (p<0.01). Savings do not appear to be increasing over time. Savings do not appear to be increasing over time.

Mental Health Costs

Results: Substance Abuse Average cost savings of $112 per day (p<0.01). Average cost savings of $112 per day (p<0.01). Savings do not appear to be increasing over time. Savings do not appear to be increasing over time.

Mental Health Costs

Sensitivity Analysis RTPs were associated with a slight decrease in the costs of outpatient psychiatry. RTPs were associated with a slight decrease in the costs of outpatient psychiatry. RTPs were associated with a decrease in FTE RTPs were associated with a decrease in FTE

Limitations Not clear if RTPs could be better– are they treating the right patient? Not clear if RTPs could be better– are they treating the right patient? Endogeneity of RTPs Endogeneity of RTPs –1 and 2 year lags (medical centers with RTPs in 1994 and 1995) are not associated with costs –There does not appear to be self-selection in RTPs.

Any Questions?

Design References Trochim, W. Research Methods Knowledge Database Rossi, PH, and HE Freeman. Evaluation: A systematic approach. 5th ed. New York: Sage, 1993.

Regression References Wm. Greene. Econometric Analysis. Wm. Greene. Econometric Analysis. J Wooldridge. Econometric Analysis of Cross Section and Panel Data. J Wooldridge. Econometric Analysis of Cross Section and Panel Data.

You’ve Almost Made It June11 th Mark Smith, Endogeneity June11 th Mark Smith, Endogeneity TBA: Todd Wagner: Using Stata TBA: Todd Wagner: Using Stata