Average Treatment Effects and Propensity Score Methods Zhehui Luo, MS, PhD Department of Epidemiology Michigan State University

What experimental data to use? Treatment group Control group Both treatment and control groups Where to draw comparison sample? Specifications for propensity scores (balancing tests) Mean difference by p-score strata Standardized differences Hotelling’s T 2 tests of joint null Regression based tests Matching methods Nearest Neighbor With replacement Without replacement Seed for the random number generator Kernel/local linearBandwidth CaliperRadius Bias estimation Differentials b/t treatment and comparison against experimental benchmark Differentials b/t control and comparison against zero Stratification

Cognitive Behavioral Intervention on reducing symptom severity during chemotherapy for cancer patients, – CBI Cognitive Behavioral Intervention on reducing symptom severity during chemotherapy for cancer patients, – CBI Problem Solving Therapy on reducing symptom severity during chemotherapy for cancer patients, – WCI Problem Solving Therapy on reducing symptom severity during chemotherapy for cancer patients, – WCI Community Based Family Care Study, , short term impact of cancer treatments on newly diagnosed patients – FCS Community Based Family Care Study, , short term impact of cancer treatments on newly diagnosed patients – FCS Data provided by research supported by the following: Given, B., Given, C.W., McCorkle, R. Family Home Care for Cancer, a community-based model. Grant #R01 CA79280, funded by NCI, NINR, NIA P30 AG08808, in collaboration with Walther Cancer Institute, IN Data provided by research supported by the following: Given, B., Given, C.W., McCorkle, R. Family Home Care for Cancer, a community-based model. Grant #R01 CA79280, funded by NCI, NINR, NIA P30 AG08808, in collaboration with Walther Cancer Institute, IN

CBI treatment group CBI control group FCS all WCI allCBI all WCI treatment group WCI control group FCS-1 FCS-2 FCS-3 FCS-1: Undergoing Chemo Therapy FCS-2: Advanced Cancer Stage FCS-3: Advanced and Undergoing Chemo Therapy

Table 1 Characteristics of CBI, WCI and FCS patients (4.79) (5.12) (4.79) (5.15) (11.65) (13.06) (11.18) (9.80) Age (std) % Late Stage % Lung Cancer % Colon Cancer % Breast Cancer 85 (59.0%) 157 (60.2%) 206 (70.6%) 678 (76.4%) 46 (74.2%) 38 (61.3%) 86 (72.9%) 79 (67.5%) 20-week Baseline N FCS-3FCS-2FCS-1FCS- All WCI- Con WCI- Exp CBI- Con CBI- Exp

Table 1 continued (13.94) (15.32) (16.09) (15.20) (19.20) (13.94) (19.21) (16.38) SF-36 MH 20-week (15.28) (15.99) (15.78) (15.34) (17.77) (19.16) (18.63) (17.25) SF-36 MH Baseline (30.38) (30.48) (29.01) (26.59) (23.99) (27.35) (30.23) (22.90) SF-36 PF 20-week (29.00) (30.05) (28.37) (28.79) (27.27) (27.38) (29.98) (28.59) SF-36 PF Baseline FCS-3FCS-2 FCS-1FCS- All WCI- Con WCI- Exp CBI- Con CBI- Exp

Table 2 Testing between 10 composite samples Note: Yellow cells indicate significant differences at 0.05 level for baseline variables. Light green cells indicate significant differences at 0.05 or 0.1 levels at 20-week.

Table 3 Treatment effects on SF-36 Physical Functioning, using CBI-treated group and various comparisons 4.23 (4.47)-7.70 (3.07)**-5.47 (4.65)5.64 (3.76)7.33 (3.29)**match, n(4) (11.90)-5.21 (5.67)33.13 (13.59)** (12.99)**34.52 (12.21)*** psmatch2, n(4) (25.66) (26.44)70.77 (32.38)**55.77 (27.94)*25.77 (25.97)psmatch2, n(1) (5.33)***15.35 (8.76)*6.72 (16.21)12.64 (8.73)9.05 (9.32)Attk (0.06) (5.04)***34.90 (15.45)**7.26 (15.23)4.97 (8.25)6.98 (7.37)atts (6.24)**35.29 (21.69)-7.13 (19.17)48.16 (28.77)*-8.01 (16.05)attnd (5.68)**3.54 (4.96)7.33 (8.34)6.05 (6.57)7.51 (6.54)Model 1 | p 8.22 (4.56)*-0.44 (3.24)-0.93 (5.74)-2.48 (4.74)4.53 (4.21)Model (3.97)***-1.14 (2.63)6.64 (3.94)*1.53 (3.40)3.20 (3.05)Model (4.62)***1.84 (3.17)12.40 (4.32)**6.14 (3.90)6.95 (3.64)*Model 1 WCI-conFCS-allFCS-3FCS-2FCS-1 Non- experimental Estimates 7.17 (3.81)*5.17 (3.87)8.70 (4.30)** Model 3Model 2Model 1 Experimental Estimates * significant at 10%; ** significant at 5%; *** significant at 1%

Table 4 Treatment effects on SF-36 Mental Health, using CBI-treated group and various comparisons (3.81)12.08 (2.02)***7.41 (2.64)***3.41 (2.38)7.12 (2.37)***match, n(4) (10.85) (7.43)-3.32 (8.05)1.68 (8.60)-4.32 (8.14)psmatch2, n(4) (20.38) (14.41)7.68 (15.93)11.68 (15.18) (19.10)psmatch2, n(1) (6.44)8.99 (3.04)***6.38 (5.75)9.66 (4.87)*7.55 (5.15)attk 1.11 (4.23)9.88 (2.52)***8.54 (4.80)*7.49 (3.84)*7.96 (3.18)**atts (6.07)10.96 (5.58)*6.76 (3.75)*17.00 (7.60)**5.78 (5.34)attnd (4.14)7.93 (2.91)***9.66 (5.27)*7.46 (4.02)*8.09 (4.03)**Model 1 | p (4.20)2.97 (2.35)-0.15 (4.27)0.73 (3.46)4.72 (3.22)Model (3.59)1.23 (1.84)0.04 (2.75)0.73 (2.36)2.05 (2.21)Model (3.61)1.85 (1.86)1.74 (2.74)2.00 (2.38)2.91 (2.34)Model 1 WCI-conFCS-allFCS-3FCS-2FCS-1 Non- experimental Estimates 1.97 (2.78)2.50 (2.66)3.02 (3.00) Model 3Model 2Model 1 Experimental Estimates * significant at 10%; ** significant at 5%; *** significant at 1%

Concluding Remarks Cautionary tale Cautionary tale Comparison Samples Comparison Samples Stata Commands Stata Commands Specification Tests Specification Tests

Questions to the audience Why did psmatch2 produce such drastically different results? Why did psmatch2 produce such drastically different results? Why should differences between estimated ATE using various econometric techniques and ATE from randomized trial be viewed as specification errors in econometric methods and not as selection bias in the experimental results? Why should differences between estimated ATE using various econometric techniques and ATE from randomized trial be viewed as specification errors in econometric methods and not as selection bias in the experimental results?

Pre-treatment Treated group, with treatment Treated group, without treatment Control group, w/o treatment Comparison group Post-treatment