1. 1 Arlene Ash QMC - Third Tuesday September 21, 2010 Analyzing Observational Data: Focus on Propensity Scores

2. 2 The Problem Those with the intervention and those without have markedly different values for important measured risk factors & Outcome is related to the risk factors that are imbalanced between the groups & It is not clear how the risk factors and outcome are related Why may standard analyses be misleading?

3. 3 True and Modeled Relationship Between Risk and Outcome

4. 4 Is Imbalance in Risk a Problem? If we correctly model the relationship between risk factors and outcome, we correctly estimate effect of the intervention With many risk factors, hard to know if the relationship between risk factors and outcome is correctly modeled Propensity score - a way to reduce the effect of imbalance in measured risk when models may be inadequate

5. 5 Propensity Score Method (Key Idea) Draw a sub-sample that is more balanced on risk factors Use standard techniques to analyze the sub- sample

6. 6 Typical Propensity Score Approach Estimate a model to predict the “probability of receiving the intervention” – This is “the propensity score” Divide the full population into quintiles of the propensity score Sample equal numbers of cases and controls from each quintile Re-fit the model to estimate the effect of the intervention in the sampled cases

7. 7 Propensity Score Sampling Example PS Quintile# Cases# Controls# Sampled Lowest128124 2nd306760 Middle443876 4th531530 Highest78 816 Total 217 209 206

8. 8 Propensity Score Sampling Example: Treatments for Drug Abusers Patients seeking substance abuse detoxification in Boston receive either Residential detoxification Lasts ~ one week + encouragement for post- detox treatment, or Acupuncture Acute (daily) detox + 3-6 months of maintenance with acupuncture and motivational counseling

9. 9 Data From Boston’s publicly-funded substance abuse treatment system All cases discharged from residential detox or acupuncture between 1/93 and 9/94 Client classified (only once) as residential or acupuncture based on the modality of first discharge

10. 10 Outcome Is client re-admitted to detox within 6 months? (Y/N) Study question: Are acupuncture clients more likely to be re-admitted than residential detox clients

11. 11 Client Characteristics Available At Time Of Admission Gender Race/ethnicity Age Education Employment status Income Health insurance status Living situation Prior mental health treatment Primary drug Substance abuse treatment history

12. 12 Residential Detox & Acupuncture Cases: % with Various Characteristics Characteristic Residential (n = 6,907) Acupuncture (n = 1,104) Gender: female 2933 Race/ethnicity: black 46 Hispanic 12 10 White4143 Education: HS grad5659 College graduate413

13. Employment: unemployed 86.8 43.2 Insurance: uninsured 65.4 52.3 Medicaid 28.221.2 Private insurance 3.015.4 Lives: with child 9.519.3 In shelter 30.32.9 Characteristic Residential (n = 6,907) Acupuncture (n = 1,104) Characteristics of Residential Detox & Acupuncture Clients (2)

14. 14 Prior mental health treatment 12.327.8 Primary drug: alcohol 42.332.4 Cocaine 16.2 16.6 Crack 15.920.2 Heroin 24.619.0 Characteristic Residential (n = 6,907) Acupuncture (n = 1,104) Characteristics of Residential Detox & Acupuncture Clients (3)

15. 15 Substance abuse admits in the last year Residential detox: 0 1 2+ Short-term residential: 0 Long-term residential: 0 Outpatient: None Acupuncture: None 56.7 20.2 23.1 76.2 80.5 80.6 95.9 81.0 12.1 7.0 94.8 93.5 54.3 90.1 Characteristic Residential (n = 6,907) Acupuncture (n = 1,104) Characteristics of Residential Detox & Acupuncture Clients (4)

16. 16 Results Of Standard Analysis Percentage of clients re-admitted to detox within 6 months Among 1,104 acupuncture cases, 18% re-admitted Among 6,907 residential detox cases, 36% re-admitted Raw odds ratio = 0.40 From a multivariable stepwise logistic regression model: Odds ratio for acupuncture:0.71 (CI = 0.53-0.95)

17. 17 Concern Given large differences in risk adjustors between the groups and possibility of model mis-specification, should we be concerned about the estimated effect of acupuncture? Stratum-specific modeling has been used to address such concerns historically –Strata defined by a limited number of particularly important risk adjustors Propensity scores, a generalization –Used when there are many important predictors

18. 18 Propensity Score Application Use stepwise logistic regression to build a model to predict whether a client “is a case”(i.e., receives acupuncture) Select sub-samples of cases and controls that have similar distribution of the “propensity score” (predicted probability of being a case) Model (as before) on the sub-sample

19. 19 Sampling Results Able to match 740 cases (out of the full sample of 1,104 cases) with 740 comparable controls (out of the full sample of 6,907 controls) Much more balance in terms of risk in this sub-sample

20. 20 Characteristic Residential Acupuncture College graduate Employed Private Insurance Lives with child or adult Lives in shelter Prior mental health Rx 7% 41% 9% 72% 5% 21% (4%) (13%) (3%) (55%) (30%) (12%) 7% 42% 6% 77% 4% 21% (13%) (57%) (15%) (76%) (3%) (28%) Characteristics of Clients in Subsample (Full Sample)

21. 21 Comparing Standard and Propensity Score Findings From the multivariable model fit to all cases: Odds Ratio for acupuncture: 0.71 95% Confidence Interval: 0.53-0.95 From multivariable model fit to more comparable sub- sample: OR for acupuncture: 0.61 95% CI:0.39-0.94

22. 22 Summary In this case, results were similar - Why? Original model was very good (C-statistic = 0.96) What was learned from the propensity score analysis: –Could find a subset of controls (about 10%) who look very similar to cases –Found similar results in this subset, increasing the credibility of the findings

23. 23 Which Belong in the PS Model? Confounders (Brookhart’s X 1 variables)? Simple predictors (X 2 s)? Simple selectors (X 3 s)? Let’s work together to fill in the following table

24. Variable typeDirectly affects Belongs in which model Outcome (Y) Exposure (E)PSRegression X1Confounder11?? X2Predictor10?? X3Selector01?? 24