Constructing Propensity score weighted and matched Samples Stacey L Constructing Propensity score weighted and matched Samples Stacey L. Houston, II, MA JBS International Propensity score matching

Introduction to Use of Propensity Scores (P Scores) Non-experimental designs and observational studies lack random assignment Selection bias: large differences in group characteristics Influence estimated group effects Methods for adjusting include: Propensity Score Weighting (PSW) and Propensity Score Matching (PSM) Specify that bias based on who selects into treatment – there might be characteristics of individuals in the treatment group that make them more likely to go into treatment – for example people who are more econ stable may be more likely to enter treatment because they have more time and resources to enter treatment which could influence the outcome

An Example Home Visitation No Home Visitation

What is a Propensity Score? An estimate of the probability of finding the subject in a particular group of interest Logistic regression can predict likelihood of being in a particular group given individual covariates Covariates to include Variables hypothesized to be associated with both treatment and outcome All available data Typically use longitudinal data with baseline characteristics in match

An Example Home Visitation No Home Visitation .8 .8 .2 .2 .2 .8 .8 .8 Important to emphasize .8 .8 Home Visitation No Home Visitation

Propensity Score Uses Propensity Score Weighting (PSW) Pros: retain entire sample Cons: difficult explanation Propensity Score Matching (PSM) Pros: simple representation of data; lower variance in treatment effect estimate; can use any analytic method Cons: sample size dependent; requires sufficient overlap between groups Any analytic method: Meaning once you have the matched sample you can then conduct any analysis that youre interested in such as regression, survival analysis, Hierarchical modeling Match can occur whether it’s a sufficient match or not

Weighting - Unweighted .8 .8 .8 .8 .2 .2 .8 .2 .8 .8 .8 .8 Home Visitation No Home Visitation

Weighting - Weighted Home Visitation No Home Visitation IPW=1/1-ps 1.25 5 5 1.25 1 5 5 1.25 1.25 Make clear that first slide is p score and p score and emphasizing that weight is create differently for both groups 1.25 1.25 5 5 1 Home Visitation No Home Visitation

Matching - Unmatched Home Visitation No Home Visitation .8 .8 .8 .8 .2 .8 .8 .2 .2 .8 .2 .8 .8 .8 .8 Home Visitation No Home Visitation

Matching – Matched (Exact) .8 .8 .8 .8 .2 .2 .2 .8 .8 .8 .8 .8 Home Visitation No Home Visitation

Complexities Multiple Covariates/Characteristics Matching Methods Exact matching Nearest neighbor Caliper matching With(out) replacement Multiple Treatment Groups Another key issue is whether controls can be used as matches for more than one treated unit; whether the matching should be done “with replacement” or “without replacement.” Matching with replacement can often yield better matches because controls that look similar to many treated units can be used multiple times. Additionally, like optimal matching, when matching with replacement the order in which the treated units are matched does not matter. However, a drawback of matching with replacement is that it may be that only a few unique control units will be selected as matches; the number of times each control is matched should be monitored Do nearest neighbor example with or without replacement

An Example Are families who participate in B&B less likely to have a subsequent child protective services referral than those who do not?

Propensity Score Weighting Table 1. Means, Percentages, and Standard Deviations (SD) for All Study Variables Unweighted   Propensity Score Weighted Treatment Control (N=893) (N=9,708) Mean/ Variables Percent Demographics Person was White (versus nonwhite) 28.20 36.80 *** 37.40 36.10 Person's primary language was English (versus any other) 82.80 82.90 81.40 Age in 2015 27.40 30.54 30.94 30.29 ** Unique Referrals At/Before Point of Eligibility for the Program Number of Unique Perpetrator or Other (restricted to those when person was 17 or older) Referrals 3.10 2.04 2.41 2.14 Number of Unique Victim Referrals 2.45 0.66 0.81 0.82 Age of Person's First Referral 16.71 23.76 23.61 23.17 *p<.05; **p<.01; ***p<.001 (treatment compared to control group). Maybe just do demographics! Probably just do the two means Mention the sample sizes and how we get to use the full sample Take some time to orient them to the table itself – put a white block over the weighted

Propensity Score Matching Table 1. Means, Percentages, and Standard Deviations (SD) for All Study Variables Unmatched Matched 2:1 Treatment Control (N=496) (N=9,210) (N=493) (N=985) Mean/   Variables Percent Demographics Person was White (versus nonwhite) 29.20 36.70 *** 29.40 30.60 Person's primary language was English (versus any other) 83.70 82.90 83.60 87.00 Age in 2015 27.46 30.39 27.48 27.54 Unique Referrals At/Before Point of Eligibility for the Program Number of Unique Perpetrator or Other (restricted to those when person was 17 or older) Referrals 2.94 1.92 2.87 2.61 Number of Unique Victim Referrals 2.41 0.70 2.35 Age of Person's First Referral 16.56 23.67 16.63 16.62 *p<.05; **p<.01; ***p<.001 (treatment compared to control group). Maybe just do demographics! Probably just do the two means

Propensity Score Best Practices If covariates are known to impact outcome variable, be sure to include them when possible Always demonstrate level of imbalance both before and after matching or weighting Try to use P scores only when imbalance is high initially Iteratively check balance and respecify propensity score regression requires sufficient overlap between groups

Summary Propensity scores can be an effective method of reducing selection bias DO NOT match or weight just because of recent trends Even if p-scores are not used, still a beneficial practice to estimate p-scores for group assignment of interest Especially circumstances when random assignment isn’t feasible and you have sufficiently available comparison group data and when you follow

Thank you! Questions? Additional resources and step by step instructions for conducting a survival analysis can be found at: