1. Constructing Propensity score weighted and matched Samples Stacey L
Constructing Propensity score weighted and matched Samples Stacey L. Houston, II, MA JBS International
Propensity score matching

2. Introduction to Use of Propensity Scores (P Scores)
Non-experimental designs and observational studies lack random assignment
Selection bias: large differences in group characteristics
Influence estimated group effects
Methods for adjusting include:
Propensity Score Weighting (PSW) and Propensity Score Matching (PSM)
Specify that bias based on who selects into treatment – there might be characteristics of individuals in the treatment group that make them more likely to go into treatment – for example people who are more econ stable may be more likely to enter treatment because they have more time and resources to enter treatment which could influence the outcome

3. An Example
Home Visitation
No Home Visitation

4. What is a Propensity Score?
An estimate of the probability of finding the subject in a particular group of interest
Logistic regression can predict likelihood of being in a particular group given individual covariates
Covariates to include
Variables hypothesized to be associated with both treatment and outcome
All available data
Typically use longitudinal data with baseline characteristics in match

5. An Example Home Visitation No Home Visitation .8 .8 .2 .2 .2 .8 .8 .8
Important to emphasize .8 .8
Home Visitation
No Home Visitation

6. Propensity Score Uses Propensity Score Weighting (PSW)
Pros: retain entire sample
Cons: difficult explanation
Propensity Score Matching (PSM)
Pros: simple representation of data; lower variance in treatment effect estimate; can use any analytic method
Cons: sample size dependent; requires sufficient overlap between groups
Any analytic method: Meaning once you have the matched sample you can then conduct any analysis that youre interested in such as regression, survival analysis, Hierarchical modeling
Match can occur whether it’s a sufficient match or not

7. Weighting - Unweighted.8 .8 .8 .8 .2 .2 .8 .2 .8 .8 .8 .8
Home Visitation
No Home Visitation

8. Weighting - Weighted Home Visitation No Home Visitation IPW=1/1-ps
1.25 5 5
1.25 1 5 5
1.25
1.25
Make clear that first slide is p score and p score and emphasizing that weight is create differently for both groups
1.25
1.25 5 5 1
Home Visitation
No Home Visitation

9. Matching - Unmatched Home Visitation No Home Visitation .8 .8 .8 .8 .2.8 .8 .2 .2 .8 .2 .8 .8 .8 .8
Home Visitation
No Home Visitation

10. Matching – Matched (Exact).8 .8 .8 .8 .2 .2 .2 .8 .8 .8 .8 .8
Home Visitation
No Home Visitation

11. Complexities Multiple Covariates/Characteristics Matching Methods
Exact matching
Nearest neighbor
Caliper matching
With(out) replacement
Multiple Treatment Groups
Another key issue is whether controls can be used as matches for more than one treated unit; whether the matching should be done “with replacement” or “without replacement.” Matching with replacement can often yield better matches because controls that look similar to many treated units can be used multiple times. Additionally, like optimal matching, when matching with replacement the order in which the treated units are matched does not matter. However, a drawback of matching with replacement is that it may be that only a few unique control units will be selected as matches; the number of times each
control is matched should be monitored
Do nearest neighbor example with or without replacement

12. An Example
Are families who participate in B&B less likely to have a subsequent child protective services referral than those who do not?

13. Propensity Score Weighting
Table 1. Means, Percentages, and Standard Deviations (SD) for All Study Variables
Unweighted
Propensity Score Weighted
Treatment
Control
(N=893)
(N=9,708)
Mean/
Variables
Percent
Demographics
Person was White (versus nonwhite)
28.20
36.80
***
37.40
36.10
Person's primary language was English (versus any other)
82.80
82.90
81.40
Age in 2015
27.40
30.54
30.94
30.29 **
Unique Referrals At/Before Point of Eligibility for the Program
Number of Unique Perpetrator or Other (restricted to those when person was 17 or older) Referrals
3.10
2.04
2.41
2.14
Number of Unique Victim Referrals
2.45
0.66
0.81
0.82
Age of Person's First Referral
16.71
23.76
23.61
23.17
*p<.05; **p<.01; ***p<.001 (treatment compared to control group).
Maybe just do demographics!
Probably just do the two means
Mention the sample sizes and how we get to use the full sample
Take some time to orient them to the table itself – put a white block over the weighted

14. Propensity Score Matching
Table 1. Means, Percentages, and Standard Deviations (SD) for All Study Variables
Unmatched
Matched 2:1
Treatment
Control
(N=496)
(N=9,210)
(N=493)
(N=985)
Mean/
Variables
Percent
Demographics
Person was White (versus nonwhite)
29.20
36.70
***
29.40
30.60
Person's primary language was English (versus any other)
83.70
82.90
83.60
87.00
Age in 2015
27.46
30.39
27.48
27.54
Unique Referrals At/Before Point of Eligibility for the Program
Number of Unique Perpetrator or Other (restricted to those when person was 17 or older) Referrals
2.94
1.92
2.87
2.61
Number of Unique Victim Referrals
2.41
0.70
2.35
Age of Person's First Referral
16.56
23.67
16.63
16.62
*p<.05; **p<.01; ***p<.001 (treatment compared to control group).
Maybe just do demographics!
Probably just do the two means

15. Propensity Score Best Practices
If covariates are known to impact outcome variable, be sure to include them when possible
Always demonstrate level of imbalance both before and after matching or weighting
Try to use P scores only when imbalance is high initially
Iteratively check balance and respecify propensity score regression
requires sufficient overlap between groups

16. Summary
Propensity scores can be an effective method of reducing selection bias
DO NOT match or weight just because of recent trends
Even if p-scores are not used, still a beneficial practice to estimate p-scores for group assignment of interest
Especially circumstances when random assignment isn’t feasible and you have sufficiently available comparison group data and when you follow

17. Thank you!
Questions?
Additional resources and step by step instructions for conducting a survival analysis can be found at: