1. Issues in Selecting Covariates for Propensity Score Adjustment William R Shadish University of California, Merced

2. Shadish, Clark & Steiner Randomly assigned participants to be in a randomized or nonrandomized experiment –Extensive pretesting on covariates Then tested whether we could reproduce the RE results by adjusting the NRE results. –Masking of NRE analyst from RE analyst Here is the design:

3. Random Assignment N = 445 Undergrad Psych Students Randomized Experiment N = 235 Randomly Assigned to Nonrandomized Experiment N = 210 Self-Selected into Mathematics Training N = 119 Mathematics Training N = 79 Vocabulary Training N = 131 Vocabulary Training N = 116 All Participants Post-tested on both Vocabulary and Mathematics Outcomes

4. Results Propensity score analysis could reduce bias in NRE estimates 58-96%, depending on the exact adjustment used. –Ordinary analysis of covariance did as well or better. –So did structural equation modeling –In fact, analytic method didn’t seem to matter much (By the way, this has been replicated in Germany) The quality of the measurement seemed to be the key:

5. Predictors of Convenience First hint of the importance of quality of measurement of predictors came from a sub-analysis of this study. If we limited our analysis to demographic predictors (age, gender, marital status, ethnicity) that are usually conveniently available, bias reduction was poor:

6. Mathematics outcome

7. Vocabulary Outcome

8. Exploring Covariate Sets 1 Imagine we split our 25 covariates into 5 concept domains –Demographics –Proxy Pretests –Topic Preference –Academic Achievement –Psychological Personality Traits Let’s explore how they relate to bias reduction 1 This section based on Steiner, Cook, Shadish & Clark (submitted).

9. Vocabulary Bias Remaining Depending on Which Covariate Sets Used Good bias reduction if you use most or all the covariates

10. Vocabulary Bias Remaining Depending on Which Covariate Sets Used But also good bias reduction using just a few of the “right” covariates.

11. Math Bias Remaining Depending on Which Covariate Sets Used Ditto for Math Good bias reduction if you use all covariates. But also good bias reduction with the “right”ones. In both math and vocab, topic preference and pretest were key.

12. Comments One strategy is to pick the “right” variables –But the best we can say to help researchers find them is they were among the most highly correlated with treatment and outcome An alternative is the “kitchen sink” strategy. –Measure as many covariates as possible and hope the key ones are in there. But does the kitchen sink need to include the “right” variables?

13. Vocabulary Bias Remaining using Individual Covariates For bias reduction in vocabulary outcome, the key individual variables were preference for math and vocabulary pretest scores.

14. Vocabulary Bias Remaining using Individual Covariates But bias reduction in vocabulary outcome was also good if you measured the right “domains” even if you didn’t have the “right” variables. Even if those domains were measured by variables that individually did not reduce bias very well.

15. Math Bias Remaining using Individual Covariates And the same was true for bias reduction in math outcome.

16. Comments So the best advice is to have the right variables. –But except for ensuring they are correlated with treatment and outcome, it is hard to know for sure. Alternative advice is the “kitchen sink” –With the “right” domains—possibly an easier task –With multiple measures in each domain even if none of the measures are the “right” ones.

17. Measurement Error 1 Some parts of the literature, like propensity score analysis, have mostly ignored the role of measurement error. We used the data from this study to simulate the effects of adding various amounts of measurement error to covariates. –2000 replications –Re-estimating PS’s each time –For three sets of All covariates Effective covariates Ineffective covariates 1 This section based on Steiner, Cook & Shadish (submitted).

18. Bias Reduction in Vocab Outcome Increasing measurement error decreases bias reduction for (a) all and (b) effective covariates, but has little impact on (c) ineffective covariates.

19. Bias Reduction in Math Outcome And the same is true for math outcome.

20. Some References Steiner, P.M., Cook, T.D., Shadish, W.R., & Clark, M.H. (under revision). The Importance of Covariate Selection in Controlling for Selection Bias in Observational Studies. Psychological Methods. Steiner, P.M., Cook, T.D., & Shadish, W.R. (under revision). On the Importance of Reliable Covariate Measurement in Selection Bias Adjustments Using Propensity Scores. Journal of Educational and Behavioral Statistics. Shadish, W.R., Clark, M.H., & Steiner, P.M. (2008). Can Nonrandomized Experiments Yield Accurate Answers? A Randomized Experiment Comparing Random to Nonrandom Assignment. Journal of the American Statistical Association, 103, 1334-1343. Shadish, W.R., Clark, M.H., & Steiner, P.M. (2008). [Can Nonrandomized Experiments Yield Accurate Answers? A Randomized Experiment Comparing Random to Nonrandom Assignment]: Rejoinder. Journal of the American Statistical Association, 103, 1353-1356. Shadish, W.R., & Cook, T.D. (2009). The Renaissance of Field Experimentation in Evaluating Interventions. Annual Review of Psychology, 60, 607-629.

21. The End Acknowledgements: M.H. Clark (SIU), Peter Steiner (NU), Tom Cook (NU)