1. Stanford University School of Medicine
Multiple Imputation Strategies for Handling Missing Data When Generalizing Randomized Clinical Trial Findings Through Propensity Score-Based Methodologies
Albee Ling, Maya Mathur, Kris Kapphahn, Maria Montez-Rath, Manisha Desai
Stanford University School of Medicine
JSM 2018

2. RCTs have limited external validity
Findings from randomized clinical trials (RCTs) are internally valid, but not necessarily externally valid.
For example, the Frequent Hemodialysis Network (FHN) Daily Trial showed a benefit of an intensive dialysis schedule vs a standard schedule among patients with end stage renal disease (ESRD) undergoing hemodialysis.
Physicians still not sure whether findings apply to their typical ESRD patients because patients were highly selected as those that might be most likely to demonstrate a benefit.
We want to generalize findings to typical ESRD patient today.
Randomized clinical trials are known for its high internal validity compared to observational studies. However, trial participants might no be representative of the target population.

3. PS methods are used to generalize RCT findings
In classical causal inference setting, propensity score (PS) is defined as the conditional probability of treatment
In generalizing RCT findings, PS has been adapted to be the probability of selection into trial
PS is estimated using both trial sample and target population combined
Reweighting or matching to trial sample to estimate treatment effect for target population (Cole & Stuart 2010)

4. Key covariates can be missing in target population
Key covariates measured in both trial sample and target population are used to estimated PS
Some can be partially missing in the target population
Commonly used methods can result in bias and inefficiency
Complete case analysis (CC)
Complete variable analysis (CVA)
Single imputation

5. Research goals
Compare and characterize estimators for deriving and integrating propensity scores to generalize RCT findings in the presence of missing data through MI techniques
Inform a set of guidelines for best practices
Illustrate differences in treatment effects between best and common practices using data from real clinical trial findings

6. Application of MI in estimating PS with missing data
Var1
Var2
Var3 … 45 2 1 70 56 58 44 3
Var1
Var2
Var3 … 45 2 1 70
missing 56 58 3
Var1
Var2
Var3 … 45 2 1 70 56 3 58
What is MI/Why use MI:
Simulation based statistical tool to handle missing data
Imputation is done multiple times to reflect the uncertainty of the process
Filling in missing values from a plausible distribution based on observed data
Complete data analyses are performed on each data set and combined to give one summary finding
Assumes a more flexible missing data mechanism (MAR)
Theoretically proved to yield statistically valid results
Var1
Var2
Var3 … 45 2 1 70 56 58 60 3

7. How to impute when estimating PS?
Var1
Var2
Var3 … 45 2 1 70 56 58 50 3 PS
0.5
0.9
0.1
0.6
Var1
Var2
Var3 … 45 2 1 70
missing 56 58 3
MI-Passive
Var1
Var2
Var3 … PS 45 2 1
0.5 70
missing 56 58
0.6 3
Var1
Var2
Var3 … PS 45 2 1
0.5 70
0.9 56
0.1 58
0.6 50 3
Imputing PS, derived variable
Imputing covariates and then estimating PS (MI-passive)
Not considering covariance structure of all variables could potentially lead to bias
Imputing PS directly (MI-active)
Can result in nonsensical answers
Most people have evaluate the first one not the second one
MI-Active

8. How to integrate PS into analyses?
Imputed Data Set 1
Imputed Data Set 2
Imputed Data Set 3
Original Data Set 𝑃𝑆 𝑃𝑆 𝑃𝑆 𝑃𝑆
Treated
Control
Treated
Control
Treated
Control
Treated
Control
Integrating PS into analyses
Matching/IPTW using PS within each imputed data set
Matching/IPTW after pooling PSs from all imputed data sets 𝛽 𝛽1 𝛽2 𝛽3 𝛽
PSI-Within
PSI-Across

9. Addressing the gap in current literature
Prior literature: Qu & Lipkovich 2009, Hill 2004, Mattei 2009, Mitra & Reiter 2016, de Vries 2017
derPassive
regActive
MI-Passive
MI-Active
regPassive
redActive
impPassive
PSI-Within
PSI-Across
No work has been done in redActive, regPassive and impPassive

10. Simulation study design
𝑌 𝑖 = 𝛽 𝑇 ( 𝑋 𝑖 , 𝑇 𝑖 )
Continuous outcome
Binary and continuous covariates
Binary treatment
Induce missingness in one continuous confounder
Missingness is related to PS and outcome or an auxiliary variable
𝑃𝑆 𝑖 =𝑙𝑜𝑔𝑖𝑡(𝑎 𝑠𝑢𝑏𝑠𝑒𝑡 𝑜𝑓 𝑋 𝑖 )
True treatment effect = 6
30% patients treated and 70% untreated
MDM:
50% of a continuous confounder is missing
The auxiliary variable is highly correlated with the missing variable
Mild MAR: missingness related to PS and outcome, both dichotomized
Moderate MAR1: missingness related to PS, outcome and auxiliary variable (all dichotomized)
Moderate MAR2: missingness related to PS, outcome and auxiliary variable (all continuous)
One-to-one nearest neighbor matching with caliper
Estimating treatment effect

11. Missing data methods considered
Baseline methods:
Complete case analysis (CC)
Complete variable analysis (CVA)
Mean imputation
Missing indicator imputation
MI methods:
Active vs passive
Within vs across

12. Evaluating metrics Balance diagnosis: Bias Variance MSE (relative MSE)
Percentage matched
Number of balanced PS variables
Bias
Variance
Rubin’s Rule
Bootstrap
MSE (relative MSE)
Coverage

13. Standard error estimation
Need to take into account uncertainties from PS estimation, matching process, as well as MI procedure in estimating standard error (SE)
Problems with estimation methods in the current literature
Rubin’s Rule: lack of analytical solutions to estimate SE from PS matching
bootstrap procedure: SE is non-smooth
SE obtained using the bootstrap for multiply imputed data
i. Sample with replacement n rows from the incomplete dataset z to obtain a bootstrapped dataset zb.
ii. Impute m data sets for zb, for k = 1,2,...,m denoted as zb(k).
iii. Apply the analysis procedure (e.g. Within or Across approach) to the m imputed datasets to obtain a single effect estimate for zb.
iv. Repeat steps i.-iii. B times to obtain B bootstrap replicates from which to estimate the SE of our treatment effect
Bootstrap SE is the standard error from treatment effects estimated in all bootstrap samples

14. Performance of common missing data methods
100 simulations
Mean imputation is performing the best in terms of bias (probably because we are missing a continuous variable)
CC, CVA and mean missing indicator performs poorly in terms of bias
CVA is very consistent across all MDMs in terms of both bias and SE

15. Performance of MI methods
100 simulations
Bias
Comparing across difference missing data mechanisms, mild MAR is doing better than the two moderate MARs
Passive approaches have more similar bias than actives. But depending on the MDM, passive or active can be more or less biased
SE (empirical)
Passive within < passive across < active within/active across

16. Standard error estimation
100 simulations
200 bootstrap samples
Across all three missing data mechaniss
Passive: empirical SE is smaller than Rubin’s Rule or bootstrap SE
Active: empirical SE is bigger than Rubin’s Rule or bootstrap SE
There is no consistent pattern when comparing Rubin’s Rule to bootstrap SEs
Will need to increase the number of iterations (currently 100) and bootstrap sample size (currently 200) to see if the current results retain

17. Future directions
Evaluate redActive, regPassive and impPassive approaches
Evaluate all approaches in the presence of auxiliary terms
Propose new ways of estimating SE
Assess performance of novel and standard methods under different missing data mechanisms for PS weighting approaches

18. Stanford Graduate Fellowship
Acknowledgement
Manisha Desai
Maria Montez-Rath
Maya Mathur
Kris Kapphahn
Stanford Graduate Fellowship

19. Questions?

20. Supplementary Slides

21. Proposed MI methods to evaluate

22. Simulation study design details
Covariates first generated jointly using R package BinNor so that they are correlated with one another with rho=0.1
Confounders in bold and missing variable in red
Exposure: Treatment 𝑇 𝑖 generated as a function of those covariates strongly and moderately associated with treatment as described below
Outcome ( 𝑌 𝑖 ): continuous and generated as a function of treatment, and covariates associated with outcome as described below
Association with treatment
strong
moderate
none
Association with outcome
𝑩 𝟏 , 𝑪 𝟏
𝑩 𝟐 , 𝑪 𝟐
𝐵 3 , 𝐶 3
𝑩 𝟒 , 𝑪 𝟒
𝑩 𝟓 , 𝑪 𝟓
𝐵 6 , 𝐶 6
𝐵 7 , 𝐶 7
𝐵 8 , 𝐶 8
𝐵 9 , 𝐶 9
R package BinNor only generates normal and binary variables not categorical

23. Propensity score matching specifics
PS:
Conditional probability of treatment
Estimated using logistic regression on B 1 , C 1 , B 2 , C 2 , B 3 , C 3 , B 4 , C 4 , B 5 , C 5 , B 6 , C 6 (including both true confounders and other covariates associated with outcome but not treatment)
PS Matching
One-to-one nearest neighbor matching with caliper
Effect estimate = average difference between treated and non-treated pairs, averaged across m data sets
Austin, Peter C. "An introduction to propensity score methods for reducing the effects of confounding in observational studies." Multivariate behavioral research 46.3 (2011):
Austin, Peter C., Paul Grootendorst, and Geoffrey M. Anderson. "A comparison of the ability of different propensity score models to balance measured variables between treated and untreated subjects: a Monte Carlo study." Statistics in medicine 26.4 (2007):

24. Inducing missingness in simulated datasets
One continuous confounder is missing 50% data
Missingness is associated with the following variables:
Mild MAR: PS and outcome both dichotomized
Moderate MAR1: PS, outcome, and auxiliary variable, all dichotomized
Moderate MAR2: PS, outcome, and auxiliary variable, all continuous
(auxiliary variable is highly correlated with the missing variable)

25. Bootstrap method to estimated standard error
Bootstrap Procedure
Sample with replacement n rows from the incomplete dataset z to obtain a bootstrapped dataset zb.
Impute m data sets for zb, for k = 1,2,...,m denoted as zb(k).
Apply the analysis procedure (e.g. Within or Across approach) to the m imputed datasets to obtain a single effect estimate for zb.
Repeat steps 1-3. B times to obtain B bootstrap replicates from which to estimate the SE of our treatment effect
Bootstrap SE is the standard error of b treatment effects estimated from each bootstrap sample
Concerns of using bootstrap to estimate SE
Bootstrap fails to capture uncertainties in the context of PS matching
MI makes the SE non-smooth, thus violating bootstrap assumptions