Stanford University School of Medicine Multiple Imputation Strategies for Handling Missing Data When Generalizing Randomized Clinical Trial Findings Through Propensity Score-Based Methodologies Albee Ling, Maya Mathur, Kris Kapphahn, Maria Montez-Rath, Manisha Desai Stanford University School of Medicine JSM 2018

RCTs have limited external validity Findings from randomized clinical trials (RCTs) are internally valid, but not necessarily externally valid. For example, the Frequent Hemodialysis Network (FHN) Daily Trial showed a benefit of an intensive dialysis schedule vs a standard schedule among patients with end stage renal disease (ESRD) undergoing hemodialysis. Physicians still not sure whether findings apply to their typical ESRD patients because patients were highly selected as those that might be most likely to demonstrate a benefit. We want to generalize findings to typical ESRD patient today. Randomized clinical trials are known for its high internal validity compared to observational studies. However, trial participants might no be representative of the target population.

PS methods are used to generalize RCT findings In classical causal inference setting, propensity score (PS) is defined as the conditional probability of treatment In generalizing RCT findings, PS has been adapted to be the probability of selection into trial PS is estimated using both trial sample and target population combined Reweighting or matching to trial sample to estimate treatment effect for target population (Cole & Stuart 2010)

Key covariates can be missing in target population Key covariates measured in both trial sample and target population are used to estimated PS Some can be partially missing in the target population Commonly used methods can result in bias and inefficiency Complete case analysis (CC) Complete variable analysis (CVA) Single imputation

Research goals Compare and characterize estimators for deriving and integrating propensity scores to generalize RCT findings in the presence of missing data through MI techniques Inform a set of guidelines for best practices Illustrate differences in treatment effects between best and common practices using data from real clinical trial findings

Application of MI in estimating PS with missing data Var1 Var2 Var3 … 45 2 1 70 56 58 44 3 Var1 Var2 Var3 … 45 2 1 70 missing 56 58 3 Var1 Var2 Var3 … 45 2 1 70 56 3 58 What is MI/Why use MI: Simulation based statistical tool to handle missing data Imputation is done multiple times to reflect the uncertainty of the process Filling in missing values from a plausible distribution based on observed data Complete data analyses are performed on each data set and combined to give one summary finding Assumes a more flexible missing data mechanism (MAR) Theoretically proved to yield statistically valid results Var1 Var2 Var3 … 45 2 1 70 56 58 60 3

How to impute when estimating PS? Var1 Var2 Var3 … 45 2 1 70 56 58 50 3 PS 0.5 0.9 0.1 0.6 Var1 Var2 Var3 … 45 2 1 70 missing 56 58 3 MI-Passive Var1 Var2 Var3 … PS 45 2 1 0.5 70 missing 56 58 0.6 3 Var1 Var2 Var3 … PS 45 2 1 0.5 70 0.9 56 0.1 58 0.6 50 3 Imputing PS, derived variable Imputing covariates and then estimating PS (MI-passive) Not considering covariance structure of all variables could potentially lead to bias Imputing PS directly (MI-active) Can result in nonsensical answers Most people have evaluate the first one not the second one MI-Active

How to integrate PS into analyses? Imputed Data Set 1 Imputed Data Set 2 Imputed Data Set 3 Original Data Set 𝑃𝑆 𝑃𝑆 𝑃𝑆 𝑃𝑆 Treated Control Treated Control Treated Control Treated Control Integrating PS into analyses Matching/IPTW using PS within each imputed data set Matching/IPTW after pooling PSs from all imputed data sets 𝛽 𝛽1 𝛽2 𝛽3 𝛽 PSI-Within PSI-Across

Addressing the gap in current literature Prior literature: Qu & Lipkovich 2009, Hill 2004, Mattei 2009, Mitra & Reiter 2016, de Vries 2017 derPassive regActive MI-Passive MI-Active regPassive redActive impPassive PSI-Within PSI-Across No work has been done in redActive, regPassive and impPassive

Simulation study design 𝑌 𝑖 = 𝛽 𝑇 ( 𝑋 𝑖 , 𝑇 𝑖 ) Continuous outcome Binary and continuous covariates Binary treatment Induce missingness in one continuous confounder Missingness is related to PS and outcome or an auxiliary variable 𝑃𝑆 𝑖 =𝑙𝑜𝑔𝑖𝑡(𝑎 𝑠𝑢𝑏𝑠𝑒𝑡 𝑜𝑓 𝑋 𝑖 ) True treatment effect = 6 30% patients treated and 70% untreated MDM: 50% of a continuous confounder is missing The auxiliary variable is highly correlated with the missing variable Mild MAR: missingness related to PS and outcome, both dichotomized Moderate MAR1: missingness related to PS, outcome and auxiliary variable (all dichotomized) Moderate MAR2: missingness related to PS, outcome and auxiliary variable (all continuous) One-to-one nearest neighbor matching with caliper Estimating treatment effect

Missing data methods considered Baseline methods: Complete case analysis (CC) Complete variable analysis (CVA) Mean imputation Missing indicator imputation MI methods: Active vs passive Within vs across

Evaluating metrics Balance diagnosis: Bias Variance MSE (relative MSE) Percentage matched Number of balanced PS variables Bias Variance Rubin’s Rule Bootstrap MSE (relative MSE) Coverage

Standard error estimation Need to take into account uncertainties from PS estimation, matching process, as well as MI procedure in estimating standard error (SE) Problems with estimation methods in the current literature Rubin’s Rule: lack of analytical solutions to estimate SE from PS matching bootstrap procedure: SE is non-smooth SE obtained using the bootstrap for multiply imputed data i. Sample with replacement n rows from the incomplete dataset z to obtain a bootstrapped dataset zb. ii. Impute m data sets for zb, for k = 1,2,...,m denoted as zb(k). iii. Apply the analysis procedure (e.g. Within or Across approach) to the m imputed datasets to obtain a single effect estimate for zb. iv. Repeat steps i.-iii. B times to obtain B bootstrap replicates from which to estimate the SE of our treatment effect Bootstrap SE is the standard error from treatment effects estimated in all bootstrap samples

Performance of common missing data methods 100 simulations Mean imputation is performing the best in terms of bias (probably because we are missing a continuous variable) CC, CVA and mean missing indicator performs poorly in terms of bias CVA is very consistent across all MDMs in terms of both bias and SE

Performance of MI methods 100 simulations Bias Comparing across difference missing data mechanisms, mild MAR is doing better than the two moderate MARs Passive approaches have more similar bias than actives. But depending on the MDM, passive or active can be more or less biased SE (empirical) Passive within < passive across < active within/active across

Standard error estimation 100 simulations 200 bootstrap samples Across all three missing data mechaniss Passive: empirical SE is smaller than Rubin’s Rule or bootstrap SE Active: empirical SE is bigger than Rubin’s Rule or bootstrap SE There is no consistent pattern when comparing Rubin’s Rule to bootstrap SEs Will need to increase the number of iterations (currently 100) and bootstrap sample size (currently 200) to see if the current results retain

Future directions Evaluate redActive, regPassive and impPassive approaches Evaluate all approaches in the presence of auxiliary terms Propose new ways of estimating SE Assess performance of novel and standard methods under different missing data mechanisms for PS weighting approaches

Stanford Graduate Fellowship Acknowledgement Manisha Desai Maria Montez-Rath Maya Mathur Kris Kapphahn Stanford Graduate Fellowship

Questions?

Supplementary Slides

Proposed MI methods to evaluate

Simulation study design details Covariates first generated jointly using R package BinNor so that they are correlated with one another with rho=0.1 Confounders in bold and missing variable in red Exposure: Treatment 𝑇 𝑖 generated as a function of those covariates strongly and moderately associated with treatment as described below Outcome ( 𝑌 𝑖 ): continuous and generated as a function of treatment, and covariates associated with outcome as described below   Association with treatment strong moderate none Association with outcome 𝑩 𝟏 , 𝑪 𝟏 𝑩 𝟐 , 𝑪 𝟐 𝐵 3 , 𝐶 3 𝑩 𝟒 , 𝑪 𝟒 𝑩 𝟓 , 𝑪 𝟓 𝐵 6 , 𝐶 6 𝐵 7 , 𝐶 7 𝐵 8 , 𝐶 8 𝐵 9 , 𝐶 9 R package BinNor only generates normal and binary variables not categorical

Propensity score matching specifics PS: Conditional probability of treatment Estimated using logistic regression on B 1 , C 1 , B 2 , C 2 , B 3 , C 3 , B 4 , C 4 , B 5 , C 5 , B 6 , C 6 (including both true confounders and other covariates associated with outcome but not treatment) PS Matching One-to-one nearest neighbor matching with caliper Effect estimate = average difference between treated and non-treated pairs, averaged across m data sets Austin, Peter C. "An introduction to propensity score methods for reducing the effects of confounding in observational studies." Multivariate behavioral research 46.3 (2011): 399-424.   Austin, Peter C., Paul Grootendorst, and Geoffrey M. Anderson. "A comparison of the ability of different propensity score models to balance measured variables between treated and untreated subjects: a Monte Carlo study." Statistics in medicine 26.4 (2007): 734-753.

Inducing missingness in simulated datasets One continuous confounder is missing 50% data Missingness is associated with the following variables: Mild MAR: PS and outcome both dichotomized Moderate MAR1: PS, outcome, and auxiliary variable, all dichotomized Moderate MAR2: PS, outcome, and auxiliary variable, all continuous (auxiliary variable is highly correlated with the missing variable)

Bootstrap method to estimated standard error Bootstrap Procedure Sample with replacement n rows from the incomplete dataset z to obtain a bootstrapped dataset zb. Impute m data sets for zb, for k = 1,2,...,m denoted as zb(k). Apply the analysis procedure (e.g. Within or Across approach) to the m imputed datasets to obtain a single effect estimate for zb. Repeat steps 1-3. B times to obtain B bootstrap replicates from which to estimate the SE of our treatment effect Bootstrap SE is the standard error of b treatment effects estimated from each bootstrap sample Concerns of using bootstrap to estimate SE Bootstrap fails to capture uncertainties in the context of PS matching MI makes the SE non-smooth, thus violating bootstrap assumptions