1. 29 th TRF 2003, Denver July 14 th, 2003 www.kiprc.uky.edu Jenny H. Qin and Mike Singleton Kentucky CODES Kentucky Injury Prevention & Research Center University of Kentucky Performing Sensitivity Analyses of Imputed Missing Values

2. Multiple Imputation in Public Health Research Handling Missing Data in Nursing Research with Multiple Imputation Application of Multiple Imputation in Medical Studies: from AIDS to NHANES NHTSA: Transitioning to Multiple Imputation! A new Method to Impute Missing BAC values in FARS Multiple Imputation Publications

3. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu Questions??? May I use MI to deal with missing data problems for my data sets? How can I believe that the MI will give me better analysis results? What should I do to get good results from MI?

4. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu???Answers Sensitivity Analyses on Imputed Values A sensitivity analysis tests if our study results are sensitive to our assumptions (missing data mechanism), data conditions (missing data rate), and choices (imputation models or number of imputations) made for obtaining the results

5. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu MI Process Data Set of Interest Missing Data Mechanism 1 Missing Data Rate 2 Proc MI Results Analysis Model Imputation Model 3 Proc MI Options 4 Set 1 Set 3 Set 2 Set n...... Proc MIANALYZE Set n Results n Results 3 Results 2 Results 1......

6. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu Research Question: What was the relationship between driving under the influence of drugs and/or alcohol, and being killed or hospitalized in a crash, for motorcycle riders in Kentucky in 2001? Outcome (Dependent Variable): Killed or Hospitalized (K/H) Risk Factor Candidates (Independent Variables): Age, gender, suspected DUI, posted speed limit, helmet use, fixed object, head-on collision, collision time, rural vs. urban CODES Application

7. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu Logistic Regression Model: K/H = β 0 + β 1 *DUI + β 2 *Speed + β 3 *Fixed + β 4 *Head-On Total records in our study Data set: 1,226 Records with missing values: 14 (1.1%) Analysis Model

8. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu Results for the Gold Standard ParameterOR(95% CI)EstimateSEP DUI2.51 (1.58 3.98)0.91890.23640.0001 Speed1.58 (1.18 2.10)0.45460.14560.0018 Fixed1.70 (1.24 2.33)0.53110.15990.0009 Head-on1.70 (1.04 2.77)0.53160.24860.0380 This Gold Standard result is used to compare with all other results. Conclusion: comparing motorcyclists with DUI to motorcyclists without DUI, the odds of being killed or hospitalized are 2.5 times greater than the odds of not being killed or hospitalized, when other factors are controlled.

9. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu Analysis Model: K/H = β 0 + β 1 *DUI + β 2 *Speed + β 3 *Fixed + β 4 *Head-On Imputation Model: K/H DUI Speed Fixed Head-On Note: The imputation model does not have to be identical to the analysis model, but at least it should include all of the analysis covariates. You can add any additional variables that are correlated to the variables that have missing values. Imputation Model

10. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu MCARMARNMAR Study Data Set Missing Data Mechanism 1 Missing Data Rate 2 Proc MI Data Analysis Proc MIANALYZE Results Analysis Model Imputation Model 3 Proc MI options 4 SA: Missing Data Mechanism 1

11. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu SA: Missing Completely At Random (MCAR) –DFN: the missing data values are a simple random sample of all data values. –We simulated this condition by using SAS Proc SurveySelect to pick a random sample from the study data set, then set DUI = missing for those selected cases. Missing At Random (MAR) -DFN: the probability of missing values on one variable is unrelated to the values of this variable, after controlling for other variables in the analysis -We simulated this condition by setting DUI = missing for riders aged 46 or older Not Missing At Random (NMAR) –DFN: the probability of missing values on one variable is related to the values of this variable even if we control other variables in the analysis –We simulated this condition by setting DUI = missing for uninjured riders who were not suspected of DUI (DUI=‘NO’). Missing Data Mechanism 1

12. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu Created 3 data sets from the study data set with different missing data mechanisms, but with the same percent missing values for DUI (25%) MCAR 25% missing on DUI MAR 25% missing on DUI NMAR 25% missing on DUI ParameterESEPE PE P Intercept-1.73360.10960.0001-1.72590.10920.0001-1.72040.10920.0001 DUI0.85440.26640.00160.82860.26230.00180.57910.22230.0092 Speed0.50180.14490.00050.48430.14480.00080.48120.14430.0009 Fixed0.49270.16100.00220.50790.15970.00150.54000.15780.0006 Head-on0.51330.24850.03880.51330.24860.03890.51030.24750.0393

13. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu Sensitivity analysis on missing data mechanism: Different Same What is the result? Imputation Model 3 Proc MI Options 4 Missing Data Rate (25%) 2 Missing Data Mechanism 1

14. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu Conclusions of SA on Missing Data Mechanism Even if we used the simplest imputation model MI was able to produce results that are consistent with the Gold Standard when the missing data mechanisms were MCAR or MAR, but not NMAR we would predict the increased odds of death or hospitalization for riders suspected of DUI to be 1.78 (1.15 2.76) for NMAR, while our Gold Standard predicts it to be 2.51 (1.58 3.98).

15. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu 6%25%50% Study Data Set Missing Data Mechanism 1 Missing Data Rate 2 Proc MI Data Analysis Proc MIANALYZE Results Analysis Model Imputation Model 3 Proc MI options 4 SA: Missing Data Rate 2

16. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu SA: Data sets with MCAR ( Test on percentage of values missing for DUI as 6%, 25%, 50% respectively) Data sets with MAR ( Test on percentage of values missing for DUI as 6%, 25%, 50% respectively) Missing Data Rate 2

17. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu Create 3 data sets with MCAR from the study data set having values missing for DUI as 6%, 25%, and 50% respectively. MCAR 6% missing on DUI MCAR 25% missing on DUI MCAR 50% missing on DUI ParameterESEPE PE P Intercept-1.73610.10940.0001-1.73360.10960.0001-1.73770.11190.0001 DUI0.94470.24290.00010.85440.26640.00160.84570.29730.0065 Speed0.48120.14460.00090.50180.14490.00050.48310.14600.0009 Fixed0.52130.15840.00100.49270.16100.00220.52000.16170.0013 Head-on0.52450.24890.03510.51330.24850.03880.49360.25080.0490

18. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu Create 3 data sets with MAR from the study data set having values missing for DUI as 6%, 25%, and 50% respectively. MAR 6% missing on DUI MAR 25% missing on DUI MAR 50% missing on DUI ParameterESEPE PE P Intercept-1.73820.10950.0001-1.72590.10920.0001-1.75020.11090.0001 DUI0.91910.23340.00010.82860.26230.00181.27220.32980.0002 Speed0.48360.14490.00080.48430.14480.00080.50630.14730.0006 Fixed0.50760.15900.00140.50790.15970.00150.52340.15970.0010 Head-on0.51740.24860.03740.51330.24860.03890.53710.24870.0308

19. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu Sensitivity analysis on Missing Data Rate? Same Different Same What is the result? Imputation Model 3 Proc MI Options 4 Missing Data Rate 2 Missing Data Mechanism MCAR or MAR 1

20. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu Conclusions of SA on Missing Data Rate For both missing data mechanisms, the 50% missing case produced the DUI parameter estimate farthest from the Gold Standard estimate, as well as the widest 95% CI. However, for MCAR the difference from the Gold Standard estimate was -7%, whereas for MAR it was 42%. In addition, the 95% CI for 50%MCAR was 19% wider than the Gold Standard 95% CI, whereas for 50%MAR it was 106% wider. It shows that the simplest imputation model is not sufficient to handle very high missing data rates.

21. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu Study Data Set Missing Data Mechanism 1 Missing Data Rate 2 Proc MI Data Analysis Proc MIANALYZE Results Analysis Model Imputation Model 3 Proc MI options 2 SA: Imputation Model 3 Model1Model2Model3Model4

22. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu SA: Data set with MAR and values missing for DUI=50% Tests on the following 4 Imputation models –Model1: D/H DUI Speed Fixed Head-on Model1 = Analysis model, it is the simplest imputation model –Model2: Model1 + age_group + colltime (Categorical) –Model3: Model1 + age_group + hour (Continuous) –Model4: Model1 + age_group + hour_normal (Continuous) We are adding age and collision time to help predict DUI in Model2, Model3, and Model4 Imputation Model 3

23. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu Use 4 different imputation models to do MI on the same data set with MAR, 50% missing on DUI. Model 2 50% missing on DUI Model 3 50% missing on DUI Model 4 50% missing on DUI ParameterESEPE PE P Intercept-1.81100.12220.0001-1.80810.12350.0001-1.80340.12380.0001 DUI1.01270.29480.00160.98140.29660.00240.95630.28130.0015 Speed0.50790.14660.00050.50210.14630.00060.50810.14690.0005 Fixed0.53700.16040.00080.54040.16010.00070.53710.15980.0008 Head-on0.55540.25370.02860.54770.25520.03200.55610.25210.0274

24. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu Sensitivity analysis on Imputation Model Same Different Same What is the result? Imputation Models 3 Proc MI Options 4 Missing Data Rate (50%) 2 Missing Data Mechanism MAR 1

25. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu Conclusions of SA on Imputation Models Models 2, 3, and 4 are all improvements over model 1, and produced DUI parameter estimates and 95% CI widths close to those of the Gold Standard. So even with 50% missing values (MAR), we are able to get a good result by using a richer imputation model. The higher percent missing values (MAR) in your data set, the more you must include additional predictors in the imputation model.

26. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu Comparison of No MI and Model 4 to the Gold Standard

27. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu Comparison of No MI and Model 4 to the Gold Standard No MI G.S. MI

28. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu Study Data Set Missing Data Mechanism 1 Missing Data Rate 2 Proc MI Data Analysis Proc MIANALYZE Results Analysis Model Imputation Model 3 Proc MI: number of MI 4 N=2N=0N=5N=10N=20 SA: Proc MI: Number of Imputations 4

29. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu SA: Data set with MAR and values missing for DUI=50%, use Model4 to do MI Test on different number of imputations –N=0 –N=2 –N=5 –N=10 –N=20 4 Proc MI: Number of Imputations

30. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu Use same imputation model (Model4), but different number of imputations to do MI on the same data set with MAR, 50% missing on DUI. N=5 50% missing on DUI N=10 50% missing on DUI N=20 50% missing on DUI ParameterESEPE PE P Intercept-1.79750.11770.0001-1.80340.12380.0001-1.78980.12040.0001 DUI0.86580.25370.00230.95630.28130.00150.99420.31760.0026 Speed0.49710.14570.00060.50810.14690.00050.50160.14650.0006 Fixed0.54480.16100.00070.53710.15980.00080.52860.15990.0010 Head-on0.56520.25220.02510.55610.25210.02740.55060.25090.0282

31. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu Sensitivity analysis on Number of Imputations Same Different What is the result? Imputation Model 3 Number of Imputation 4 Missing Data Rate (50%) 2 Missing Data Mechanism MAR 1

32. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu Conclusions of SA on Number of Imputations In our example, n=5 to 10 is enough to get good results for data set with 50% MAR on DUI. No MI (complete cases only), we would conclude that: motorcyclists with DUI had 4.2 (2.1, 8.4) times more likely killed or hospitalized than motorcyclists without DUI. But from the Gold Standard, the OR is 2.5 (1.5, 4.0)

33. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu Summary---Answers? May I use MI to deal with missing data problems for my data sets? Seems a good idea to try MI. Depend on the missing data mechanisms of variables with missing values in your data sets (however, even our results with MI for NMAR were better than No MI) How can I believe that the MI will give me the better analysis results? We found that using MI on our example gave us much better analysis results than No MI (the complete cases only) How can I get better analysis results by using MI? Understand the relationship between variables in your data sets; Know the missing data mechanisms of variables; Determine the percent of missing information; Build a reasonable imputation model; Use Proc MI options wisely

34. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu Q1. I like Denver. Q2. I like TRF. Q3. I liked the talk. Q4. I will use the MI. Missing Data Problems Everywhere Poll Results Like DenverLike TRFLiked the TalkUse MI YYYY Missing (left session early) YMissing (too nice to say “NO”) N YNYY YNNMissing (not sure yet) NMissing (daydreaming) YY Missing (fell asleep) YMissingN NNN N YY

35. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu Acknowledgment Special thanks to Dr. Mike McGlincy, who gave us helpful suggestions during our study of sensitivity analyses on imputed values and insightful comments on the analysis results.

36. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu Thank You

37. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu Questions?

38. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu Can We Improve Analysis Results for NMAR by Using a More Complex Imputation Model? Model5=Model1+age+hour +gender+safety Model4=Model1+age+hour Model1=K/H + DUI + Speed + Fixed + Head-on No MI=Complete cases only

39. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu Multiple Imputation inference involves three distinct phases: 1. The missing data are filled in m times to generate m complete data sets (using imputation model) 2. The m complete data sets are analyzed by using standard procedures (using analysis model) 3. The results from the m complete data sets are combined for the inference

40. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu Statistical Assumptions for Multiple Imputation 1. The MI procedure assumes that the data are from a continuous multivariate distribution. It also assumes that the data are from a multivariate normal distribution when the MCMC method is used According to Schafer’s MI FAQ page, MI tends to be quite forgiving of assumption for normal distribution. For example: when working with binary or ordered categorical variables, it is often acceptable to impute under a normality assumption and then round off the continuous imputed values to the nearest category. Variables whose distributions are heavily skewed may be transformed to approximate normality and then transformed back to their original scale after imputation. 2.Proc MI and Proc MIANALYZE assume that the missing data are Missing At Random (MAR) MCAR is unlikely for real world crash datasets NMAR may be shifted to MAR by using a richer imputation model to help predict missing values. Because crash datasets include many related variables that can help predict each other

41. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu

42. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu

43. 29 th TRF 2003, Denver July 14 th, 2003www.kiprc.uky.edu