Calibration Drift Among Regression and Machine Learning Models for Hospital Mortality S117: Acute Setting Predictive Analytics Sharon E. Davis, MS PhD Candidate Department of Biomedical Informatics Vanderbilt University

Disclosure I and my spouse have no relevant relationships with commercial interests to disclose. AMIA 2017 | amia.org

Motivation Evolving role of clinical prediction models EHR-based models have access to more data, can support more complexity in real-time Move from classification to individual-level predicted probabilities Increasing focus on calibration Discrimination Ability to separate cases and non-cases Supports risk stratification Calibration Alignment between predicted and observed probabilities More difficult to assess, evolving metrics Calibration hierarchy and associated metrics1 Agreement within each covariate pattern Agreement of prediction and outcome rate among similar patients No systematic over/underprediction or over/underfitting Agreement on average Strong Moderate Weak Mean 1 Van Calster et al. “A calibration hierarchy for risk models was defined: from utopia to empirical data.” JCE 2016. AMIA 2017 | amia.org

Motivation Drifting performance over time, particularly in terms of calibration Driving forces Population shifts in outcome rate, case mix Clinical practice and documentation changes (predictor-outcome associations) Limited understanding Focus on logistic models, despite proliferation of more complex modeling methods and some evidence that methods impact drift Focus on crude measures of calibration, despite evidence that metrics impact understanding Limited exploration of drivers of calibration drift Davis SE, Lasko TA, Chen G, Siew ED, Matheny ME. 2017. “Calibration Drift in Regression and Machine Learning Models for Acute Kidney Injury.” JAMIA. 24(6): 1052-1061. AMIA 2017 | amia.org

Objectives Compare performance over time of prediction models developed using common frequentist statistical methods and machine learning techniques Expanded set of modeling methods Calibration measured at multiple levels of stringency Link shifts in patient populations with model performance to identify drivers of calibration drift across models Event rate shift Case mix shift Predictor-outcome association shift AMIA 2017 | amia.org

Study Population National cohort of admissions to VA facilities, 2006-2013 Total sample size – 1,893,284 admissions One year development period, multi-year validation period Outcome: 30-day mortality after hospital admission Predictors based on prior modeling Demographics (3) Laboratory results (20) Diagnoses (32) Vitals (7) Care utilization (3) Admission type (1) AMIA 2017 | amia.org

Methods – Model Development Training set – admissions occurring in 2006 (n = 235,548) Modeling methods Parallel models developed with each method Hyperparameters selected with 5-fold cross-validation Internal validation with 200 bootstrap iterations Logistic L1-L2 regularized logistic (elastic net) L1 regularized logistic (lasso) Random forest L2 regularized logistic logistic (ridge) Neural network AMIA 2017 | amia.org

Methods – Model Validation Validation period: 2007-2013 (n = 1,657,736) AMIA 2017 | amia.org

Methods – Model Validation Discrimination - AUC Calibration Calibration Plot Metrics – flexible calibration curves, estimated calibration index (ECI) 1 Metrics: Cox recalibration intercept and slope Metric: observed to expected outcome ratio (O:E) Strong Moderate Weak Mean 1 Van Hoorde et al. “A spline-based tool to assess and visualize the calibration of multiclass risk predictions.” JBI. 2015. AMIA 2017 | amia.org

Methods – Extending Flexible Calibration Curves Determining regions of calibration Tracking regions over time Rescaling regions by volume of data Time Data concentrated at low predicted probabilities Rescaled regions of calibration by proportion of observations in each region Assessed magnitude of miscalibration using within region ECIs AMIA 2017 | amia.org 10

Methods – Population Data Shifts Event rate shift Definition: Changes in the prevalence of the outcome Assessment: Distribution of mortality rate over time Case mix shift Definition: Changes in the distribution of risk factors in the population Assessment: Distributions of predictors over time Discrimination and model structure of membership models1 Predictor-outcome association shifts Definition: Changes in form or magnitude of relationships between risk factors and the outcome Assessment: Changes in the structure of models refit in each 3-month period Differences that arise over time between the population on which the model was developed and the population on which the model is applied 1 Debray et al. “A new framework to enhance the interpretation of external validation studies of clinical prediction models.” JCE 2015.. AMIA 2017 | amia.org

Discrimination Over Time AMIA 2017 | amia.org

Calibration Over Time – Observed to Expected Outcome Ratio Ideal value: 1 AMIA 2017 | amia.org

Calibration Over Time – Estimated Calibration Index Ideal value: 0 AMIA 2017 | amia.org

Calibration Over Time – Rescaled Regions of Calibration AMIA 2017 | amia.org

Linking Population Shifts and Performance Event rate shift dominated by seasonal variation in mortality rate Model Outcome Rate Shift Case Mix Shift Association Shift Logistic regression u L1 logistic L2 logistic L1-L2 logistic Random forest Neural network Susceptibility – u High u Moderate u Low Mortality rate declined from 5.0% to 4.8% over study period AMIA 2017 | amia.org

Linking Population Shifts and Performance Outcome rate shift dominated by seasonal variation in mortality rate Limited evidence of predictor-outcome association shifts Model Outcome Rate Shift Case Mix Shift Association Shift Logistic regression u L1 logistic L2 logistic L1-L2 logistic Random forest Neural network Susceptibility – u High u Moderate u Low Mortality rate declined from 5.0% to 4.8% over study period AMIA 2017 | amia.org

Linking Population Shifts and Performance Outcome rate shift dominated by seasonal variation in mortality rate Limited evidence of predictor-outcome association shifts Case mix shift occurred throughout the validation period Model Outcome Rate Shift Case Mix Shift Association Shift Logistic regression u L1 logistic L2 logistic L1-L2 logistic Random forest Neural network Membership Model Discrimination Susceptibility – u High u Moderate u Low Mortality rate declined from 5.0% to 4.8% over study period AMIA 2017 | amia.org

Integration With Additional Research Corresponding analysis in separate cohort experiencing different population data shifts Acute kidney injury models experienced Event rate shift Case mix shift Predictor-outcome association shift Age inclusion in L1 penalized logistic regression model for AKI Estimated calibration index for AKI models Davis SE, Lasko TA, Chen G, Siew ED, Matheny ME. 2017. “Calibration Drift in Regression and Machine Learning Models for Acute Kidney Injury.” JAMIA. 24(6): 1052-1061. AMIA 2017 | amia.org

Integration With Additional Research Corresponding analysis in separate cohort experiencing different population data shifts Acute kidney injury models experienced Event rate shift Case mix shift Predictor-outcome association shift Model Outcome Rate Shift Case Mix Shift Association Shift Logistic regression u L1 logistic L2 logistic L1-L2 logistic Random forest Neural network Susceptibility – u High u Moderate u Low Davis SE, Lasko TA, Chen G, Siew ED, Matheny ME. 2017. “Calibration Drift in Regression and Machine Learning Models for Acute Kidney Injury.” JAMIA. 24(6): 1052-1061. AMIA 2017 | amia.org

Limitations Statistical versus clinically meaningful miscalibration Additional modeling methods remain to be considered Limited population data shift scenarios explored Variable magnitudes of population data shifts Variable combinations of event rate, case mix, association shifts AMIA 2017 | amia.org

Conclusions Calibration drift varies by modeling method and form of underlying population shifts Selection of calibration metrics impacts understanding of drift across methods Model updating strategies will need to be tailored to the unique vulnerabilities of modeling methods AMIA 2017 | amia.org

Funding Agencies Research Team Michael Matheny, MD, MS, MPH VA HSR&D IIR 11-292 VA HSR&D IIR 13-052 NLM 5T15LM007450 NLM 1R21LM011664-01 Research Team Michael Matheny, MD, MS, MPH Guanhua Chen, PhD Thomas Lasko, MD, PhD Department of Biomedical Informatics AMIA 2017 | amia.org

Email: sharon.e.davis@vanderbilt.edu Thank you! Email: sharon.e.davis@vanderbilt.edu Department of Biomedical Informatics