Prediction Models in Medicine Clinical Decision Support The Road Ahead Chapter 10

Overview Prediction models currently in use in Clinical Decision Support What models are used (or not used) in the community How are the models evaluated Examples of currently used prediction models

Limited adaptation of learning algorithms in practice Most clinical decision support systems do not use machine-learning/data mining techniques 1. Data is not available or is not structured enough 2. Learning techniques are not well disseminated or well evaluated enough 3. Rules defined by experts are more understandable to clinicians

Model Preference “Simple, understandable models” are preferred Linear and Logistic Regression is by far the most popular SVMs, Neural Networks, and other “sophisticated models” are not very popular Unsupervised Learning is not used at all

Model Evaluation Discrimination How well the model discriminates positive and negative cases How large is P(1|Positive Case) P(0|Negative Case) Based on the ROC(Receiver operating characteristic) curve

Model Evaluation Calibration How close is the model’s estimated probability to the “true” underlying probability For logistic regression, calibration is assessed by Hosmer-Lemeshow goodness-of-fit test

Case Study 1: Prognosis of ICU Mortality APACHE (Acute Physiology and Chronic Health Evaluation) series of models Predict the individual patient's risk of hospital death, based on a variety of physiological variables History: APACHE (1981): Expert-based scoring system APACHE II (1985): Logistic Regression on 5,815 cases from 13 hospitals APACHE III (1991): Logistic Regression on 17,440 cases based on 40 hospitals. [Commercial product]

Case Study 1: Prognosis of ICU Mortality Large number of reviews and external evaluations show good discrimination, but variable calibration Other systems (more popular in Europe) SAPS-II MPM-II Multiple studies compare LR to ANN Some studies suggest that the models are equivalent Some suggest that ANN achieves superior discrimination

Case Study 2: Cardiovascular Disease Risk Estimates the risk of developing future heart disease Based on most recent 10-year heart disease data from Framingham cohort (in US) Uses Logistic Regression External validation shows good discrimination and moderate calibration (but limited to similar demographic) The model is used to determine the risk factors for heart disease (used to generate guidelines for care)

Case Study 3: Pneumonia Severity of Illness Index Predicts the risk of death within 30 days for adult patients with pneumonia Developed by Pneumonia Patient Outcome Research Team (PORT) 1997 Logistic Regression! The model was validated over 50,000 patients in 275 US and Canadian hospitals Using this model, 26 to 31 percent of patients can be treated safely as outpatients Savings of more than 1.2 Billion dollars per year in US