Prognostic models in the ICU From development to clinical practice L. Minne, MSc. Dr. S. Eslami, PharmD Dr. D.A. Dongelmans, MD Prof. Dr. S.E.J.A. de Rooij, MD Prof. Dr. A. Abu-Hanna Dept. of Medical Informatics Dept. of Intensive Care Academic Medical Center Amsterdam, the Netherlands Prof. Dr. E. de Jonge, MD Dept. of Intensive Care Leiden University Medical Center Leiden, the Netherlands

Use of prognostic models 1) Benchmarking 2) Decision-making Expected mortality: 30%12% SMR: Hospital 1Hospital 2 Observed mortality: 25%15% Estimates from prognostic model

Use of prognostic models Your probability to survive is: (SAPS II score * ) + ( * (ln (SAPS II score + 1))) 1) Benchmarking 2) Decision-making

Barriers for use in clinical practice  Lack of evidence for: External validity Clinical credibility Impact on decisions and patient outcomes  Selffulfilling prophecy  Population level vs individual level

Overview of our research project 1) Identify 1) Identify prognostic models, their validity and use in clinical practice benchmarking 2) Assess prognostic model behaviour over time + effects on benchmarking decision-making 3) Assess clinicians’ predictions, (need for) prognostic models, their validity and impact in decision-making

Red (critical) zone Yellow (warning) zone Green (safe) zone Yellow (warning) zone Red (critical) zone Time Standardized Mortality Ratio Mean value Upper control limit (usually at 3 sigma) Lower control limit (usually at 3 sigma) > mean + 4 sigma mean + 2 sigma : mean + 4 sigma mean : mean + 2 sigma mean : mean - 2 sigma mean - 2 sigma : mean - 4 sigma < mean - 4 sigma Benchmarking – Temporal validation

SMR > 1 in 15% of the hospitals

Benchmarking – Temporal validation Time p=16 Data used for recalibration p=19 Data used for recalibration Effect of continuous updating (first level recalibration)

Benchmarking – Temporal validation SMR > 1 in 35% of the hospitals  effect on quality of care assessment!

Age Gender... Demography Physiology Laboratory... Admission Mortality (Length of Stay) (...) Outcomes organ scores day1 organ scores day2 organ scores day3 … During Stay SAPS score SOFA Decision-making – Model development

25 Day 3Day Day 4Day 1Day Renal Hepatic Circulatory Respiratory Neurological Coagulation SAPS 998 SOFA score 12 Decision-making – Model development HHMHH d = 3

LP = a 0 + a 1 SAPS + a 2 admission_type + a 3 day + A 4 number_of_readmissions + + b 1 Pattern 1 + b 2 Pattern 2 + … Example at day 3 LP = *SAPS * * SOFA{H,H} SOFA{M,H,H} Decision-making – Model development

Decision-making – Model performance

Decision-making – The end-of-life decision-making process  Observation of multidisciplinary meetings  poorly structured no clear guidelines  Factors (implicitly) considered in decision: Degree of organ failure Patient preferences Severity of illness Chance of cognitive limitations  Wish to receive objective information

Conclusions and future work  Decision-making process unstructured  Possible role for mathematical models  But… insufficient evidence on their impact and external validation  Before-after study to measure impact on decision-making

Any questions?

Decision-making – Human predictions Kappa = %NursesPhysiciansAUC Var6-7%7-8%