1. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 1 Physician Prognostic Accuracy for In-Hospital Mortality in Percutaneous Coronary Intervention Michael E. Matheny, MD Medical Informatics Fellow Decision Systems Group Brigham & Women’s Hospital Boston, MA

2. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 2 Specific Aims Primary Hypothesis Primary Hypothesis –Accuracy of subjective physician estimations of in-hospital mortality will be similar or improved when compared with accepted objective risk assessment methods for percutaneous coronary intervention (PCI)

3. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 3 Specific Aims Secondary Hypotheses Secondary Hypotheses –Accuracy of subjective physician estimations of in- hospital major adverse cardiac events (MACE) will be similar or improved when compared with accepted objective risk assessment methods for PCI –Qualitative collection of risk factors could identify additional important risk factors currently not included in the objective risk models –Incorporating subjective physician estimates into an objective risk model will outperform either separately

4. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 4 Background Discrimination Discrimination –Ability to predict an outcome on a population level –Area under the Receiver Operating Characteristic Curve (AUC)

5. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 5 Background Calibration Calibration –Ability to predict an outcome on a case/small group level –Hosmer-Lemeshow Goodness-of-Fit Test (HL-GF) –Brier Score

6. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 6 Background Subjective vs APACHE II Medical ICU Mortality 1 Subjective vs APACHE II Medical ICU Mortality 1 –Discrimination: Objective Better –Calibration: Subjective Better –Forecasting Improves with Training Subjective vs APACHE II Medical ICU Mortality 2 Subjective vs APACHE II Medical ICU Mortality 2 –Discrimination: Subjective Better –Calibration: No Difference

7. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 7 Background Subjective vs LR Acute CHF 90 day and 1 year Mortality 3 4 Subjective vs LR Acute CHF 90 day and 1 year Mortality 3 4 –Discrimination: No Difference –Calibration:No Difference –All estimations poor Subjective vs SNAP Neonatal ICU Mortality 5 Subjective vs SNAP Neonatal ICU Mortality 5 –Discrimination:No Difference –Calibration:No Difference

8. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 8 Background Subjective + PRISM III Pediatric ICU Mortality 6 Subjective + PRISM III Pediatric ICU Mortality 6 –Discrimination: No Difference –Calibration:No Difference –Combined model Discrimination:Improved from eitherDiscrimination:Improved from either Calibration:Improved from eitherCalibration:Improved from either

9. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 9 Background Subjective vs LR Model of Post-Op mortality for Open Heart Surgeries 7 Subjective vs LR Model of Post-Op mortality for Open Heart Surgeries 7 –Discrimination:No Difference –Calibration:No Difference –Combined model Discrimination:No DifferenceDiscrimination:No Difference Calibration:No DifferenceCalibration:No Difference –Subjective assessments were more calibrated at the extremes of probability

10. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 10 Background Subjective Physician Assessments Subjective Physician Assessments –Multiple Forms of Bias 8 Ego BiasEgo Bias RegretRegret Ignoring Negative EvidenceIgnoring Negative Evidence FramingFraming

11. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 11 Background No work has been done evaluating subjective physician estimates for in-hospital mortality in percutaneous coronary interventions. No work has been done evaluating subjective physician estimates for in-hospital mortality in percutaneous coronary interventions.

12. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 12 Logistic Regression Models Logistic Regression Models NationalNational –American College of Cardiology 9 –50123 pts 1998 - 2000 RegionalRegional –Northern New England 10 –15331 pts 1994 - 1996 LocalLocal –Brigham & Women’s Hospital 11 –2804 pts 1997 - 1999 Background PCI Objective Risk Model Gold Standards

13. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 13 Recent Evaluation of Models on Local Institution Data 12 Recent Evaluation of Models on Local Institution Data 12 –Discrimination (AUC) ACC 0.90ACC 0.90 NNE 0.89NNE 0.89 BWH 0.89BWH 0.89 –Calibration (HL-GF) ACC <0.001ACC <0.001 NNE <0.001NNE <0.001 BWH <0.001BWH <0.001 Background Pilot Data

14. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 14 Background Objective Assessment Models Objective Assessment Models –Multiple Forms of Bias Population/Demographic BiasPopulation/Demographic Bias –Regional Variances Selection BiasSelection Bias –Population referral bias Temporal BiasTemporal Bias –Medical Care Standards –Data Documentation Data NoiseData Noise –Heterogeneous Data Standards –Variation in Data Element Collection –Data Entry Quality Variations

15. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 15 Background Incomplete model information? Incomplete model information? Best Described Risk Factors Best Described Risk Factors AgeCHF Sex Unstable Angina Ejection Fraction Renal Failure Recent MI Hx (CAD, DM, COPD, HTN) Hemodynamic Stability Prior CABG or PCI Intra-Aortic Balloon Pump

16. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 16 Study Design Prospective Cohort Study Prospective Cohort Study

17. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 17 Study Population Location Location –Brigham & Women’s Interventional Cardiology Suites Inclusion Criteria Inclusion Criteria –All Patients presenting for pre-operative evaluation for PCI Exclusion Criteria Exclusion Criteria –Procedural Team declines to participate in survey

18. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 18 Study Data Collection Paper Survey Paper Survey –Administration refused to allow survey to be part of medical record Subjective mortality assessment (0-100%) before and after procedure Subjective mortality assessment (0-100%) before and after procedure –Attendings –Fellows –Scrub Nurse Qualitative additional risk factors from Attendings Qualitative additional risk factors from Attendings

19. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 19 Exposures Percutaneous Coronary Transluminal Angiography with or without Coronary Stenting Percutaneous Coronary Transluminal Angiography with or without Coronary Stenting

20. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 20 Covariates/Confounders AgeCHF Sex Unstable Angina Ejection Fraction Renal Failure Recent MI Hx (CAD, DM, COPD, HTN) Hemodynamic Stability Prior CABG or PCI Intra-Aortic Balloon Pump

21. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 21 Outcomes In-Hospital Death In-Hospital Death In-Hospital MACE In-Hospital MACE

22. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 22 Analysis Plan Measure Discrimination & Calibration on local data for: Measure Discrimination & Calibration on local data for: –Objective MACE & Mortality Models NationalNational RegionalRegional LocalLocal –Subjective MACE & Mortality “Models” Pair-wise Comparison of Objective and Subjective models for statistical differences Pair-wise Comparison of Objective and Subjective models for statistical differences Develop LR model with subjective data as a covariate, and perform pair-wise comparisons with objective and subjective models to determine if new model shows improvement Develop LR model with subjective data as a covariate, and perform pair-wise comparisons with objective and subjective models to determine if new model shows improvement

23. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 23 Analysis Plan Local Institution Data Local Institution Data –~1% Death Rate –~5% MACE Rate –~200 Cases / month

24. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 24 Analysis Plan Rough Guess Sample Size Calc Sample Size Calc –Binomial Fisher’s Exact –Α = 0.05 –Power = 0.80 –Effect Size & Estimated Sample MortalityMortality –1% to 1.5% = 8150 –1% to 2% = 2514 MACEMACE –5% to 7.5% = 1550 –5% to 10% = 473

25. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 25 Analysis Plan Recruitment Multi-Center Multi-Center –Exploring recruitment possibilities from Beth- Israel and Massachusetts General Cath Labs No Termination Date No Termination Date –Implemented as Quality Control method in BWH Cath Lab

26. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 26 Limitations Sample Size Sample Size Paper Survey Paper Survey Study Population Compliance Study Population Compliance Selection Bias Selection Bias

27. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 27 Time Table IRB Approval IRB Approval –Completed Physician Survey Template Physician Survey Template –August 2005 IC Lab Tech Data Collection Training IC Lab Tech Data Collection Training –September 2005 Data Collection Data Collection –September 2005 - Open

28. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 28 Acknowledgements Co-Authors Co-Authors –Nipun Arora, MD –Lucila Ohno-Machado, MD, PhD –Frederic S. Resnic, MD, MS Funding Funding –NLM 1-T15-LM-07092

29. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 29 mmatheny@dsg.harvard.edu Michael Matheny, MD Brigham & Women’s Hospital Thorn 309 75 Francis Street Boston, MA 02115 The End