1. Comparing the accuracy of prediction methods Michael W. Kattan, Ph.D. Associate Attending Outcomes Research Scientist Memorial Sloan-Kettering Cancer Center Associate Professor of Public Health Cornell University

2. How is risk typically computed? Based on features, we make a crude tree. Most cancer staging systems do this. BT=high H=Agg And DE=E HIGH RISK LOW RISK Y N Y N

3. The problem with crude trees They are very easy to use. But they do not predict outcome optimally. »High risk groups are very heterogeneous. –A single risk factor may qualify a patient as high risk. Other approaches, like a Cox regression model, predict more accurately.

4. Some simple steps that will make a difference 1. Build the most accurate model possible. 2. Take model to bedside »As a nomogram, »In stand-alone software (desktop, handheld, web) »Built into the electronic medical record Doing this will predict patient outcome more accurately, resulting in »better patient counseling »better treatment decision making

5. Desirable characteristics of an error measure Understandable/interpretable Sensitive to model improvement Model-free Unaffected by censoring

6. CONCORDANCE INDEX (censored data) probability that, given two randomly drawn patients, the patient who fails first had a higher probability of failure. assumes that the patient with the shorter follow-up fails does not apply if both patients fail at the same time, or the censored patient has shorter follow-up. Usable patient pairs with consistent outcome Usable patient pairs usable patient pair - patient with the shorter follow-up must fail consistent outcome - failure more likely for the shorter follow-up patient tied predicted probabilities get 1/2 (Harrell, 1982)

7. Gastric Cancer Disease-Specific Survival by AJCC Stage

8. Gastric Cancer Disease-Specific Survival Nomogram MM SM MP SS S2 S3 S1 Kattan et al., JCO, 2003

9. How to tell if we are doing any better than existing models? Compare jackknife predicted probabilities of new model to existing model predictions: MethodConcordance Index AJCC Stage0.77 Nomogram(jackknife)0.80 (p<0.001).

10. How to tell if we are doing any better than existing models? Validation dataset Concordance Index MethodOriginalDutch Trial (n=459) AJCC Stage0.770.75 Nomogram0.800.77 (p<0.001)

11. Heterogeneity within stages AJCC IV (32) II (117) IA (102) IB (115) IIIA (69) IIIB (24)

12. Carroll et al., J. Urol, 2004

13. Nomograms for clinical trial design Example: CALGB 90203, preoperative therapy for patients at high risk of failure following surgery for prostate cancer < 60%

14. Continuous Models vs. Staging/Grouping Systems ModelComparatorCI (M vs C) PreopL/I/H Risk Groups0.67 vs. 0.64 Preop + IL6/TGFβ1 L/H Risk Groups0.84 vs. 0.73 Pre XRTL/I/H Risk Groups0.76 vs. 0.69 Melanoma SLN+AJCC Stage0.69 vs. 0.66 Pancreatic CaAJCC Stage0.64 vs. 0.56 Gastric CaAJCC Stage0.77 vs. 0.75 Breast CaNPI Groups0.69 vs. 0.64 SarcomaCART Groups0.77 vs. 0.74

15. Software to facilitate real-time predictions Software is free from http://www.mskcc.org/predictiontools Prostate, renal cell, gastric, sarcoma, breast, lung available now. Pancreatic, melanoma available soon. Software for the Palm Pilot, PocketPC, and Windows Desktop Computers Models

16. Levels of discrimination for some prediction tools 0.5 0.6 0.70.8 0.9 1.0 Preoperative Zero ability to predict Discriminate Perfectly Brachytherapy Radiotherapy Positive subsequent biopsy Postoperative Survival with progressive metastatic disease Indolent Ca LN+ OC Preop with IL-6 & TGFβ1 Sarcoma Renal Cell Gastric Pancreatic Lung Melanoma

17. When The Patient Wants A Prediction, What Options Does The Clinician Have? Quote an overall average to all patients Deny ability to predict at the individual patient level Assign the patient to a risk group, i.e. high, intermediate, or low Apply a model Predict based on knowledge and experience

18. Nomogram for predicting the likelihood of additional nodal metastases in breast cancer patients with a positive sentinel node biopsy Vanzee K, et al., Ann Surg Oncol., 2003. Lobular

19. Breast Cancer Prediction: 17 Clinicians vs. Model on 33 Patients Model CI 0.72 Clinician CI 0.54 Sensitivity: Proportion of women with positive nodes predicted to have positive nodes Specificity: Proportion of women with negative nodes predicted to have negative nodes

20. ROC Curves Individual Clinicians and Model Areas 0.75 0.72 Model 0.68 0.65 0.63 0.59 0.58 0.55 0.53 0.52 0.50 0.49 0.47 0.43 0.42 0.40

21. Conclusions Concordance index is a useful metric by which to compare rival prediction models. The decision whether to use any model vs. assume homogeneous risk is context dependent.

22. Collaborators Methods »Biostatistics –Mithat Gonen –Glenn Heller –Peter Bach –Colin Begg –Frank Harrell »Informatics –Paul Fearn –David Ladanyi –John Davey –Pat Turi –Jacob Rockowitz –Drumbeat Digital Applications »Peter Scardino »Murray Brennan »Marty Karpeh »Kim VanZee »Dan Coit