SC-PM6: Prediction Models in Medicine: Development, Evaluation and Implementation Michael W. Kattan, Ph.D. Ewout Steyerberg, Ph.D. Brian Wells, M.S., M.D.

When The Patient Wants A Prediction, What Options Does The Clinician Have? Quote an overall average to all patients Deny ability to predict at the individual patient level Assign the patient to a risk group, i.e. high, intermediate, or low Apply a model Predict based on knowledge and experience

How do we typically compute risk? Based on features, we make a crude tree. Most cancer staging systems do this. BT=high H=Agg And DE=E HIGH RISK LOW RISK Y N Y N

The problem with crude trees They are very easy to use. But they do not predict outcome optimally. – High risk groups are very heterogeneous. – A single risk factor may qualify a patient as high risk. Other approaches, like a Cox regression statistical model, predict more accurately. Readily applicable presentation essential.

Biopsy Gleason Grade  2+   3+  4  2+3  4+ ? Total Points Month Rec. Free Prob  2 Clinical Stage T1cT1ab T2aT2cT3a T2b Points PSA Preoperative Nomogram for Prostate Cancer Recurrence Instructions for Physician: Locate the patient’s PSA on the PSA axis. Draw a line straight upwards to the Points axis to determine how many points towards recurrence the patient receives for his PSA. Repeat this process for the Clinical Stage and Biopsy Gleason Sum axes, each time drawing straight upward to the Points axis. Sum the points achieved for each predictor and locate this sum on the Total Points axis. Draw a line straight down to find the patient’s probability of remaining recurrence free for 60 months assuming he does not die of another cause first. Instruction to Patient: “Mr. X, if we had 100 men exactly like you, we would expect between and to remain free of their disease at 5 years following radical prostatectomy, and recurrence after 5 years is very rare.”  1997Michael W. Kattan and Peter T. Scardino Kattan MW et al: JNCI 1998; 90:

Some simple steps that will make a difference 1. Build the most accurate model possible. 2. Take model to bedside – As a nomogram, – In stand-alone software (desktop, handheld, web) – Built into the electronic medical record Doing this will predict patient outcome more accurately, resulting in – better patient counseling – better treatment decision making

What is a Nomogram? A prediction device Usually a regression model presented graphically Continuous-based prediction

Making a nomogram Usually a regression model (Cox or logistic) – Could consider machine learning techniques (neural nets, optimized trees like CART) Keep continuous variables continuous but relax linearity assumptions P-values for predictors don’t matter No variable selection or univariable screening Bottom line is its predictive accuracy

CaPSURE Heterogeneity within Risk Groups Risk Group Nomogram Values by Prostate Cancer Risk Group Preoperative Nomogram Predicted Probability LowIntermediateHigh J Urol Apr;173(4):

Kattan MW, et al., J Clin. Oncol., 2000.

10 50 T2c T3c D Conformal Radiation Therapy Nomogram for PSA Recurrence Kattan MW, et al., J Clin. Oncol., 2000.

Discrimination on Cleveland Clinic data (N=912) Kattan MW et al., J Clin. Oncol.,

Why Nomograms Matter: A Particular Example Mr. X, from the Cleveland Clinic: – PSA=6, clinical stage = T2c, biopsy Gleason sum=9, planned dose of 66.6 Gy without neoadjuvant hormones Shipley risk stratification: 5 yr. Surgery nomogram: 5yr. Radiation therapy nomogram: 5yr. Kattan MW, et al., J Clin. Oncol., 2000.

When The Patient Wants A Prediction, What Options Does The Clinician Have? Quote an overall average to all patients Deny ability to predict at the individual patient level Assign the patient to a risk group, i.e. high, intermediate, or low Apply a model Predict based on knowledge and experience

Urologists vs. Preoperative Nomogram 10 case descriptions from 1994 MSKCC patients presented to 17 urologists – In addition to PSA, biopsy Gleason grades, and clinical stage, urologists were provided with patient age, systematic biopsy details, previous biopsy results, and PSA history. Preoperative nomogram was provided. Urologists were asked to make their own predictions of 5 year progression-free probabilities with or without use of the preoperative nomogram. Concordance indices: – Nomogram = 0.67 – Urologists = 0.55, p<0.05 Ross P et al., Semin Urol Oncol, 2002.

Nomogram for predicting the likelihood of additional nodal metastases in breast cancer patients with a positive sentinel node biopsy Vanzee K, et al., Ann Surg Oncol., 2003.

Breast Cancer Prediction: 17 Clinicians vs. Nomogram on 33 Patients Nomogram AUC 0.72 Clinician AUC 0.54 Sensitivity: Proportion of women with positive nodes predicted to have positive nodes Specificity: Proportion of women with negative nodes predicted to have negative nodes

ROC Curves Individual Clinicians and Nomogram Areas Nomogram

19 All of these patients received radical prostatectomy, are now experiencing rising PSA, and have not started ADT. AgeRace Clinical Stage Biopsy PSA Biopsy Gleason Sum Adjuvant Radiation Months from Surgery to Today Pathological Gleason SumCap.invECEMarginSVILN PSA at BCR PSA Doubling time (months) If you had 100 patients just like this one, how many do you think would have a positive bone scan 1 year from today if left untreated? (Enter a number between 0 and 100) 67WT2A2.77N PPNPP WT2B12.77N PNPNN WT1C20.06N PNPPP WT1C13.27N PNPNN WT2C101.05N PPPPP WT2B11.14N PPPPN WT2B23.910N PPPPP WT2A13.56N PPPNN WT1C25.86N PPPNN WT1C13.56N PPPNN WT1C10.17N PNPNN WT1C26.86N PPPNN WT2A4.57N PPNNN WT1C4.77N PPNPN WT1C10.76N PPNNN WT1C7.46N PNNNN WT1C5.07N PNNNN WT1C13.37N PNNNN WT1C14.69N PPPPN WT1C14.68N PNPNN WT1C15.89N PNPNN WT2A7.17N PPNNN WT1C4.67N PNPNN WT2A4.47N PNNNN WT1C4.27N PPPPN

Nomogram to Predict Bone Scan Positivity (Cont.) Slovin SF, et al. Clin Can Res. 2005;11: Nonogram Used to Predict Patient-Specific Probabilities of Metastasis-free Survival at 1 and 2 Years, and the Median Progression-free Survival Time AUC=0.69 Points bPSA, ng/mL PSADT, mo Gleason Total Points 1-Year PFS Year PFS T Stage Median PFS

Predictions by Docs and Nomogram, by Patient

Survey Results

Discrimination accuracy for nomogram and docs

Feedback recall, overconfident, hindsight bias, chance Biases in Human Prediction adapted from Hogarth, 1988

Conclusions for Part 1 Continuous statistical prediction models offer accuracy advantages over: – Crude risk groups – Clinical judgment Remaining challenges: – How to evaluate accuracy – If accuracy is acceptable, how to deploy/disseminate