1. Chapter 9.2 ROC Curves How does this relate to logistic regression?

2. 2 Two Types of Error False negative (“miss”), FN alarm doesn’t sound but person is carrying metal = 1-sensitivity False positive (“false alarm”), FP alarm sounds but person is not carrying metal =1-specificity Slide copied from : Lecture on Cost-Sensitive Classifier Evaluation by Robert Holte at the Computing Science Dept. University of Alberta

3. 3 ROC Receiver Operating Characteristic (historic name from radar studies) Relative Operating Characteristic (psychology, psychophysics) Operating Characteristic (preferred by some) Slide adapted from : An Overview of Contemporary ROC Methodology in Medical Imaging and Computer-Assist Modalities by Robert F. Wagner, Ph.D., OST, CDRH, FDA

4. 4 Non-diseased patient Diseased patient Test result value or likelihood that patient is diseased –>P( Y = 1) =  (based on proc logistic) Threshold Slide adapted from :An Overview of Contemporary ROC Methodology in Medical Imaging and Computer-Assist Modalities by Robert F. Wagner, Ph.D., OST, CDRH, FDA

5. 5 Non-diseased patient Diseased patient Test result value ( Logistic P(Y=1) =  ) Typical Results of Testing Slide adapted from :An Overview of Contemporary ROC Methodology in Medical Imaging and Computer-Assist Modalities by Robert F. Wagner, Ph.D., OST, CDRH, FDA Threshold Negative Test  <0.5 Positive test  >0.5 False Negatives False Positives

6. 6 Threshold TPF, sensitivity FPF, 1-specificity less aggressive mindset Non-diseased patient Diseased patient Slide adapted from :An Overview of Contemporary ROC Methodology in Medical Imaging and Computer-Assist Modalities by Robert F. Wagner, Ph.D., OST, CDRH, FDA  >0.7 is positive test

7. 7 Threshold TPF, sensitivity FPF, 1-specificity moderate mindset Non-diseased patient Diseased patient Slide adapted from :An Overview of Contemporary ROC Methodology in Medical Imaging and Computer-Assist Modalities by Robert F. Wagner, Ph.D., OST, CDRH, FDA  >0.5 is positive test

8. 8 Threshold TPF, sensitivity FPF, 1-specificity more aggressive mindset Non-diseased patient Diseased patient Slide adapted from :An Overview of Contemporary ROC Methodology in Medical Imaging and Computer-Assist Modalities by Robert F. Wagner, Ph.D., OST, CDRH, FDA  >0.3 is positive test

9. 9 Threshold Non-diseased patients Diseased patients TPF, sensitivity FPF, 1-specificity Entire ROC curve Slide adapted from :An Overview of Contemporary ROC Methodology in Medical Imaging and Computer-Assist Modalities by Robert F. Wagner, Ph.D., OST, CDRH, FDA  >0 is positive test

10. 10 TPF, sensitivity FPF, 1-specificity Entire ROC curve Skill to predict Y=1 correctly chance line Slide adapted from :An Overview of Contemporary ROC Methodology in Medical Imaging and Computer-Assist Modalities by Robert F. Wagner, Ph.D., OST, CDRH, FDA

11. 11  Suppose the n individuals undergo a test for predicting the event and the test is based on the estimated probability of the event (  ).  Higher values of this estimated probability are assumed to be associated with the event.  A receiver operating characteristic (ROC) curve can be constructed by varying the cut-point that determines which estimated event probabilities are considered to predict the event.  The statistic c estimates the area under the ROC curve.

12. ROC Curves in SAS /* area under the curve is c statistic – generally speaking bigger c is better */ proc logistic descending; model event=diabetes gender diabetes_gender/outroc=roc1; RUN; RUN; /* plotting the roc curve */ symbol1 i=join v=none c=blue; proc gplot data=roc1; proc gplot data=roc1; title 'ROC Curve'; title 'ROC Curve'; plot _sensit_*_1mspec_=1 / vaxis=0 to 1 by.1 cframe=ligr; plot _sensit_*_1mspec_=1 / vaxis=0 to 1 by.1 cframe=ligr; run; run;

13. 13 Model event=diabetes gender diabetes_gender Association of Predicted Probabilities and Observed Responses Percent Concordant 47.7 Somers' D 0.206 Percent Discordant 27.1 Gamma 0.276 Percent Tied 25.2 Tau-a 0.055 Pairs 41741 c 0.603 Measures area under ROC curve

14. 14 Model event=diabetes gender diabetes_gender

15. 15 ( Chest film study by E. James Potchen, M.D., 1999 )

16. 16

17. 17 ROC curve in Logistic Regression Goto: www.biostat.umn.edu/~susant/PH6415DATA.html www.biostat.umn.edu/~susant/PH6415DATA.html C Slim down data to contain any event, history of diabetes, gender and history of hypertension. Slim down data to contain any event, history of diabetes, gender and history of hypertension. Use Proc Logistic to model the relationship between cardiac event and diabetes history, gender and hypertension history. Use Proc Logistic to model the relationship between cardiac event and diabetes history, gender and hypertension history. Use Gplot to plot the ROC curve. Use Gplot to plot the ROC curve. Compare the value of c to Compare the value of c to 0.603 <- our previous value.