1. Chapter 9.2 ROC Curves How does this relate to logistic regression?

2. 2 Two Types of Error False negative (“miss”), FN alarm doesn’t sound but person is carrying metal = 1-sensitivity False positive (“false alarm”), FP alarm sounds but person is not carrying metal =1-specificity Slide copied from : Lecture on Cost-Sensitive Classifier Evaluation by Robert Holte at the Computing Science Dept. University of Alberta

3. 3 ROC Receiver Operating Characteristic (historic name from radar studies) Relative Operating Characteristic (psychology, psychophysics) Operating Characteristic (preferred by some) Slide adapted from : An Overview of Contemporary ROC Methodology in Medical Imaging and Computer-Assist Modalities by Robert F. Wagner, Ph.D., OST, CDRH, FDA

4. 4 Non-diseased patient Diseased patient Test result value or likelihood that patient is diseased –>P( Y = 1) =  (based on proc logistic) Threshold Slide adapted from :An Overview of Contemporary ROC Methodology in Medical Imaging and Computer-Assist Modalities by Robert F. Wagner, Ph.D., OST, CDRH, FDA

5. 5 Non-diseased patient Diseased patient Test result value ( Logistic P(Y=1) =  ) Typical Results of Testing Slide adapted from :An Overview of Contemporary ROC Methodology in Medical Imaging and Computer-Assist Modalities by Robert F. Wagner, Ph.D., OST, CDRH, FDA Threshold Negative Test  <0.5 Positive test  >0.5 False Negatives False Positives

6. 6 Threshold TPF, sensitivity FPF, 1-specificity less aggressive mindset Non-diseased patient Diseased patient Slide adapted from :An Overview of Contemporary ROC Methodology in Medical Imaging and Computer-Assist Modalities by Robert F. Wagner, Ph.D., OST, CDRH, FDA  >0.7 is positive test

7. 7 Threshold TPF, sensitivity FPF, 1-specificity moderate mindset Non-diseased patient Diseased patient Slide adapted from :An Overview of Contemporary ROC Methodology in Medical Imaging and Computer-Assist Modalities by Robert F. Wagner, Ph.D., OST, CDRH, FDA  >0.5 is positive test

8. 8 Threshold TPF, sensitivity FPF, 1-specificity more aggressive mindset Non-diseased patient Diseased patient Slide adapted from :An Overview of Contemporary ROC Methodology in Medical Imaging and Computer-Assist Modalities by Robert F. Wagner, Ph.D., OST, CDRH, FDA  >0.3 is positive test

9. 9 Threshold Non-diseased patients Diseased patients TPF, sensitivity FPF, 1-specificity Entire ROC curve Slide adapted from :An Overview of Contemporary ROC Methodology in Medical Imaging and Computer-Assist Modalities by Robert F. Wagner, Ph.D., OST, CDRH, FDA  >0 is positive test

10. 10 TPF, sensitivity FPF, 1-specificity Entire ROC curve Skill to predict Y=1 correctly chance line Slide adapted from :An Overview of Contemporary ROC Methodology in Medical Imaging and Computer-Assist Modalities by Robert F. Wagner, Ph.D., OST, CDRH, FDA

11. 11  Suppose the n individuals undergo a test for predicting the event and the test is based on the estimated probability of the event (  ).  Higher values of this estimated probability are assumed to be associated with the event.  A receiver operating characteristic (ROC) curve can be constructed by varying the cut-point that determines which estimated event probabilities are considered to predict the event.  The statistic c estimates the area under the ROC curve.

12. ROC Curves in SAS /* area under the curve is c statistic – generally speaking bigger c is better */ proc logistic descending; model event=diabetes gender diabetes_gender/outroc=roc1; RUN; RUN; /* plotting the roc curve */ symbol1 i=join v=none c=blue; proc gplot data=roc1; proc gplot data=roc1; title 'ROC Curve'; title 'ROC Curve'; plot _sensit_*_1mspec_=1 / vaxis=0 to 1 by.1 cframe=ligr; plot _sensit_*_1mspec_=1 / vaxis=0 to 1 by.1 cframe=ligr; run; run;

13. 13 Model event=diabetes gender diabetes_gender Association of Predicted Probabilities and Observed Responses Percent Concordant 47.7 Somers' D 0.206 Percent Discordant 27.1 Gamma 0.276 Percent Tied 25.2 Tau-a 0.055 Pairs 41741 c 0.603 Measures area under ROC curve

14. 14 Model event=diabetes gender diabetes_gender

15. 15 ( Chest film study by E. James Potchen, M.D., 1999 )