Chapter 9.2 ROC Curves How does this relate to logistic regression?

2 Two Types of Error False negative (“miss”), FN alarm doesn’t sound but person is carrying metal = 1-sensitivity False positive (“false alarm”), FP alarm sounds but person is not carrying metal =1-specificity Slide copied from : Lecture on Cost-Sensitive Classifier Evaluation by Robert Holte at the Computing Science Dept. University of Alberta

3 ROC Receiver Operating Characteristic (historic name from radar studies) Relative Operating Characteristic (psychology, psychophysics) Operating Characteristic (preferred by some) Slide adapted from : An Overview of Contemporary ROC Methodology in Medical Imaging and Computer-Assist Modalities by Robert F. Wagner, Ph.D., OST, CDRH, FDA

4 Non-diseased patient Diseased patient Test result value or likelihood that patient is diseased –>P( Y = 1) =  (based on proc logistic) Threshold Slide adapted from :An Overview of Contemporary ROC Methodology in Medical Imaging and Computer-Assist Modalities by Robert F. Wagner, Ph.D., OST, CDRH, FDA

5 Non-diseased patient Diseased patient Test result value ( Logistic P(Y=1) =  ) Typical Results of Testing Slide adapted from :An Overview of Contemporary ROC Methodology in Medical Imaging and Computer-Assist Modalities by Robert F. Wagner, Ph.D., OST, CDRH, FDA Threshold Negative Test  <0.5 Positive test  >0.5 False Negatives False Positives

6 Threshold TPF, sensitivity FPF, 1-specificity less aggressive mindset Non-diseased patient Diseased patient Slide adapted from :An Overview of Contemporary ROC Methodology in Medical Imaging and Computer-Assist Modalities by Robert F. Wagner, Ph.D., OST, CDRH, FDA  >0.7 is positive test

7 Threshold TPF, sensitivity FPF, 1-specificity moderate mindset Non-diseased patient Diseased patient Slide adapted from :An Overview of Contemporary ROC Methodology in Medical Imaging and Computer-Assist Modalities by Robert F. Wagner, Ph.D., OST, CDRH, FDA  >0.5 is positive test

8 Threshold TPF, sensitivity FPF, 1-specificity more aggressive mindset Non-diseased patient Diseased patient Slide adapted from :An Overview of Contemporary ROC Methodology in Medical Imaging and Computer-Assist Modalities by Robert F. Wagner, Ph.D., OST, CDRH, FDA  >0.3 is positive test

9 Threshold Non-diseased patients Diseased patients TPF, sensitivity FPF, 1-specificity Entire ROC curve Slide adapted from :An Overview of Contemporary ROC Methodology in Medical Imaging and Computer-Assist Modalities by Robert F. Wagner, Ph.D., OST, CDRH, FDA  >0 is positive test

10 TPF, sensitivity FPF, 1-specificity Entire ROC curve Skill to predict Y=1 correctly chance line Slide adapted from :An Overview of Contemporary ROC Methodology in Medical Imaging and Computer-Assist Modalities by Robert F. Wagner, Ph.D., OST, CDRH, FDA

11  Suppose the n individuals undergo a test for predicting the event and the test is based on the estimated probability of the event (  ).  Higher values of this estimated probability are assumed to be associated with the event.  A receiver operating characteristic (ROC) curve can be constructed by varying the cut-point that determines which estimated event probabilities are considered to predict the event.  The statistic c estimates the area under the ROC curve.

ROC Curves in SAS /* area under the curve is c statistic – generally speaking bigger c is better */ proc logistic descending; model event=diabetes gender diabetes_gender/outroc=roc1; RUN; RUN; /* plotting the roc curve */ symbol1 i=join v=none c=blue; proc gplot data=roc1; proc gplot data=roc1; title 'ROC Curve'; title 'ROC Curve'; plot _sensit_*_1mspec_=1 / vaxis=0 to 1 by.1 cframe=ligr; plot _sensit_*_1mspec_=1 / vaxis=0 to 1 by.1 cframe=ligr; run; run;

13 Model event=diabetes gender diabetes_gender Association of Predicted Probabilities and Observed Responses Percent Concordant 47.7 Somers' D Percent Discordant 27.1 Gamma Percent Tied 25.2 Tau-a Pairs c Measures area under ROC curve

14 Model event=diabetes gender diabetes_gender

15 ( Chest film study by E. James Potchen, M.D., 1999 )