1. Sensitivity, Specificity and ROC Curve Analysis

3. Criteria for Evaluating a Screening Test Validity : provide a good indication of who does and does not have disease -Sensitivity of the test -Specificity of the test Reliability : (precision): gives consistent results when given to same person under the same conditions Yield : Amount of disease detected in the population, relative to the effort -Prevalence of disease/predictive value

4. Validity of Screening Test (Accuracy) - Sensitivity: Is the test detecting true cases of disease? Ideal is 100%: 100% of cases are detected; =Pr(T+|D+) -Specificity: Is the test excluding those without disease? Ideal is 100%: 100% of non-cases are negative; =Pr(T-|D-) - See Gehlbach, Chp. 10

5. True Cases of Glaucoma YesNo IOP > 22:Yes50100 No501900 (total)1002000 Sensitivity = 50% (50/100) False Negative=50% Specificity = 95% (1900/2000) False Positive=5% Example: Screening for Glaucoma using IOP

6. Consider: -The impact of high number of false positives: anxiety, cost of further testing -Importance of not missing a case: seriousness of disease, likelihood of re-screening Where do we set the cut-off for a screening test?

7. Yield from the Screening Test: Predictive Value Relationship between Sensitivity, Specificity, and Prevalence of Disease Prevalence is low, even a highly specific test will give large numbers of False Positives Predictive Value of a Positive Test (PPV): Likelihood that a person with a positive test has the disease Predictive Value of a Negative Test (NPV): Likelihood that a person with a negative test does not have the disease

8. True Cases of Glaucoma YesNo IOP > 22:Yes50100 No501900 (total)1002000 Specificity = 95% (1900/2000) False Positive=5% Positive Predictive Value =33% (50/150) Screening for Glaucoma using IOP

9. How Good does a Screening Test have to be? IT DEPENDS - Seriousness of disease, consequences of high false positivity rate: - Rapid HIV test should have >90% sensitivity, 99.9% specificity -Screen for nearsighted children proposes 80% sensitivity, >95% specificity -Pre-natal genetic questionnaire could be 99% sensitive, 80% specific

10. Choosing a cut-point: receiver operating characteristic curves Situation where screening test yields results as a continuous value (e.g., intraocular pressure for glaucoma) Want to select a value above (or below) which to call “diseased” or “at risk” How do we select that value?

11. Non-diseased cases Diseased cases Test result value or subjective judgment of likelihood that case is diseased Threshold

12. 12 Non-diseased cases Diseased cases Test result value or subjective judgment of likelihood that case is diseased More typically:

13. Threshold TP Fraction (sensitivity) FP Fraction (1-specificity) less aggressive mindset Non-diseased cases Diseased cases

14. Threshold moderate mindset Non-diseased cases Diseased cases TP Fraction (sensitivity) FP Fraction (1-specificity)

15. Threshold more aggressive mindset Non-diseased cases Diseased cases TP Fraction (sensitivity) FP Fraction (1-specificity)

16. Threshold Non-diseased cases Diseased cases Entire ROC curve TP Fraction (sensitivity) FP Fraction (1-specificity)

17. Entire ROC curve Reader Skill and/or Level of Technology chance line TP Fraction (sensitivity) FP Fraction (1-specificity) Highly discriminate (good) Somewhat discriminate (not as good) Non-informative (no better than chance) Use area under to curve (AUC) to judge discriminating ability. Gehlbach: want AUC>80%

18. Luke Neff: Refractory Burn Shock Data Logistic Regression and ROC Curve Analysis Response Profile Ordered Value PET Total Frequency 1022 2120 Testing Global Null Hypothesis: BETA=0 TestChi-SquareDFPr > ChiSq Likelihood Ratio20.26511<.0001 Score15.32701<.0001 Wald10.193010.0014

19. Luke Neff: Refractory Burn Shock Data Logistic Regression and ROC Curve Analysis Analysis of Maximum Likelihood Estimates ParameterDFEstimate Standard Error Wald Chi-Square Pr > ChiSq Intercept1-3.06490.951410.37710.0013 Admission Lactate10.84360.264210.19300.0014 Odds Ratio Estimates Effect Point Estimate 95% Wald Confidence Limits Admission Lactate2.3251.3853.902

20. Luke Neff: Refractory Burn Shock Data Logistic Regression and ROC Curve Analysis Area Standard Error 0.84890.0633 95% Wald Confidence Limits 0.72490.9729

21. Pred ProbTrue PosTrue NegFalse Pos False NegSe1 - Sp 0.99951220190.050 0.98632220180.10 0.98383220170.150 0.964220160.20 0.94026220140.30 0.93537220130.350 0.91828220120.40 0.8899220110.450 0.840110220100.50 0.82841122090.550 0.78941222080.60 0.6751221180.60.05 0.6371220280.60.09 0.57671218480.60.18 0.53511317570.650.23 0.4931417560.70.23 0.43021416660.70.27 0.40961516650.750.27 0.389416 640.80.27 0.36951716630.850.27 0.33121815720.90.32 0.31271814820.90.36 0.26111813920.90.41 0.229918121020.90.45 0.188119101210.950.55 0.16371981410.950.64 0.15251971510.950.68 0.14191951710.950.77 0.12261941810.950.82 0.10561922010.950.91 0.09071912110.95 0.071820022011 Corresponds to lactate value of about 3.0 Point that Maximizes sum of sensitivity and specificity.