Sensitivity, Specificity and ROC Curve Analysis

Criteria for Evaluating a Screening Test Validity : provide a good indication of who does and does not have disease -Sensitivity of the test -Specificity of the test Reliability : (precision): gives consistent results when given to same person under the same conditions Yield : Amount of disease detected in the population, relative to the effort -Prevalence of disease/predictive value

Validity of Screening Test (Accuracy) - Sensitivity: Is the test detecting true cases of disease? Ideal is 100%: 100% of cases are detected; =Pr(T+|D+) -Specificity: Is the test excluding those without disease? Ideal is 100%: 100% of non-cases are negative; =Pr(T-|D-) - See Gehlbach, Chp. 10

True Cases of Glaucoma YesNo IOP > 22:Yes50100 No (total) Sensitivity = 50% (50/100) False Negative=50% Specificity = 95% (1900/2000) False Positive=5% Example: Screening for Glaucoma using IOP

Consider: -The impact of high number of false positives: anxiety, cost of further testing -Importance of not missing a case: seriousness of disease, likelihood of re-screening Where do we set the cut-off for a screening test?

Yield from the Screening Test: Predictive Value Relationship between Sensitivity, Specificity, and Prevalence of Disease Prevalence is low, even a highly specific test will give large numbers of False Positives Predictive Value of a Positive Test (PPV): Likelihood that a person with a positive test has the disease Predictive Value of a Negative Test (NPV): Likelihood that a person with a negative test does not have the disease

True Cases of Glaucoma YesNo IOP > 22:Yes50100 No (total) Specificity = 95% (1900/2000) False Positive=5% Positive Predictive Value =33% (50/150) Screening for Glaucoma using IOP

How Good does a Screening Test have to be? IT DEPENDS - Seriousness of disease, consequences of high false positivity rate: - Rapid HIV test should have >90% sensitivity, 99.9% specificity -Screen for nearsighted children proposes 80% sensitivity, >95% specificity -Pre-natal genetic questionnaire could be 99% sensitive, 80% specific

Choosing a cut-point: receiver operating characteristic curves Situation where screening test yields results as a continuous value (e.g., intraocular pressure for glaucoma) Want to select a value above (or below) which to call “diseased” or “at risk” How do we select that value?

Non-diseased cases Diseased cases Test result value or subjective judgment of likelihood that case is diseased Threshold

12 Non-diseased cases Diseased cases Test result value or subjective judgment of likelihood that case is diseased More typically:

Threshold TP Fraction (sensitivity) FP Fraction (1-specificity) less aggressive mindset Non-diseased cases Diseased cases

Threshold moderate mindset Non-diseased cases Diseased cases TP Fraction (sensitivity) FP Fraction (1-specificity)

Threshold more aggressive mindset Non-diseased cases Diseased cases TP Fraction (sensitivity) FP Fraction (1-specificity)

Threshold Non-diseased cases Diseased cases Entire ROC curve TP Fraction (sensitivity) FP Fraction (1-specificity)

Entire ROC curve Reader Skill and/or Level of Technology chance line TP Fraction (sensitivity) FP Fraction (1-specificity) Highly discriminate (good) Somewhat discriminate (not as good) Non-informative (no better than chance) Use area under to curve (AUC) to judge discriminating ability. Gehlbach: want AUC>80%

Luke Neff: Refractory Burn Shock Data Logistic Regression and ROC Curve Analysis Response Profile Ordered Value PET Total Frequency Testing Global Null Hypothesis: BETA=0 TestChi-SquareDFPr > ChiSq Likelihood Ratio <.0001 Score <.0001 Wald

Luke Neff: Refractory Burn Shock Data Logistic Regression and ROC Curve Analysis Analysis of Maximum Likelihood Estimates ParameterDFEstimate Standard Error Wald Chi-Square Pr > ChiSq Intercept Admission Lactate Odds Ratio Estimates Effect Point Estimate 95% Wald Confidence Limits Admission Lactate

Luke Neff: Refractory Burn Shock Data Logistic Regression and ROC Curve Analysis Area Standard Error % Wald Confidence Limits

Pred ProbTrue PosTrue NegFalse Pos False NegSe1 - Sp Corresponds to lactate value of about 3.0 Point that Maximizes sum of sensitivity and specificity.