1. MEASURES OF TEST ACCURACY AND ASSOCIATIONS DR ODIFE, U.B SR, EDM DIVISION

2. OUTLINE INTRODUCTION SENSITIVITY SPECIFICITY PREDICTIVE VALUES RECEIVER OPERATOR CHARACTERISTICS CURVE ODDS RATIO SUMMARY REFERENCES

3. INTRODUCTION Sensitivity, specificity, predictive values and receiver operator characteristics curves are measures of test accuracy Odds ratio is a measure of test associations.

4. SENSITIVITY The proportion of those people who really have the disease who are correctly identified as such Sensitivity = TP/(TP+FN) TPFP FNTN Disease Present Absent Test Positive Negative

5. SPECIFICITY The proportion of those people who really do not have the disease who are correctly identified as such Specificity = TN/(TN+FP) TPFP FNTN Disease Present Absent Test Positive Negativ e

6. SENSITIVITY AND SPECIFICITY Sensitivity and specificity are intrinsic characteristics of a test Both the sensitivity and specificity of a test need to be known in order to assess its usefulness for a diagnosis. A discriminating test would have sensitivity and specificity close to 100%. A test with high sensitivity may have low specificity and vice versa.

7. SENSITIVITY AND SPECIFICITY 7 0 5 10 15 20 Quantitative result of the test TN Non-affected: Affected: TP Number of people tested Threshold for positive result Ideal situation

8. Sensitivity and specificity 8 0 5 10 15 20 TNTP FNFP Non-affected: Threshold for positive result Quantitative result of the test Number of people tested Affected: Realistic situation

9. SENSITIVITY AND SPECIFICITY They are not affected by the prevalence of a disease. Both can be altered by changing the threshold or cut-off point for diagnosing a disease. Lowering the threshold improves sensitivity but reduces specificity (i.e. more FP) Raising the threshold improves specificity but reduces sensitivity (i.e. more FN)

10. SENSITIVITY AND SPECIFICITY 10 TN TP FN FP Non-affected: Affected: Threshold for positive result Number of people tested Quantitative result of the test 0 5 10 15 20

11. Sensitivity and specificity 11 0 5 10 15 20 TN TP FN FP Non-affected: Affected: Threshold for positive result Number of people tested Quantitative result of the test

12. Positive Predictive Value It is the proportion of the people who test positive who truly have the disease Positive predictive value (PPV) = TP/(TP+FP) TPFP FNTN Disease PresentAbsent Test Positive Negative

13. Negative Predictive Value Is the proportion of the people who test negative who truly do not have the disease Negative predictive value = TN/(TN+FN) TPFP FNTN Disease PresentAbsent Test Positive Negative

14. PPV and NPV PPV and NPV give a direct assessment of the usefulness of the test They are highly dependent on the prevalence of the disease in the population When the prevalence is low, the PPV will be low and NPV will be high. When the prevalence is high, the PPV will be high and NPV will be low.

15. Predictive value positive of a test according to prevalence and specificity Specificity

16. Predictive value negative of a test according to prevalence and sensitivity Sensitivity

17. Receiver operating characteristic (ROC) curve Developed in the 1950's during World War II for the analysis of radar radio signals. It is a by-product of research into making sense of radio signals contaminated by noise. Characterize the trade-off between positive hits and false alarms

18. ROC curve Recognized in the 1970's as useful tool for interpreting medical test results. Has become very popular in biomedical applications, particularly radiology and imaging Can be used to compare overall performance of diagnostic tests/procedures

19. ROC curve ROC curve represents the relationship between the true-positive rate (sensitivity) and the false-positive rate (1-specificity) ROC curve plots sensitivity (on the y-axis) against 1-specificity (on the x-axis) Each point on the ROC curve represents a sensitivity/specificity pair corresponding to a particular diagnostic threshold.

20. ROC curve The performance of a diagnostic variable can be quantified by calculating the area under the ROC curve (AUROC). The ideal test would have an AUROC of 1, whereas a random guess would have an AUROC of 0.5. The ability of diagnostic variables to diagnose an outcome can be compared using ROC curves and their AUROCs.

21. True Positive Rate (sensitivity) 0% 100% False Positive Rate (1-specificity) 0% 100% ROC curve

22. Ideal classifier chance always negative always positive ROC Curves

23. True Positive Rate 0% 100% False Positive Rate 0%0% 100% True Positive Rate 0% 100% False Positive Rate 0% 100% A good test: A poor test: ROC Curves

24. True Positive Rate 0%0% 100% False Positive Rate 0%0% 100 % True Positive Rate 0%0% 100% False Positive Rate 0%0% 100 % True Positive Rate 0%0% 100% False Positive Rate 0%0% 100 % AUC = 50% AUC = 90% AUC = 65% AUC = 100% True Positive Rate 0%0% 100% False Positive Rate 0%0% 100 % ROC Curves

25. ROC Curve Uses Used in communications sectors to examine false alarm rates Evaluate the discriminatory ability of a test to distinguish diseased and normal subjects Help define optimal cut-off value of a test Comparing diagnostic efficacy of 2 or more medical tests Comparing 2 or more observers measuring the same test

26. ROC Curve 26 0 020406080100 20 40 60 80 100 IFA Dilutions 1/10 1/20 1/40 1/80 1/160 1/320 1/640 100 - Specificity (%): Proportion of false positives Sensitivity (%)

27. ROC Curves 27 0 255075100 IFA ELISA 0 20 40 60 80 100 100 - Specificity (%) Sensitivity (%) Area under the ROC curve (AUC)

28. 28

29. ROC Curves

30. Odds Odds are ratio of two probabilities The probability of an event occurring divided by the probability of the event not occurring. The odds of an event = probability/(1– probability). Odds refer to single entity It ranges from zero to infinity.

31. Odds A die has 6 sizes, each with a difference number On one die, the odds of rolling a 1 is…? 1 side has a 1 on it, 5 sides do NOT Odds of rolling a 1: = 1/5, or 20%

32. Odds ratio The odds of the outcome in one group divided by the odds of the outcome in the other group It is a ratio of two odds E.g p1 refers to the probability of the outcome in group 1, and p2 is the probability of the outcome in group 2. Odds ratio (OR) = p1/(1- p1)/ p2 /(1- p2)

33. Odds ratio As a ratio, it ranges from zero to infinity. Interpretation of OR: – OR = 1: exposure has no association with disease – OR > 1: exposure may be positively associated with disease – OR < 1: exposure may be negatively associated with disease

34. Odds ratio Odds of a heart attack in men = 68/32, or 2.125 Odds of a heart attack in women = 42/58, or 0.724 Odds ratio = 2.125 / 0.724 = 2.9 Cause of Death in Men and Women Heart Attack? YesNo Sex Men6832 Women4258

35. Odds Ratio The odds of receiving a death sentence if the defendant was Black = 28/45 = 0.6222 The odds of receiving a death sentence if the defendant was not Black = 22/52 = 0.4231 The impact of being black on receiving a death penalty is measured by the odds ratio which equals: = the odds if black ÷ the odds if not black = 0.6222 ÷ 0.4231 = 1.47

36. Odds ratio Uses Appropriate measure of relative effect in case- control studies. Commonly used in meta-analysis ORs are the output of logistic regression

37. Odds ratio OR in case-control Study Probability of case being exposed = P case Probability of case being non-exposed =1-P case Odds of case being exposed = P case /1- P case Probability of control being exposed = P control Probability of control being non-exposed =1-P control Odds of control being exposed = P control / 1-P control 37

38. Target population Exposed in past Not exposed Exposed Not Exposed Odds ratio Disease (Cases) No Disease (Controls)

39. SUMMARY Sensitivity, specificity, predictive values, ROC curves and odds ratio are measures of test accuracy and associations. Sensitivity is the proportion of those people who really have the disease who are correctly identified as such Specificity is the proportion of those people who really do not have the disease who are correctly identified as such Sensitivity and specificity can be altered by changing the threshold or cut-off point for diagnosing a disease.

40. SUMMARY Sensitivity and specificity are intrinsic characteristics of a test and are not affected by the prevalence of a disease Predictive values of a test can be either positive or negative PPV is the proportion of the people who test positive who truly have the disease NPV is the proportion of the people who test negative who truly do not have the disease

41. SUMMARY ROC curve was developed in the 1950's during World War II for the analysis of radio signals. ROC curve is a useful tool for interpreting medical test results. ROC curve represents the relationship between sensitivity and specificity for a test

42. SUMMARY Predictive values are affected by the prevalence of a disease ROC curve can be used to compare overall performance of diagnostic tests The performance of a diagnostic variable can be quantified by calculating the AUROC An ideal test would have an AUROC of 1, whereas a random guess would have an AUROC of 0.5.

43. SUMMARY Odd of an event probability of an event occurring divided by the probability of the event not occurring. Odd ratio is the odds of the outcome in one group divided by the odds of the outcome in the other group The values of odds and OR ranges from ranges from zero to infinity.

44. SUMMARY OR of 1 means exposure has no association with disease, > 1 means exposure may be positively associated with disease and < 1 means exposure may be negatively associated with disease Odd ratios are useful in case-control studies, meta-analysis and logistic regression

45. REFERENCES Bland JM, Altman DG. Statistics notes. The odds ratio. BMJ 2000;320:1468. Holcomb WL, Chaiworapongsa T, Luke DA, Burgdorf KD. An odd measure of risk: use and misuse of the odds ratio. Obstet Gynecol 2001;98:685–8 Katz KA. The (relative) risks of using odds ratios. Arch Dermatol 2006;142: 761–4.

46. REFERENCES Davies HT, Crombie IK, Tavakoli M. When can odds ratios mislead? BMJ 1998;316:989–91 Schechtman E. Odds ratio, relative risk, absolute risk reduction, and the number needed to treat: which of these should we use? Value Health 2002;5:431–6.