1. Shaughnessy STFM PreDoc 06 The Case of Baby Jeff Teaching Diagnostic Sensitivity and Specificity (Is Bayes’ Theorem Really Important?)

2. Shaughnessy STFM PreDoc 06 Goals How to make boring and mind-numbing concepts of diagnostic test characteristics – specificity and sensitivity – come alive in the minds of medical students Three explanations of sensitivity, specificity and predictive value, only one of which uses numbers Discuss other issues that impact on screening for disease

3. Shaughnessy STFM PreDoc 06 This is about...... Uncertainty!

4. Shaughnessy STFM PreDoc 06 Bayes’ Theorem Bayes' theorem tells how to update or revise beliefs in light of new evidence. ProbablityProbability Some measure After test = before test X of likelihood

5. Shaughnessy STFM PreDoc 06 The Problem with Probabilities Probability theory is not intuitive Probability is a theoretical construct not often, seemingly, confirmed in practice Lab reports, test results look so “black and white” The James T. Kirk problem -- bending the laws of physics: “Scotty, I need more power!” The long tradition of the “grope-o-gram” The discomfort of ambiguity

6. Shaughnessy STFM PreDoc 06 “Physicians can do more to admit the existence of uncertainty, both to themselves and to their patients. Although this will undoubtedly be unsettling, it is honest, and it opens the way for a more intensive search for ways to reduce uncertainty.” DAVID M. EDDY, MD, PhD Eddy DM. Clinical Decision Making. Chicago; American Medical Association, 1996

7. Shaughnessy STFM PreDoc 06 Illustrations The Case of Baby Jeff

8. Shaughnessy STFM PreDoc 06 Baby Jeff and screening for muscular dystrophy Technical Precision of CPK test: –Sensitivity (ability to rule out disease): 100% –Specificity (ability to identify disease): 99.98% But, The prevalence of MD is 1 in 5000 (0.02%)

9. Shaughnessy STFM PreDoc 06 Does Baby Jeff have M.D.? Of 100,000 males, 20 will have M.D. (1 in 5,000, or 0.02% prevalence) –The test will correctly identify all 20 who have the disease (sensitivity = 100%)

10. Shaughnessy STFM PreDoc 06 Does Baby Jeff have M.D.? Of the 99,980 without M.D. –Specificity = 99.98% –99,980 x 0.9998 = 99,960 will be negative –Therefore, false positives = 20

11. Shaughnessy STFM PreDoc 06 “... The Rest of the Story” Therefore, –Out of 100,000 infants, 20 will be truly positive and 20 will be false positive –Positive predictive value = 50% –The child with a positive screening test only has a 50/50 chance of actually having MD!

12. Shaughnessy STFM PreDoc 06 Another Example: Lyme Disease Antibody assay –Sensitivity= 95%; specificity= 95% High Lyme Disease prevalence (20%) –Positive predictive value = 83% Low Lyme Disease prevalence (2%) –Positive predictive value = 28% Brown SL. Role of serology in the diagnosis of Lyme disease. JAMA 1999;282:62-6.

13. Shaughnessy STFM PreDoc 06 Another Example: Mammography Mammography in women between 40-50 yrs If 100,000 women are screened: 6,034 mammograms will be abnormal 5,998 (99.4%) will be false-positive 36 will actually have breast cancer Why? Prevalence = 0.04% (including 4 false negatives) Hamm RM, Smith SL. The accuracy of patients' judgments of disease probability and test sensitivity and specificity J Fam Pract 1998;47:44-52. Kerlikowske K, et al. Likelihood ratios for modern screening mammography. Risk of breast cancer based on age and mammographic interpretation. JAMA 1996;276:39-43.

14. Shaughnessy STFM PreDoc 06 Heart disease and Echo results Patients at low risk (example: yearly physical): prevalence = 10% Sensitivity = 90%; specificity = 90% Positive predictive value = 50%

15. Shaughnessy STFM PreDoc 06 And the WINNER! The Proteonomic Pattern test for screening for ovarian cancer “Better than CA125 to identify ovarian cancer” (Petricoin EF, Ardekani AM, Hitt BA, et al. Use of proteomic patterns in serum to identify ovarian cancer. Lancet. 2002;359:572-7.) Sensitivity: 100% Specificity: 95% Prevalence in women < 35 years old: 1 in 2,500

16. Shaughnessy STFM PreDoc 06 How many women with a positive test will have ovarian cancer? One out of every 126 0.8% 99.2% of tests will be falsely positive

17. Shaughnessy STFM PreDoc 06 Discussion: So What? Is there any harm to a false-positive test?

18. Shaughnessy STFM PreDoc 06 THE CLASSIC 2x2 TABLE TEST + TEST -

19. Shaughnessy STFM PreDoc 06 Exercise #1: Constructing a 2x2 table A study evaluating the usefulness of BNP testing to identify patients with HF found the following: –It correctly identified 98/110 patients who had heart failure –It correctly ruled out HF in 88 of 96 patients who didn’t have heart failure

20. Shaughnessy STFM PreDoc 06 Exercise #2: 2x2 table construction In the study of ultrasound to diagnose DVT, the following results were found: N = 1677 Using the gold standard, 20% had DVT Ultrasound correctly identified 62% of the patients who had DVT Ultrasound correctly ruled out 94% of patients who didn’t have a DVT

21. Shaughnessy STFM PreDoc 06 Illustrating Technical Precision Sensitivity and Specificity Sensitivity –The percentage of patients with the disease who have a positive test –Number with true positive test/Number with disease – TP/(TP+FN) –A/(A+C)

22. Shaughnessy STFM PreDoc 06 TEST + TEST - Sensitivity

23. Shaughnessy STFM PreDoc 06 Illustrating Technical Precision Sensitivity and Specificity Specificity –The percentage of patients without the disease who have a negative test –Number with true negative test/Number without disease – TN/(FP+TN) –D/(B+D)

24. Shaughnessy STFM PreDoc 06 TEST + TEST - Specificity

25. Shaughnessy STFM PreDoc 06 Technical Precision of a Test Specificity: Remember SpPin –When a test has a high Specificity, a Positive test rules IN the disorder. –“Nothing else looks like this” Sensitivity: Remember SnNout –When a test has a high Sensitivity, a Negative result rules OUT the disorder.

26. Shaughnessy STFM PreDoc 06 Illustrating Sensitivity and Specificity #2 Fishing –Big fish = people with the disease –Small fish = people without the disease

27. Shaughnessy STFM PreDoc 06 Sensitivity Small holes catch all the big fish and many small fish. (If there are not big fish in the net, they probably aren’t out there – SnNout)

28. Shaughnessy STFM PreDoc 06 Specificity Large holes catch most of the big fish but let through the small fish (most of the fish will be the big fish you want – SpPin)

29. Shaughnessy STFM PreDoc 06 The Yin & Yang of Sensitivity and Specificity Benefit: –Sensitivity and specificity are unaffected by prevalence of disease Detriment: –Sensitivity and specificity are unaffected by prevalence of disease

30. Shaughnessy STFM PreDoc 06 Clinical Precision Predictive Values Positive Predictive Value –The proportion of patients with a positive test who have the disease –The number of true positive tests/Total positive test –TP/(TP + FP) –A/(A + B)

31. Shaughnessy STFM PreDoc 06 Predictive Values Negative Predictive Value –The proportion of patients with a negative test who don’t have the disease. –The number of true negatives/all negative tests –TN/TN + FN –D/(C + D) Predictive values are affected by prevalence

32. Shaughnessy STFM PreDoc 06 TEST + TEST - Positive Predictive Value

33. Shaughnessy STFM PreDoc 06 TEST + TEST - Negative Predictive Value

34. Shaughnessy STFM PreDoc 06 TEST + TEST - Putting it all together Positive Predictive Value Sensitivity Specificity Negative Predictive Value

35. Shaughnessy STFM PreDoc 06 Exercise #3 The 36 y/o patient’s pap smear report mentions the presence of Trichomonas. What’s the probability that she really has Trich? A 17 y/o patient with 5 sexual partners in the last month has a pap smear reporting Trich. What’s the probability that she really has Trich?

36. Shaughnessy STFM PreDoc 06

37. Exercise #4: DVTs in the Office Your 50-year-old male patient presents to your office complaining of swelling in his right leg. Of note, he just had a cast taken off the leg a few weeks ago, and there is swelling and pitting edema in the affected leg. You measure the swelling, and find that his right leg is at least 3 cm bigger than his other leg. What is the probability this is a DVT? What assuredness will an ultrasound give us?

38. Shaughnessy STFM PreDoc 06 Using the DVT clinical rule

39. Shaughnessy STFM PreDoc 06 Using the diagnostic calculator

40. Shaughnessy STFM PreDoc 06 Medical Wisdom Levels of “POEMness” for Diagnostic Tests 1.Sensitivity & specificity 2.Does it change diagnoses? 3.Does it change treatment? 4.Does it change outcomes? 5.Is it worthwhile (to patients and/or society)? (examples: HbA1C for DM, CPK vs T4/PKU in newborns, electron beam tomography for CAD, CRP, BMD) Fryback DG, Thornbury JR. The efficacy of diagnostic imaging. Med Decis Making 1991; 11:88-94

41. Shaughnessy STFM PreDoc 06 So,... the importance of “Bayes’ Theorem” At low prevalence (e.g. screening, primary care), even great tests can have significant false positives At high prevalence (confirmatory testing), great tests can have significant false negatives, leading to confusion Hazards of inappropriate testing/diagnosis: Remember Baby Jeff

42. Shaughnessy STFM PreDoc 06 Discussion What do students really need to know about screening and diagnostic testing?