1. Assessing Information from Multilevel and Continuous Tests Likelihood Ratios for results other than “+” or “-” Tom Newman (based on previous lectures by Michael Kohn) 10/4/2012 1

2. Overview More on Chapter 3, “X-graphs” Septic arthritis example Interval LR for multi-level tests Why not dichotomize? ROC Curves and C-statistic (AUROC) “Walking Man” approach to ROC curves Additional examples WBC for meningitis Absolute Neutrophul Count (ANC) for sepsis in newborns 2

3. Take-home points ΣProbability ✕ cost = Expected cost Avoid dichotomizing multi-level tests Do not calculate one or more LR(+) or LR(-) for a multilevel test! The area under the ROC curve summarizes the ability of the test to discriminate between D+ and D- individuals. ROC curves tell you more than just the area Slope of ROC curve = LR(result) Length of segments relates to numbers of subjects 3

4. Recall from Chapter 3…a graph of expected cost 4

5. What do we mean by expected cost? Expected cost is the sum of costs of different possible outcomes, each weigted by its probability “Costs” include everything bad about the outcome, not just money Example: 30% probability of outcome with cost $100; 70% probability of outcome with cost $0. What is “expected cost”? 5

6. What is expected cost… …of “Do Not Treat” strategy when P(D+) = 0? When P(D+) is 100%? When P(D+) is 50%? Equation for line: Expected cost = P x B 6 0.5B

7. What is expected cost… …of “Treat” strategy when P(D+) = 0? When P(D+) is 100%? When P(D+) is 60%? Equation for line: Expected cost = C  (1–P) = C – CP 7 0.4C

8. Treatment threshold probability (P TT ) is the probability of disease above which you should treat because expected cost at P>P TT is less 8

9. Now consider a (costless) test with Sensitivity=0.9 and Specificity=0.8. What is expected cost in D+ and D-? 0.1B 0.2C Test 9

10. The way human beings actually make decisions is a bit more complicated! 10

11. Questions? 11

12. Septic Arthritis Bacterial infection in a joint. 12

13. Clinical Scenario Does this Adult Patient Have Septic Arthritis? 13

14. Clinical Scenario Does this Adult Patient Have Septic Arthritis?* A 48-year-old woman with a history of rheumatoid arthritis who has been treated with long-term, low- dose steroids presents to the emergency department with a 2-day history of a red, swollen, tender right knee. On examination, she is afebrile and has a fluid in her right knee joint. The white blood cell count in her blood is 11 000/µL (normal). An arthrocentesis (needle in the joint) is done to obtain some joint fluid for analysis *Margaretten, M. E., J. Kohlwes, et al. (2007). JAMA 297(13): 1478-88. You have the synovial fluid white blood cell (WBC) count. 14

15. Clinical Scenario Does this Adult Patient Have Septic Arthritis? The authors estimated the pre-test probability of septic arthritis is 0.38. How do you use the synovial WBC result to revise the probability of septic arthritis? Margaretten, M. E., J. Kohlwes, et al. (2007). JAMA 297(13): 1478-88. 15

16. Chapter 3 method: Dichotomize at 25,000 Synovial Septic Arthritis WBC Count YesNo >25,000 77% 27% ≤ 25,000 23% 73% TOTAL100% 100% Margaretten, M. E., J. Kohlwes, et al. (2007). Jama 297(13): 1478-88.

17. Why Not Make It a Dichotomous Test? Sensitivity = 77% Specificity = 73% LR(+) = 0.77/(1 - 0.73) = 2.9 LR(-) = (1 - 0.77)/0.73 = 0.32 “+” = > 25,000/uL “-” = ≤ 25,000/uL

18. Clinical Scenario Synovial WBC = 48,000/mL Pre-test prob: 0.38 Pre-test odds: 0.38/0.62 = 0.61 LR(+) = 2.9 Post-Test Odds = Pre-Test Odds x LR(+) = 0.61 x 2.9 = 1.75 Post-Test prob = 1.75/(1.75+1) = 0.64

19. Clinical Scenario Synovial WBC = 128,000/mL Pre-test prob: 0.38 LR = ? Post-Test prob =?

20. Clinical Scenario Synovial WBC = 128,000/mL Pre-test prob: 0.38 Pre-test odds: 0.38/0.62 = 0.61 LR = 2.9 (same as for WBC=48,000!) Post-Test Odds = Pre-Test Odds x LR(+) = 0.61 x 2.9 = 1.75 Post-Test prob = 1.75/(1.75+1) =.64

21. Why Not Make It a Dichotomous Test? Because you lose information. The risk associated with a synovial WBC=48,000 is equated with the risk associated with WBC=128,000. Choosing a fixed cutpoint to dichotomize a multi- level or continuous test throws away information and (usually) reduces the value of the test.

22. Summary Sensitivity, Specificity, LR(+), and LR(-) of the Synovial Fluid WBC Count for Septic Arthritis at 3 Different Cutoffs (As presented) WBC (/µL)SensitivitySpecificityLR+LR- >100,00029%99%29.00.7 >50,00062%92%7.80.4 >25,00077%73%2.90.3 Synovial WBC Count = 48,000/uL LR = 2.9 22 Margaretten, M. E., J. Kohlwes, et al. (2007). JAMA 297(13): 1478-88.

23. 23 WBC (/µL)SensitivitySpecificityLR+LR- >100,00029%99%29.00.7 >50,00062%92%7.80.4 >25,00077%73%2.90.3 Synovial WBC Count = 48,000/uL Which LR should we use? NONE of THESE! Summary Sensitivity, Specificity, LR(+), and LR(- ) of the Synovial Fluid WBC Count for Septic Arthritis at 3 Different Cutoffs (As presented)

24. Interval Likelihood Ratios: a Better Way WBC (/uL) Interval % of D+ % of D- Interval LR >100,00029%1%29.0 50,001-100,00033%7%4.7 25,001-50,00015%19%0.8 0 - 25,00023%73%0.3 24

25. Synovial Fluid WBC Count 25 LR Histogram* * Does not reflect prior probability. D+ and D- bars both sum to 100%.

26. ROC Curves from Histogram Trade-off between sensitivity and specificity depends on the cutpoint chosen to separate “positives” from “negatives.” The ROC curve is drawn by serially moving the cutpoint from most abnormal to least abnormal. True positives (sensitivity) are plotted against false positives (1-specificity). 26

27. 27 Septic arthirtis No Septic arthirtis

28. 28

29. 29

30. 30

31. 31

32. WBC Count (x1000/uL) Sensitivity1 - Specificity > highest 0% > 10029%1% > 5062%8% > 2577%27% ≥ 0100% Margaretten, M. E., J. Kohlwes, et al. (2007). Jama 297(13): 1478-88. ROC Table 32

33. Cutoff > top value Cutoff > 100k Cutoff > 50k Cutoff > 25k Cutoff ≥ 0 33 WBC Count (K/uL ) Sensi tivity 1 - Specifi city > top 0% > 10029%1% > 5062%8% > 2577%27% ≥ 0100%

34. Cutoff > top Cutoff > 100k Cutoff > 50k Cutoff > 25k Cutoff ≥ 0 Area Under Curve = 0.8114 Area Under ROC Curve 34

35. Area Under ROC Curve Summary measure of test’s discriminatory ability Probability that a randomly chosen D+ individual will have a more positive test result than a randomly chosen D- individual 35

36. Area Under ROC Curve Also called the “c statistic” Measures discrimination for logistic regression models (“lroc” command) Statistical significance tested with the Mann-Whitney U or Wilcoxon Rank Sum Tests, non-parametric equivalents of Student’s t test based on ranks. 36

37. ROC Curve Describes the Test, Not the Patient Describes the test’s ability to discriminate between D+ and D- individuals Not particularly useful in interpreting a test result for a given patient 37

38. Evaluating the Test --Test Characteristics For dichotomous tests, we discussed sensitivity P(+|D+) and specificity P(–|D–) For multi-level and continuous tests, we can use a Receiver Operating Characteristic (ROC) curve 38

39. Using the Test Result to Make Decisions about a Patient For dichotomous tests, we use the LR(+) if the test is positive and the LR(–) if the test is negative For multilevel and continuous tests, we use the LR(r), where r is the result of the test 39

40. Likelihood Ratios LR(result) = P(result|D+)/P(result|D-) P(Result) in patients WITH disease ---------------------------------------------------- P(Result) in patients WITHOUT disease 40

41. WOWO With Over WithOut 41

42. Likelihood Ratios The ratio of the height of the D+ bar to the height of the D- bar for any result (or result interval) 15% 19% LR = 15%/19% = 0.8 42

43. > 50k > 25k 15% 19% Slope = 15%/19% =0.8 43

44. Common Mistake When given an “ROC Table,” it is tempting to calculate an LR(+) or LR(-) as if the test were “dichotomized” at a particular cutoff. Example: LR(+,25,000) = 77%/27% = 2.9 This is NOT the LR of a particular result (e.g. WBC >25,000 and ≤ 50,000); it is the LR(+) if you divide “+” from “-” at 25,000. 44

45. WBC (/uL)SensitivitySpecificityLR+LR- >100,00029%99%29.00.7 >50,00062%92%7.80.4 >25,00077%73%2.90.3 Common Mistake 45

46. Common Mistake From JAMA paper: “Her synovial WBC count of 48,000/µL increases the probability from 38% to 64%.” (Used LR = 2.9) Correct calculation: Her synovial WBC count of 48,000/µL decreases the probability from 38% to 33%.” (Used LR = 0.8) 46

47. 27% 77% > 25,000 Common Mistake 47

48. WBC (/uL)SensitivitySpecificity 1- Specificity >100,00029%99%1% >50,00062%92%8% >25,00077%73%27% 48 Obtain interval LR from ROC table by subtracting adjacent rows 77% of D+ > 25K 62% of D+ > 50 K Therefore 77%-62% =15% of D+ must be >25 K and < 50K

49. WBC (/uL)SensitivitySpecificity 1- Specificity >100,00029%99%1% >50,00062%92%8% >25,00077%73%27% 49 Obtain interval LR from ROC table by subtracting adjacent rows 27% of D- > 25K 8% of D- > 50 K Therefore 27%-8% =19% of D- must be >25 K and < 50K

50. > 50k > 25k 15% 19% Slope = 15%/19% =0.8 50

51. THE WALKING MAN OR … 51

52. … WHAT CAN YOU LEARN FROM ROC CURVES LIKE THESE? Bonsu, B. K. and M. B. Harper (2003). "Utility of the peripheral blood white blood cell count for identifying sick young infants who need lumbar puncture." Ann Emerg Med 41(2): 206-14. 52

53. “Walking Man” Approach to ROC Curves Divide vertical axis into d steps, where d is the number of D+ individuals Divide horizontal axis into n steps, where n is the number of D- individuals Sort individuals from most to least abnormal test result Moving from the first individual (with the most abnormal test result) to the last (with the least abnormal test result)… 53

54. “Walking Man” (continued) …call out “D” if the individual is D+ and “N” if the individual is D- Let the walking man know when you reach a new value of the test The walking man takes a step up every time he hears “D” and a step to the right every time he hears “N” When you reach a new value of the test, he drops a stone. 54

55. DDNDN(DN)DN (NN)NNNN 55 Septic ArthritisNo Septic Arthritis 128 92 71 64 48 37 24 23 12 8 7 6 0

56. 56

57. 57 DDNDN(DN)DN (NN)NNNN

58. … WHAT CAN YOU LEARN FROM ROC CURVES LIKE THESE? 58

59. Peripheral WBC Count as Test for Bacterial Meningitis Cross sectional study of test accuracy Index test: peripheral WBC count Reference Standard: CSF culture Population: Infants 3-89 days old evaluated in a pediatric ED for serious bacterial infection. Ann Emerg Med. 2003;41:206-214. 59

60. Ann Emerg Med. 2003;41:206-214. 60 More kids had periperhal WBC than CSF WBC Only 21 kids had bacterial meningitis Some peripheral WBC results strongly suggest meningitis What can you learn besides areas under curves?

61. KP/BWH Neonatal Infection Study - CBC aim* Retrospective cross-sectional study N = 67,623 newborns ≥ 34 weeks GA < 72 h old with paired CBC and BC (within 1 h), born 1993-2007 Predictor variables: WBC, absolute neutrophil count (ANC) Outcome variable: culture positive for a bacterial pathogen from a sterile site (N=244) *Newman TB et al. Pediatrics 2010; 126:903–909 61

62. ANC 99% 90% 50% 10% 1% 62

63. ANC ROC 63

64. Likelihood Ratios for ANC Example: 6 hr old infant with ANC = 13 Pre-test probability 4/1000 Post-test probability 0.3 x 4/1000 = 1.2/1000 64

65. Stanaway et al. How fast does the Grim Reaper walk? Receiver operating characteristics curve analysis in healthy men aged 70 and over. BMJ 2011;343:d7679

66. Questions? 66