Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007.

Slides:



Advertisements
Similar presentations
2) Multilevel Tests (Michael) Likelihood ratios for results other than + or -
Advertisements

CHAPTER TWELVE ANALYSING DATA I: QUANTITATIVE DATA ANALYSIS.
Assessing Information from Multilevel (Ordinal) and Continuous Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn,
Diagnostic Tests for Lower Extremity Osteomyelitis Laura Zakowski, MD* *no financial disclosures.
Lesson 5 Histograms and Box Plots. Histograms A bar graph that is used to display the frequency of data divided into equal intervals. The bars must be.
QUANTITATIVE DATA ANALYSIS
Midterm Review Goodness of Fit and Predictive Accuracy
Lucila Ohno-Machado An introduction to calibration and discrimination methods HST951 Medical Decision Support Harvard Medical School Massachusetts Institute.
Example 3.2.6: Sampling Fruitflies In a Drosophila population, 30% of the flies are black and 70% are gray. Suppose that two flies are randomly chosen.
12.3 – Measures of Dispersion
Frequency Distribution Ibrahim Altubasi, PT, PhD The University of Jordan.
Statistics Normal Probability Distributions Chapter 6 Example Problems.
Mann-Whitney and Wilcoxon Tests.
Chapter 13 Statistics © 2008 Pearson Addison-Wesley. All rights reserved.
AP Statistics Section 13.1 A. Which of two popular drugs, Lipitor or Pravachol, helps lower bad cholesterol more? 4000 people with heart disease were.
Septic Arthritis Pamela Gregory-Fernandez, PA-C SVCMC PA Education Program.
Diagnosis Concepts and Glossary. Cross-sectional study The observation of a defined population at a single point in time or time interval. Exposure and.
6 Probability Chapter6 p Operations on events and probability An event is the basic element to which probability can be applied. Notations Event:
Statistics in Screening/Diagnosis
Medical decision making. 2 Predictive values 57-years old, Weight loss, Numbness, Mild fewer What is the probability of low back cancer? Base on demographic.
Stats Tutorial. Is My Coin Fair? Assume it is no different from others (null hypothesis) When will you no longer accept this assumption?
Evidence Based Diagnosis Mark J. Pletcher, MD MPH 6/28/2012 Combining Tests.
Assessing Information from Multilevel and Continuous Tests Likelihood Ratios for results other than “+” or “-” Tom Newman (based on previous lectures by.
Chapter 6 Probability PowerPoint Lecture Slides Essentials of Statistics for the Behavioral Sciences Eighth Edition by Frederick J. Gravetter and Larry.
9 Mar 2007 EMBnet Course – Introduction to Statistics for Biologists Nonparametric tests, Bootstrapping
Quiz 2 - Review. Descriptive Statistics Be able to interpret: -Box Plots and Histograms -Mean, Median, Standard Deviation, and Percentiles.
Evidence Based Medicine Workshop Diagnosis March 18, 2010.
1 MATB344 Applied Statistics Chapter 1 Describing Data with Graphs.
DATA IDENTIFICATION AND ANALYSIS. Introduction  During design phase of a study, the investigator must decide which type of data will be collected and.
. Ruling in or out a disease Tests to rule out a disease  You want very few false negatives  High sensitivity 
Diagnosis: EBM Approach Michael Brown MD Grand Rapids MERC/ Michigan State University.
Copyright © 2013, 2009, and 2007, Pearson Education, Inc. Chapter 6 Probability Distributions Section 6.2 Probabilities for Bell-Shaped Distributions.
LOGISTIC REGRESSION A statistical procedure to relate the probability of an event to explanatory variables Used in epidemiology to describe and evaluate.
Linear correlation and linear regression + summary of tests
Appraising A Diagnostic Test
PeptideProphet Explained Brian C. Searle Proteome Software Inc SW Bertha Blvd, Portland OR (503) An explanation.
Probabilistic and Statistical Techniques 1 Lecture 3 Eng. Ismail Zakaria El Daour 2010.
Assessing Information from Multilevel (Ordinal) and Continuous Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn,
© Department of Statistics 2012 STATS 330 Lecture 20: Slide 1 Stats 330: Lecture 20.
1 Risk Assessment Tests Marina Kondratovich, Ph.D. OIVD/CDRH/FDA March 9, 2011 Molecular and Clinical Genetics Panel for Direct-to-Consumer (DTC) Genetic.
1. Statistics Objectives: 1.Try to differentiate between the P value and alpha value 2.When to perform a test 3.Limitations of different tests and how.
Evaluating Results of Learning Blaž Zupan
Assessing Information from Multilevel and Continuous Tests Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/2/2008.
Assessing Information from Multilevel and Continuous Tests Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/13/2011.
Biostatistics in Practice Peter D. Christenson Biostatistician Session 3: Testing Hypotheses.
Prediction statistics Prediction generally True and false, positives and negatives Quality of a prediction Usefulness of a prediction Prediction goes Bayesian.
Copyright restrictions may apply JAMA Pediatrics Journal Club Slides: Procalcitonin Use to Predict Bacterial Infection in Febrile Infants Milcent K, Faesch.
Organization of statistical research. The role of Biostatisticians Biostatisticians play essential roles in designing studies, analyzing data and.
Diagnostic Test Characteristics: What does this result mean
1 Medical Epidemiology Interpreting Medical Tests and Other Evidence.
EBM --- Journal Reading Presenter :傅斯誠 Date : 2005/10/26.
Common Errors by Teachers and Proponents of EBM
Heart Disease Example Male residents age Two models examined A) independence 1)logit(╥) = α B) linear logit 1)logit(╥) = α + βx¡
Copyright © 2013, 2010 and 2007 Pearson Education, Inc. Chapter The Normal Probability Distribution 7.
BIOSTATISTICS Lecture 2. The role of Biostatisticians Biostatisticians play essential roles in designing studies, analyzing data and creating methods.
Chapter 7 The Normal Probability Distribution 7.1 Properties of the Normal Distribution.
PTP 560 Research Methods Week 12 Thomas Ruediger, PT.
Chapter 6: Descriptive Statistics. Learning Objectives Describe statistical measures used in descriptive statistics Compute measures of central tendency.
Statistics for Business and Economics Module 1:Probability Theory and Statistical Inference Spring 2010 Lecture 8: Tests of significance and confidence.
The Normal Approximation for Data. History The normal curve was discovered by Abraham de Moivre around Around 1870, the Belgian mathematician Adolph.
Nonparametric statistics. Four levels of measurement Nominal Ordinal Interval Ratio  Nominal: the lowest level  Ordinal  Interval  Ratio: the highest.
Diagnostic Likelihood Ratio Presented by Juan Wang.
Review Design of experiments, histograms, average and standard deviation, normal approximation, measurement error, and probability.
DIAGNOSIS OF SEPTIC JOINT IN CHILDREN Sara Jane Shippee UW Orthopaedic Surgery, PGY-1 Seattle Children’s Hospital 11/1/2012.
Perioperative Testing for Joint Infection in Patients Undergoing Revision Total Hip Arthroplasty by Mark F. Schinsky, Craig J. Della Valle, Scott M. Sporer,
Bootstrap and Model Validation
Evaluating Results of Learning
Chapter 12 Statistics 2012 Pearson Education, Inc.
Refining Probability Test Informations Vahid Ashoorion MD. ,MSc,
Chapter 12 Statistics.
Presentation transcript:

Assessing Information from Multilevel (Ordinal) Tests ROC curves and Likelihood Ratios for results other than “+” or “-” Michael A. Kohn, MD, MPP 10/4/2007

Four Main Points 1) Dichotomizing a multi-level test by choosing a fixed cutpoint reduces the value of the test. 2) The ROC curve summarizes the ability of the test to differentiate between D+ and D- individuals. 3) LR(result) = P(result|D+)/P(result|D-) = slope of ROC curve. 4) Pre-Test Odds x LR(result) = Post-Test Odds

Many Tests Are Not Dichotomous Ordinal “-”, “+”, “++”, “+++” for leukocyte esterase on urine dip stick “Normal”, “Low Prob”, “Intermediate Prob”, “High Prob” on VQ scan Continuous Systolic Blood Pressure WBC Count

Evaluating the Test --Test Characteristics For dichotomous tests, we discussed sensitivity P(+|D+) and specificity P(-|D-) For multi-level and continuous tests, we will discuss the Receiver Operating Characteristic (ROC) curve

Using the Test Result to Make Decisions about a Patient For dichotomous tests, we use the LR(+) if the test is positive and the LR(-) if the test is negative For multilevel and continuous tests, we use the LR(r), where r is the result of the test

Septic Arthritis Bacterial infection in a joint.

Clinical Scenario Does this Adult Patient Have Septic Arthritis?

A 48-year-old woman with a history of rheumatoid arthritis who has been treated with long-term, low-dose prednisone presents to the emergency department with a 2-day history of a red, swollen right knee that is painful to touch. She reports no prior knee swelling and no recent trauma or knee surgery, illegal drug use, rash, uveitis, or risky sexual behavior. On examination, she is afebrile and has a right knee effusion. Her peripheral white blood cell (WBC) count is /µL and her erythrocyte sedimentation rate (ESR) is 55 mm/h. An arthrocentesis is performed, and the initial Gram stain is negative. Margaretten, M. E., J. Kohlwes, et al. (2007). Jama 297(13): You have the synovial white blood cell (WBC) count.

Clinical Scenario Does this Adult Patient Have Septic Arthritis? Assume pre-test probability of septic arthritis is How do you use the synovial WBC result to determine the likelihood of septic arthritis? Margaretten, M. E., J. Kohlwes, et al. (2007). Jama 297(13):

Why Not Make It a Dichotomous Test? SynovialSeptic Arthritis WBC Count YesNo >25,000 77% 27% ≤ 25,000 23% 73% TOTAL*100%100% *Note that these could have come from a study where the patients with septic arthritis (D+ patients) were sampled separately from those without (D- patients). Margaretten, M. E., J. Kohlwes, et al. (2007). Jama 297(13):

Why Not Make It a Dichotomous Test? Sensitivity = 77% Specificity = 73% LR(+) = 0.77/( ) = 2.9 LR(-) = ( )/0.73 = 0.32 “+” = > 25,000/uL “-” = ≤ 25,000/uL

Clinical Scenario Synovial WBC = 48,000/mL (Demonstrate LR Slide Rule?) (Demonstrate Excel?) Pre-test prob: 0.38 LR(+) = 2.9 Post-Test prob = ?

Clinical Scenario Synovial WBC = 48,000/mL Pre-test prob: 0.38 Pre-test odds: 0.38/0.62 = 0.61 LR(+) = 2.9 Post-Test Odds = Pre-Test Odds x LR(+) = 0.61 x 2.9 = 1.75 Post-Test prob = 1.75/(1.75+1) = 0.64

Clinical Scenario Synovial WBC = 128,000/mL Pre-test prob: 0.38 LR(+) = ? Post-Test prob =?

Clinical Scenario Synovial WBC = 128,000/mL Pre-test prob: 0.38 Pre-test odds: 0.38/0.62 = 0.61 LR(+) = 2.9 (same as for WBC=48,000!) Post-Test Odds = Pre-Test Odds x LR(+) = 0.61 x 2.9 = 1.75 Post-Test prob = 1.75/(1.75+1) =.64

Why Not Make It a Dichotomous Test? Because you lose information. The risk associated with a synovial WBC=48,000 is equated with the risk associated with WBC=128,000. Choosing a fixed cutpoint to dichotomize a multi- level or continuous test throws away information and reduces the value of the test.

Main Point 1: Avoid Making Multilevel Tests Dichotomous Dichotomizing a multi-level or continuous test by choosing a fixed cutpoint reduces the value of the test

WBC (/uL) Interval % of Septic Arthritis % of No Septic Arthritis >100,00029%1% >50, ,00033%7% >25,000-50,00015%19% ,00023%73% TOTAL100%

Synovial Fluid WBC Count

Histogram Does not reflect prevalence of D+ (Dark D+ columns add to 100%, Open D- columns add to 100%) Sensitivity and specificity depend on the cutpoint chosen to separate “positives” from “negatives” The ROC curve is drawn by serially lowering the cutpoint from highest (most abnormal) to lowest (least abnormal).* * Just said that choosing a fixed cutpoint reduces the value of the test. The key issues are 1) the ROC curve is for evaluating the test, not the patient, and 2) drawing the ROC curve requires varying the cutpoint, not choosing a fixed cutpoint.

WBC Count (x1000/uL) Sensitivity1 - Specificity > ∞ 0% > 10029%1% > 5062%8% > 2577%27% ≥ 0100% Margaretten, M. E., J. Kohlwes, et al. (2007). Jama 297(13):

Cutoff > ∞ Cutoff > 100k Cutoff > 50k Cutoff > 25k Cutoff ≥ 0 Area Under Curve =

Test Discriminates Well Between D+ and D- Test Result D- D+

Test Discriminates Well Between D+ and D-

Test Discriminates Poorly Between D+ and D- Test Result D- D+

Test Discriminates Poorly Between D+ and D-

Cutoff > ∞ Cutoff > 100k Cutoff > 50k Cutoff > 25k Cutoff ≥ 0 Area Under Curve = Area Under ROC Curve

Summary measure of test’s discriminatory ability Probability that a randomly chosen D+ individual will have a more positive test result than a randomly chosen D- individual

Area Under ROC Curve Corresponds to the Mann-Whitney (Wilcoxan Rank Sum) Test Statistic, which is the non-parametric equivalent of Student’s t test. Also corresponds to the “c statistic” reported in logistic regression models

“Walking Man” Approach to ROC Curves Divide vertical axis into d steps, where d is the number of D+ individuals Divide horizontal axis into n steps, where n is the number of D- individuals Sort individuals from most to least abnormal test result Moving from the first individual (with the most abnormal test result) to the last (with the least abnormal test result)…

“Walking Man” (continued) …call out “D” if the individual is D+ and “N” if the individual is D- Let the walking man know when you reach a new value of the test The walking man takes a step up every time he hears “D” and a step to the right every time he hears “N” When you reach a new value of the test, he drops a stone.

Synovial WBC Count in 5 Patients with Septic Arthritis Patient WBC Count (x 1000/uL) D1128 D292 D364 D437 D524

Synovial WBC Count in 10 Patient Without Septic Arthritis PatientWBC Count (x 1000) N171 N248 N337 N423 N512 N612 N78 N87 N96 N100

Septic ArthritisNo Septic Arthritis

DDNDN(DN)DN(NN)NNNN

Main Point 2 ROC Curve Describes the Test, Not the Patient Describes the test’s ability to discriminate between D+ and D- individuals Not particularly useful in interpreting a test result for a given patient

ROC Curve Describes the Test, Not the Patient Clinical Scenario Synovial WBC count = 48,000 Synovial WBC count = 128,000

Synovial WBC count = 48,000

Cutoff > ∞ Cutoff > 100k Cutoff > 50k Cutoff > 25k Cutoff ≥ 0

Sensitivity, Specificity, LR(+), and LR(-) of the Synovial Fluid WBC Count for Septic Arthritis at 3 Different Cutoffs WBC (/uL)SensitivitySpecificityLR+LR- >100,00029%99% >50,00062%92% >25,00077%73% Synovial WBC Count = 48,000/uL Which LR should we use?

Likelihood Ratios LR(+) = Sensitivity/(1 – Specificity) = P(+|D+)/(1-P(-|D-)) = P(+|D+)/P(+|D-) LR(-) = (1 – Sensitivity)/Specificity = (1-P(+|D+))/P(-|D-) = P(-|D+)/P(-|D-)

Likelihood Ratios LR(result) = P(result|D+)/P(result|D-) P(Result) in patient WITH disease P(Result) in patients WITHOUT disease

Likelihood Ratios The ratio of the height of the D+ distribution to the height of the D- distribution 15% 19% LR = 15%/19% = 0.8

> 50k > 25k 15% 19% Slope = 15%/19% =0.8

Likelihood Ratio WBC (/uL) Interval% of D+% of D- Interval LR >100,00029%1%29.0 >50, ,00033%7%4.7 >25,000-50,00015%19% ,00023%73%0.3

Common Mistake When given an “ROC Table,” it is tempting to calculate an LR(+) or LR(-) as if the test were “dichotomized” at a particular cutoff. Example: LR(+,25,000) = 77%/27% = 2.9 This is NOT the LR of a particular result (e.g. WBC >25,000 and ≤ 50,000); it is the LR(+) if you divide “+” from “-” at 25,000.

WBC (/uL)SensitivitySpecificityLR+LR- >100,00029%99% >50,00062%92% >25,00077%73% Common Mistake

27% 77% > 25,000 Common Mistake

Main Point 3 Likelihood Ratio P(Result) in patient WITH disease P(Result) in patients WITHOUT disease Slope of ROC Curve Do not calculate an LR(+) or LR(-) for a multilevel test.

Clinical Scenario Synovial WBC = 48,000/uL Pre-test prob: 0.38 Pre-test odds: 0.38/0.62 = 0.61 LR(WBC btw 25,000 and 50,000) = 0.8 Post-Test Odds = Pre-Test Odds x LR(48) = 0.61 x 0.8 = 0.49 Post-Test prob = 0.49/(0.49+1) = 0.33

Clinical Scenario Synovial WBC = 128,000/uL Pre-test prob: 0.38 Pre-test odds: 0.38/0.62 = 0.61 LR(128,000/uL) = 29 Post-Test Odds = Pre-Test Odds x LR(128) = 0.61 x 29 = 17.8 Post-Test prob = 17.8/(17.8+1) = 0.95

Clinical Scenario WBC = 48,000/uL Post-Test Prob = 0.33 WBC = 128,000/uL Post-Test Prob = 0.95 (Recall that dichotomizing the WBC with a fixed cutpoint of 25,000/uL meant that WBC = 48,000/uL would be treated the same as WBC = 128,000/uL and post-test prob = 0.64)

Main Point 4 Bayes’s Rule Pre-Test Odds x LR(result) = Post-Test Odds What you knew before + What you learned = What you know now

Summary 1)Dichotomizing a multi-level test by choosing a fixed cutpoint reduces the value of the test. 2)The ROC curve summarizes the discriminatory ability of the test. 3) LR(result) = P(result|D+)/P(result|D-) = Slope of ROC Curve (NOTE: Do not calculate an LR(+) or LR(-) for a multilevel test.) 4)Pre-Test Odds x LR(result) = Post-Test Odds

Calculating the c Statistic In the “walking man” approach to tracing out the ROC curve, the actual values of the test are not important for the shape of the ROC curve or the area under it--only the ranking of the values. The c statistic for the area under an ROC curve is calculated using the same information as the Wilcoxon Rank Sum statistic (or Mann-Whitney U, which is equivalent) and gives identical P values. Non-parametric equivalent of the t test statistic comparing two means.

Septic ArthritisNo Septic Arthritis

Boxes under Curve = 43.5 Total Boxes = 50 Area Under Curve = 43.5/50 = 0.87

BACTEREMIANO BACTEREMIA S = 21.5 Replace Test Results with Ranks

S = 21.5 Smin = d(d+1)/2 = 5(6)/2 = 15 Smax = dn + Smin = 5(10) + 15 = 65 C = (Smax – S) / (Smax – Smin)* = (65 – 21.5) / (65 – 15) = 43.5/50 = 0.87 * Smax – Smin = dn Calculating the C Statistic