Studying a Study and Testing a Test: Sensitivity Training, “Don’t Make a Good Test Bad”, and “Analyze This” Borrowed Liberally from Riegelman and Hirsch, 2 nd Edition

What you need to know: When will ordering a given test be helpful in making a decision on therapy? When is a test more likely to be misleading? Which test should I order when? How do I evaluate the literature regarding the value of different tests?

Sensitivity and Specificity: Sensitivity = TP/(TP+FN) Specificity = TN/(TN+FP) NEW TEST GOLD STANDARD Diseased GOLD STANDARD Disease-free Positive True Positive False positive Negative False negative True negative

Scenario #1 You read an article describing a new rapid test for diagnosis of Herpes Simplex virus infection. It was used on samples from 1000 neonatal patients with CSF pleocytosis (>30 wbc/mm3) elevated protein (>50), with negative gram stain, and the results were compared with a gold-standard test. The following results were obtained.

Sensitivity and Specificity: Question 1: What are the Sensitivity and Specificity of this test? Question 2: Is this a Good Test? NEW TEST GOLD STANDARD Diseased GOLD STANDARD Disease-free Positive Negative

Sensitivity and Specificity: Answer to Questions 1 & 2 Sensitivity = True Positives Divided by Gold Standard Diseased (True positives plus false negatives) = 400/500 = 80% Specificity = True Negatives Divided by Gold Standard Disease-free (True negatives plus false positives = 450/500 = 90% Good Test?

Take Home Message #1 Sensitivity and Specificity are Properties of the Test!!!

Scenario #2 The same diagnostic test becomes commercially available and your hospital decides that for “medicolegal reasons”, it should be done on all CSF samples, regardless of cell count and protein results. The results of the next 1,000 tests are shown on the following table.

Sensitivity and Specificity Question 3: What are the sensitivity and specificity now? Question 4: Is this a good test? NEW TEST GOLD STANDARD Diseased GOLD STANDARD Disease-free Positive 8090 Negative

Answers to 3 and 4 Sensitivity = 80/100=80% Specificity = 810/900=90% Specificity and Sensitivity do not change when you overuse the test, but the value of the positive result is less Good test? –…. 90 of the 170 patients with positive tests are actually disease-free…

Predictive Values Predictive value of a positive test = TP/(TP+FP) Predictive value of a negative test = TN/(TN+FN) NEW TEST GOLD STANDARD Diseased GOLD STANDARD Disease-free Positive 8090 Negative

Positive and Negative Predictive Value Predictive value of a positive test = Proportion of those with a positive test who have the disease = 80/170 = 47.1% Predictive value of a negative test = Proportion of those with a negative test who are disease-free = 810/830 = 97.6%

Take Home Message #2 The Predictive Value of a test depends upon the prevalence of the disease in the population in which it is applied!! Corollary: You can make a good test into a bad test by using it in a population with a very low prevalence of the disease

Scenario #3 A new extended screening test is being piloted for medium chain acyl-CoA dehydrogenase deficiency (MCAD) on newborn blood spots. The following results are obtained:

Predictive Value Question 3: What are the sensitivity, specificity, positive and negative predictive values now? Question 4: Is this a good screening test? NEW TEST GOLD STANDARD Diseased GOLD STANDARD Disease-free Positive Negative

Answers to Questions 5 and 6 Sensitivity = 99/100 = 99% Specificity = 8100/9900 = 82% Predictive value of positive = 99/1899 = 5.2% Predictive value of negative= 8100/8101= 99.99%

Characteristics of a Good Screen Very high NEGATIVE PREDICTIVE value (implies high sensitivity) Availability of follow-up Gold Standard test to confirm (may be repetition of original test) Availability of counseling and education Intervention that affects outcome

Risks and Odds Sometimes you don’t know the prevalence of the disease within a given situation, so you need to evaluate that from the literature Two questions: –What is the risk of a given pathology in the context of a given risk factor? –What are the odds before and after you assess for the presence of a risk factor?

Scenario #4: You see a febrile infant in the midst of an aseptic meningitis outbreak, but this one has never had Prevnar. You have access to some raw prospective data on the use of Prevnar in your area: Invasive Pneumococcal disease No Invasive disease No Prevnar30970 Prevnar3997 What is the relative risk of Invasive disease in an unimmmunized patient?

Relative Risk Relative Risk = probability of disease with the risk factor ÷ probability without the risk factor 0.030/0.003 = 10 BUT, this only works if the data is generated PROSPECTIVELY

What if this data were generated RETROSPECTIVELY, i.e., taking kids who had invasive disease and selecting matched controls. Invasive Pneumococcal disease No Invasive disease No Prevnar9045 Prevnar1055 What is the odds ratio of Invasive disease in an unimmmunized patient?

Odds ratio Odds ratio: odds of having a risk factor in the diseased population ÷ odds of having the risk factor in a matched control population 90/10 ÷ 45/55 = 9 ÷ 0.82 = 11

Scenario #5 An article outlines guidelines for referral to a pediatric cardiologist for infants and children with cardiac murmurs. The article describes a decision analysis methodology that allows you to compare the effectiveness of a two different paradigms, one in which EKG is combined with CXR at the same time, the other in which only abnormal quality murmurs are sent on for Echo.

Decision Analysis: Serial Application of Tests

Decision Analysis: Parallel Application of Tests

Which of the following is true? A. In a serial testing approach, the initial test must meet most criteria of a good screening test. B. In a parallel approach, more diagnostic value is gained if the outcomes of the 2 tests are independent [e.g., CXR is less useful in asthmatics, since wheeze and atelectasis occur together in a non-random fashion] C. A and B are both true.

Interventional Studies Assignment to Groups Assessment of outcomes Analysis Interpretation

Assignment (Avoid Bias) Prospective vs. Retrospective Randomized vs. Selected Blinded vs. Open-label Placebo

Assessment Define variables prior to initiation of study Choose a good test!!!

Analysis State Hypothesis as null: “There is no difference between treatment and control groups” Type I error: Falsely reject the null hypothesis, P value (0.05) is likelihood of type I Type II error: Falsely accept the null hypothesis, Power = type II error Power of 0.8 is standard

3 determinants of Power Variability of the test Incremental change Sample size

Interpretation What was the study population? Do the results apply to your population?