Performance of a diagnostic test Dagmar Rimek EPIET-EUPHEM Introductory Course 2012 Lazareto, Menorca, Spain Based on the Lecture of 2011 by Steen Ethelberg

Outline Performance characteristics of a test Sensitivity Specificity Choice of a threshold Performance of a test in a population Positive predictive value of a test (PPV) Negative predictive value of a test (NPV) Impact of disease prevalence, sensitivity and specificity on predictive values

Performance characteristics of a test in a laboratory setting

Population with affected and non-affected individuals

A perfect diagnostic test identifies the affected individuals only Non-affected

In reality, tests are not perfect Affected Non-affected

Proportion of persons testing positive among affected individuals Sensitivity of a test The sensitivity of a test is the ability of the test to identify correctly the affected individuals Proportion of persons testing positive among affected individuals Affected persons Test result + - True positive (TP) False negative (FN) Sensitivity (Se) = TP / (TP + FN) 7

Estimating the sensitivity of a test Identify affected individuals with a gold standard Obtain a wide panel of samples that are representative of the population of affected individuals Recent and old cases Severe and mild cases Various ages and sexes Test the affected individuals Estimate the proportion of affected individuals that are positive with the test

Example: Estimating the sensitivity of a new ELISA IgM test for acute Q-fever Identify persons with acute Q-fever with a gold standard (IgM Immunofluorescence Assay) Obtain a wide panel of samples that are representative of the population of individuals with acute Q-fever Recent and old cases Severe and asymptomatic cases Various ages and sexes Test the persons with acute Q-fever Estimate the proportion of persons with acute Q-fever that are positive with the ELISA IgM test

Example: Sensitivity a new ELISA IgM test for acute Q-fever Patients with acute Q-fever ELISA IgM test result + True positive (TP) 148 - False negative (FN) 2 150 Sensitivity = TP / (TP + FN) 148 / 150 = 98.7% 10

What factors influence the sensitivity of a test? Characteristics of the affected persons? YES: Antigenic characteristics of the pathogen in the area (e.g., if the test was not prepared with antigens reflecting the population of pathogens in the area, it will not pick up infected persons in the area) Characteristics of the non-affected persons? NO: The sensitivity is estimated on a population of affected persons Prevalence of the disease? Sensitivity is an INTRINSIC characteristic of the test

Proportion of persons testing negative among non-affected individuals Specificity of a test The specificity of a test is the ability of the test to identify correctly non-affected individuals Proportion of persons testing negative among non-affected individuals Non-affected persons Test result + - False positive (FP) True negative (TN) Specificity (Sp) = TN / (TN + FP) 12

Estimating the specificity of a test Identify non-affected individuals Negative with a gold standard Unlikely to be infected Obtain a wide panel of samples that are representative of the population of non-affected individuals Test the non-affected individuals Estimate the proportion of non-affected individuals that are negative with the test

Example: Estimating the specificity of a new ELISA IgM test for acute Q-fever Identify persons without Q-fever Persons without sign and symptoms of the infection Persons at low risk of infection, negative with gold standard (IgM Immunofluorescence Assay) Obtain a wide panel of samples that are representative of the population of individuals without Q-fever Test the persons without Q-fever Estimate the proportion of persons without Q-fever that are negative with the new ELISA IgM test

Specificity of a new ELISA IgM test for acute Q-fever Persons without acute Q-fever ELISA IgM test result + False positive (FP) 10 - True negative (TN) 190 200 Specificity = TN / (TN + FP) 190 / 200 = 95% 15

What factors influence the specificity of a test? Characteristics of the affected persons? NO: The specificity is estimated on a population of non- affected persons Characteristics of the non-affected persons? YES: The diversity of antibodies to various other antigens in the population may affect cross reactivity or polyclonal hypergammaglobulinemia may increase the proportion of false positives Prevalence of the disease? Specificity is an INTRINSIC characteristic of the test

+ - ­ Performance of a test Disease Test TP FN Yes FP TN No TP Se = Sp = TN + FP

To whom sensitivity and specificity matters most? INTRINSIC characteristics of the test ► To laboratory specialists!

Distribution of quantitative test results among affected and non-affected people Ideal situation Non-affected: Threshold for positive result Affected: Number of people tested TN TP 0 5 10 15 20 Quantitative result of the test

Distribution of quantitative results among affected and non-affected people Realistic situation Non-affected: Threshold for positive result Affected: TN TP Number of people tested FN FP 0 5 10 15 20 Quantitative result of the test

Effect of Decreasing the Threshold Non-affected: Threshold for positive result Affected: FP Number of people tested TP TN FN 0 5 10 15 20 Quantitative result of the test

Effect of Decreasing the Threshold Disease Test TP FN Yes + - FP TN No ­ TP Se = TP + FN TN Sp = TN + FP

Effect of Increasing the Threshold Non-affected: Threshold for positive result Affected: Number of people tested TN TP FN FP 0 5 10 15 20 Quantitative result of the test

Effect of Increasing the Threshold Disease Test TP FN Yes + - FP TN No ­ TP Se = TP + FN TN Sp = TN + FP

Performance of a test and threshold Sensitivity and specificity vary in opposite directions when changing the threshold (e.g. the cut-off in an ELISA) The choice of a threshold is a compromise to best reach the objectives of the test consequences of having false negatives? consequences of having false positives?

Using several tests One way out of the dilemma is to use several tests that complement each other First use test with a high sensitivity (e.g. screening for HIV by ELISA, or for syphilis by TPHA) Second use test with a high specificity (e.g. confirmation of HIV or syphilis by western blot)

ROC curves Receiver Operating Characteristics curve Representation of relationship between sensitivity and specificity for a test Simple tool to: Help define best cut-off value of a test Compare performance of two tests

Prevention of blood transfusion malaria: Choice of an indirect IFA threshold Sensitivity (%) 100 1/20 1/10 1/40 80 1/80 1/160 60 IFA Dilutions 1/320 40 1/640 20 20 40 60 80 100 100 - Specificity (%): Proportion of false positives

Comparison of performance of IFA and ELISA IgM tests for detection of acute Q-fever Sensitivity (%) 100 80 IFA ELISA 60 40 Area under the ROC curve (AUC) 20 25 50 75 100 100 - Specificity (%)

Performance of a test in a population

How well does the test perform in a real population? The test is now used in a real population This population is made of Affected individuals Non-affected individuals The proportion of affected individuals is the prevalence Status of persons Affected Non-affected Test Positive True + False + A+B Negative False - True - C+D A+C B+D A+C+B+D

Predictive value of a positive test The predictive value of a positive test is the probability that an individual testing positive is truly affected Proportion of affected persons among those testing positive

Positive predictive value (PPV) of a test Status of persons Affected Non-affected Test Positive A B A+B Negative C D C+D A + C B+D A+C+B+D PPV = A / (A+B) This is only valid for the sample of specimens tested 33

What factors influence the positive predictive value of a test? Status of persons Affected Non-affected Test Positive A B A+B Negative C D C+D A + C B+D A+C+B+D Sensitivity? YES: To some extend. Specificity? YES: The more the test is specific, the more it will be negative for non-affected persons (less false-positive results). Prevalence of the disease? YES: Low prevalence: Low pre-test probability for positives. The test will pick up more false positives. YES: High prevalence: High pre-test probability for positives. The test will pick up more true positives.

Positive predictive value of a test according to prevalence and specificity PPV (%)

Predictive value of a negative test The predictive value of a negative test is the probability that an individual testing negative is truly non-affected Proportion of non-affected persons among those testing negative

Negative predictive value (NPV) of a test Status of persons Affected Non-affected Test Positive A B A+B Negative C D C+D A+C B+D A+C+B+D NPV = D / (C+D) This is only valid for the sample of specimens tested 37

What factors influence the negative predictive value of a test? Status of persons Affected Non-affected Test Positive A B A+B Negative C D C+D A+C B+D A+C+B+D Sensitivity? YES: The more the test is sensitive, the more it captures affected persons (less false negatives). Specificity? YES: But to a lesser extend. Prevalence of the disease? YES: Low prevalence: High pre-test probability for negatives. The test will pick up more true negatives. YES: High prevalence: Low pre-test probability for negatives. The test will pick up more false negatives.

Negative predictive value of a test according to prevalence and sensitivity NPV (%)

Relation between predictive values and sensitivity (Se), specificity (Sp), prevalence (Pr) (1-Se)Pr + Sp(1-Pr) Disease (1-Sp)(1-Pr) Se Pr No Yes Se Pr + (1-Sp)(1-Pr) Pr 1-Pr Sp(1-Pr) (1-Se)Pr Test + -

Calculate PPV and NPV Pr) Sp)(1 (1 Pr Se PPV - + = Pr Se) (1 Pr) -

Relation between predictive values and sensitivity / specificity (1 Pr Se PPV - + = Increasing specificity  increasing PPV Pr Se) (1 Pr) - Sp(1 NPV + = Increasing sensitivity  increasing NPV

Relation between predictive values and prevalence Sp)(1 (1 Pr Se PPV - + = Increasing prevalence  increasing PPV Pr Se) (1 Pr) - Sp(1 NPV + = Decreasing prevalence  increasing NPV

Example: Screening for acute Q-fever in two settings ELISA IgM test Sensitivity = 98% Specificity = 95% Population in low endemic area Prevalence = 0.5% Patients with atypical pneumonia Prevalence = 20% 10,000 tests performed in each group

Example: Screening for acute Q-fever in a population in a low endemic area IgM ELISA test sensitivity = 98% IgM ELISA test specificity = 95% Prevalence = 0.5% Q-fever Yes No Total IgM ELISA + 49 497 546 ­- 1 9,453 9,454 50 9,950 10,000 PPV = 8.97% NPV = 99.98%

Example: Screening for acute Q-fever in patients with atypical pneumonia IgM ELISA test sensitivity = 98% IgM ELISA test specificity = 95% Prevalence = 20% Q-fever Yes No Total IgM ELISA + 1,960 400 2,360 ­- 40 7,600 7,640 2,000 8,000 10,000 PPV = 83.05% NPV = 99.48%

To whom predictive values matters most? Look at denominators! Persons testing positive Persons testing negative ► To clinicians probability that a individual with a positive test is really sick? probability that a individual with a negative test is really healthy? ► To epidemiologists! proportion of positive tests corresponding to true patients? proportion of negative tests corresponding to healthy subjects?

Summary Sensitivity and specificity matter to laboratory specialists Studied on panels of positives and negatives Intrinsic characteristics of a test Capacity to identify the affected Capacity to identify the non-affected Predictive values matter to clinicians and epidemiologists Studied on homogeneous populations Dependent on the disease prevalence Performance of a test in real life How to interpret a positive test How to interpret a negative test

Where will you do your rain dance? There? Here?