Performance of a diagnostic test Tunisia, 31 Oct 2014 Acknowledgments :Mia Brytting and Georgia Ladbury EPIET/EUPHEM Introductory Course 2014 Prof Enver Roshi Faculty of Public Health, University of Medicine, Tirane- Albania roshienvi@yahoo.com

Outline Performance characteristics of a test Sensitivity Specificity Choice of a threshold Performance of a test in a population Positive predictive value of a test (PPV) Negative predictive value of a test (NPV) Impact of disease prevalence, sensitivity and specificity on predictive values

Performance characteristics of a test in a laboratory setting

Population with affected and non-affected individuals

A perfect diagnostic test identifies the affected individuals only Non-affected

In reality, tests are not perfect Affected Non-affected

Diagnostic sensitivity of a test The sensitivity of a test is the ability of the test to identify correctly the affected individuals Proportion of persons testing positive among affected individuals Affected persons Test result + - True positive (TP) False negative (FN) Sensitivity (Se) = TP / (TP + FN) 7

Estimating the sensitivity of a test Identify affected individuals with a gold standard Obtain a wide panel of samples that are representative of the population of affected individuals Recent and old cases Severe and mild cases Various ages and sexes Test the affected individuals Estimate the proportion of affected individuals that are positive with the test

Example: Estimating the sensitivity of a new ELISA IgM test for acute Q fever Identify persons with acute Q fever with a golden standard (IgM Immunofluorescence Assay) Obtain a wide panel of samples that are representative of the population of individuals with acute Q-fever Recent Severe and asymptomatic cases Various ages and sexes Test the persons with acute Q-fever Estimate the proportion of persons with acute Q-fever that are positive with the ELISA IgM test

Example: Sensitivity a new ELISA IgM test for acute Q-fever Patients with acute Q-fever ELISA IgM test result + True positive (TP) 148 - False negative (FN) 2 150 Sensitivity = TP / (TP + FN) 148 / 150 = 98.7% 10

What factors influence the sensitivity of a test? Characteristics of the affected persons? YES: Antigenic characteristics of the pathogen in the area (e.g., if the test was not prepared with antigens reflecting the population of pathogens in the area, it will not pick up infected persons in the area) Characteristics of the non-affected persons? NO: The sensitivity is estimated on a population of affected persons Prevalence of the disease? Sensitivity is an INTRINSIC characteristic of the test 11

Diagnostic specificity of a test The specificity of a test is the ability of the test to identify correctly non-affected individuals Proportion of persons testing negative among non-affected individuals Non-affected persons Test result + - False positive (FP) True negative (TN) Specificity (Sp) = TN / (TN + FP) 12

Estimating the specificity of a test Identify non-affected individuals Negative with a gold standard Unlikely to be infected Obtain a wide panel of samples that are representative of the population of non-affected individuals Test the non-affected individuals Estimate the proportion of non-affected individuals that are negative with the test 13

Example: Estimating the specificity of a new ELISA IgM test for acute Q-fever Identify persons without Q-fever Persons without sign and symptoms of the infection Persons at low risk of infection, negative with gold standard (IgM Immunofluorescence Assay) Obtain a wide panel of samples that are representative of the population of individuals without Q-fever Test the persons without Q-fever Estimate the proportion of persons without Q-fever that are negative with the new ELISA IgM test 14

Specificity of a new ELISA IgM test for acute Q-fever Patients with acute Q-fever ELISA IgM test result + False positive (TP) 10 - True negative (TN) 190 200 Specificity = TN / (TN + FP) 190 / 200 = 95% 15

What factors influence the specificity of a test? Characteristics of the affected persons? NO: The specificity is estimated on a population of non-affected persons Characteristics of the non-affected persons? YES: The diversity of antibodies to various other antigens in the population may affect cross reactivity or polyclonal hypergammaglobulinemia may increase the proportion of false positives Prevalence of the disease? Specificity is an INTRINSIC characteristic of the test 16

Performance of a test + - ­ Disease Test TP FN Yes FP TN No TP Se = Sp = TN + FP 17

To whom sensitivity and specificity matters most? INTRINSIC characteristics of the test ► To laboratory specialists! 18

Distribution of quantitative test results among affected and non-affected people Ideal situation Non-affected: Threshold for positive result Affected: Number of people tested TN TP 0 5 10 15 20 19 Quantitative result of the test

Distribution of quantitative results among affected and non-affected people Realistic situation Non-affected: Threshold for positive result Affected: TN TP Number of people tested FN FP 0 5 10 15 20 Quantitative result of the test 20

Effect of Decreasing the Threshold Non-affected: Threshold for positive result Affected: FP Number of people tested TP TN FN 0 5 10 15 20 Quantitative result of the test 21

Effect of Decreasing the Threshold Disease Test TP FN Yes + - FP TN No ­ TP Se = TP + FN TN Sp = TN + FP 22

Effect of Increasing the Threshold Non-affected: Threshold for positive result Affected: Number of people tested TN TP FN FP 0 5 10 15 20 Quantitative result of the test 23

Effect of Increasing the Threshold Disease Test TP FN Yes + - FP TN No ­ TP Se = TP + FN TN Sp = TN + FP 24

Performance of a test and threshold Sensitivity and specificity vary in opposite directions when changing the threshold (e.g. the cut-off in an ELISA) The choice of a threshold is a compromise to best reach the objectives of the test consequences of having false negatives? consequences of having false positives? 25

When false diagnosis is worse than missed diagnosis Example: Screening for congenital toxoplasmosis One should minimise false positives Prioritise SPECIFICITY 26

When missed diagnosis is worse than false diagnosis Example: Testing for Helicobacter pylori infection One should minimise the false negatives Prioritise SENSITIVITY 27

Using several tests One way out of the dilemma is to use several tests that complement each other First use test with a high sensitivity (e.g. screening for HIV by ELISA, or for syphilis by TPHA) Second use test with a high specificity (e.g. confirmation of HIV or syphilis by western blot) 28

Performance of a test Validity Reproducibility Sensitivity Specificity 29

2. Performance of a test in a population

Would like to know… As a clinician probability that a individual with a positive test is really sick? probability that a individual with a negative test is really healthy? As an epidemiologist/PH microbiologist proportion of positive tests corresponding to true patients? proportion of negative tests corresponding to healthy subjects? 31

Positive Predictive Value (PPV) Probability that an individual testing positive is truly affected proportion of affected people among those testing positive Disease Yes No + Test TP FP PPV = TP/(TP+FP) 32

Negative Predictive Value (NPV) Probability that an individual testing negative is truly unaffected proportion of non affected among those testing negative Disease Yes No - ­ Test FN TN NPV = TN/(TN+FN) 33

What factors influence the predictive values of a test? Predicted values are not constant, they vary between different populations Sensitivity Specificity Prevalence 34

How PPV, NPV, Se, Sp and Pr are inter-related Disease Yes No FP TP + PPV = TP/(TP+FP) Test - ­ ­ FN TN NPV = TN/(TN+FN) 35

How PPV, NPV, Se, Sp and Pr are inter-related (cont.) Disease Yes No + Se Pr + (1-Sp)(1-Pr) Se Pr PPV= Se Pr (1-Sp)(1-Pr) Test - ­ (1-Se)Pr+ Sp(1-Pr) Sp(1-Pr) NPV= (1-Se)Pr Sp(1-Pr) Pr 1-Pr 36

Relation between predictive values and sensitivity / specificity (1 Pr Se PPV - + = Increasing specificity  increasing PPV Pr Se) (1 Pr) - Sp(1 NPV + = Increasing sensitivity  increasing NPV 37

Relation between predictive values and prevalence Sp)(1 (1 Pr Se PPV - + = Increasing prevalence  increasing PPV Pr Se) (1 Pr) - Sp(1 NPV + = Decreasing prevalence  increasing NPV 38

Example: Testing for acute Q-fever in two settings ELISA IgM test Sensitivity = 98% Specificity = 95% Population in low endemic area Prevalence = 0.5% Patients with atypical pneumonia Prevalence = 20% 10,000 tests performed in each group 39

Example: Testing for acute Q-fever in a population in a low endemic area Prevalence = 0.5% IgM ELISA test sensitivity = 98% IgM ELISA test specificity = 95% Q-fever Yes No Total IgM ELISA + 49 497 546 ­- 1 9,453 9,454 50 9,950 10,000 PPV = 8.97% NPV = 99.98% 40

Example: Testing for acute Q-fever in patients with atypical pneumonia IgM ELISA test sensitivity = 98% IgM ELISA test specificity = 95% Prevalence = 20% Q-fever Yes No Total IgM ELISA + 1,960 400 2,360 ­- 40 7,600 7,640 2,000 8,000 10,000 PPV = 83.05% NPV = 99.48% 41

What happens in an outbreak situation? Sensitivity Specificity PPV NPV 42

Summary Sensitivity and specificity matter to laboratory specialists Intrinsic characteristics of a test Se: capacity to identify sick people as sick Sp: capacity to identify healthy people as healthy Predictive values matter to clinicians, epidemiologists and PH microbiologists Dependent on the disease prevalence Performance of a test in a real-life population PPV: how to interpret a positive test NPV: how to interpret a negative test 43

Any questions? Thank you! Prof Enver Roshi Faculty of Public Health, University of Medicine, Tirane- Albania roshienvi@yahoo.com