1. Performance of a diagnostic test
Dagmar Rimek
EPIET-EUPHEM Introductory Course 2012
Lazareto, Menorca, Spain
Based on the Lecture of by Steen Ethelberg

2. Outline Performance characteristics of a test
Sensitivity
Specificity
Choice of a threshold
Performance of a test in a population
Positive predictive value of a test (PPV)
Negative predictive value of a test (NPV)
Impact of disease prevalence, sensitivity and specificity on predictive values

3. Performance characteristics of a test in a laboratory setting

4. Population with affected and non-affected individuals

5. A perfect diagnostic test identifies the affected individuals only
Non-affected

6. In reality, tests are not perfect
Affected
Non-affected

7. Proportion of persons testing positive among affected individuals
Sensitivity of a test
The sensitivity of a test is the ability of the test to identify correctly the affected individuals
Proportion of persons testing positive among affected individuals
Affected persons
Test result + -
True positive (TP)
False negative (FN)
Sensitivity (Se) = TP / (TP + FN) 7

8. Estimating the sensitivity of a test
Identify affected individuals with a gold standard
Obtain a wide panel of samples that are representative of the population of affected individuals
Recent and old cases
Severe and mild cases
Various ages and sexes
Test the affected individuals
Estimate the proportion of affected individuals that are positive with the test

9. Example: Estimating the sensitivity of a new ELISA IgM test for acute Q-fever
Identify persons with acute Q-fever with a gold standard (IgM Immunofluorescence Assay)
Obtain a wide panel of samples that are representative of the population of individuals with acute Q-fever
Recent and old cases
Severe and asymptomatic cases
Various ages and sexes
Test the persons with acute Q-fever
Estimate the proportion of persons with acute Q-fever that are positive with the ELISA IgM test

10. Example: Sensitivity a new ELISA IgM test for acute Q-fever
Patients with acute Q-fever
ELISA IgM test result +
True positive (TP)
148 -
False negative (FN) 2
150
Sensitivity =
TP / (TP + FN) 148 / 150 = 98.7% 10

11. What factors influence the sensitivity of a test?
Characteristics of the affected persons?
YES: Antigenic characteristics of the pathogen in the area (e.g., if the test was not prepared with antigens reflecting the population of pathogens in the area, it will not pick up infected persons in the area)
Characteristics of the non-affected persons?
NO: The sensitivity is estimated on a population of affected persons
Prevalence of the disease?
Sensitivity is an INTRINSIC characteristic of the test

12. Proportion of persons testing negative among non-affected individuals
Specificity of a test
The specificity of a test is the ability of the test to identify correctly non-affected individuals
Proportion of persons testing negative among non-affected individuals
Non-affected persons
Test result + -
False positive (FP)
True negative (TN)
Specificity (Sp) = TN / (TN + FP) 12

13. Estimating the specificity of a test
Identify non-affected individuals
Negative with a gold standard
Unlikely to be infected
Obtain a wide panel of samples that are representative of the population of non-affected individuals
Test the non-affected individuals
Estimate the proportion of non-affected individuals that are negative with the test

14. Example: Estimating the specificity of a new ELISA IgM test for acute Q-fever
Identify persons without Q-fever
Persons without sign and symptoms of the infection
Persons at low risk of infection, negative with gold standard (IgM Immunofluorescence Assay)
Obtain a wide panel of samples that are representative of the population of individuals without Q-fever
Test the persons without Q-fever
Estimate the proportion of persons without Q-fever that are negative with the new ELISA IgM test

15. Specificity of a new ELISA IgM test for acute Q-fever
Persons without acute Q-fever
ELISA IgM test result +
False positive (FP) 10 -
True negative (TN)
190
200
Specificity =
TN / (TN + FP) 190 / 200 = 95% 15

16. What factors influence the specificity of a test?
Characteristics of the affected persons?
NO: The specificity is estimated on a population of non- affected persons
Characteristics of the non-affected persons?
YES: The diversity of antibodies to various other antigens in the population may affect cross reactivity or polyclonal hypergammaglobulinemia may increase the proportion of false positives
Prevalence of the disease?
Specificity is an INTRINSIC characteristic of the test

17. + - ­ Performance of a test Disease Test TP FN Yes FP TN No TP Se =
Sp =
TN + FP

18. To whom sensitivity and specificity matters most?
INTRINSIC characteristics of the test
► To laboratory specialists!

19. Distribution of quantitative test results among affected and non-affected people
Ideal situation
Non-affected:
Threshold for
positive result
Affected:
Number of people tested TN TP
Quantitative result of the test

20. Distribution of quantitative results among affected and non-affected people
Realistic situation
Non-affected:
Threshold for
positive result
Affected: TN TP
Number of people tested FN FP
Quantitative result of the test

21. Effect of Decreasing the Threshold
Non-affected:
Threshold for
positive result
Affected: FP
Number of people tested TP TN FN
Quantitative result of the test

22. Effect of Decreasing the Threshold
Disease
Test TP FN
Yes + - FP TN No TP
Se =
TP + FN TN
Sp =
TN + FP

23. Effect of Increasing the Threshold
Non-affected:
Threshold for
positive result
Affected:
Number of people tested TN TP FN FP
Quantitative result of the test

24. Effect of Increasing the Threshold
Disease
Test TP FN
Yes + - FP TN No TP
Se =
TP + FN TN
Sp =
TN + FP

25. Performance of a test and threshold
Sensitivity and specificity vary in opposite directions when changing the threshold (e.g. the cut-off in an ELISA)
The choice of a threshold is a compromise to best reach the objectives of the test
consequences of having false negatives?
consequences of having false positives?

26. Using several tests
One way out of the dilemma is to use several tests that complement each other
First use test with a high sensitivity (e.g. screening for HIV by ELISA, or for syphilis by TPHA)
Second use test with a high specificity (e.g. confirmation of HIV or syphilis by western blot)

27. ROC curves Receiver Operating Characteristics curve
Representation of relationship between sensitivity and specificity for a test
Simple tool to:
Help define best cut-off value of a test
Compare performance of two tests

28. Prevention of blood transfusion malaria: Choice of an indirect IFA threshold
Sensitivity (%)
100
1/20
1/10
1/40 80
1/80
1/160 60
IFA Dilutions
1/320 40
1/640 20 20 40 60 80
100
100 - Specificity (%): Proportion of false positives

29. Comparison of performance of IFA and ELISA IgM tests for detection of acute Q-fever
Sensitivity (%)
100 80
IFA
ELISA 60 40
Area under the ROC curve (AUC) 20 25 50 75
100
100 - Specificity (%)

30. Performance of a test in a population

31. How well does the test perform in a real population?
The test is now used in a real population
This population is made of
Affected individuals
Non-affected individuals
The proportion of affected individuals is the prevalence
Status of persons
Affected
Non-affected
Test
Positive
True +
False +
A+B
Negative
False -
True -
C+D
A+C
B+D
A+C+B+D

32. Predictive value of a positive test
The predictive value of a positive test is the probability that an individual testing positive is truly affected
Proportion of affected persons among those testing positive

33. Positive predictive value (PPV) of a test
Status of persons
Affected
Non-affected
Test
Positive A B
A+B
Negative C D
C+D
A + C
B+D
A+C+B+D
PPV = A / (A+B)
This is only valid for the sample of specimens tested 33

34. What factors influence the positive predictive value of a test?
Status of persons
Affected
Non-affected
Test
Positive A B
A+B
Negative C D
C+D
A + C
B+D
A+C+B+D
Sensitivity?
YES: To some extend.
Specificity?
YES: The more the test is specific, the more it will be negative for non-affected persons (less false-positive results).
Prevalence of the disease?
YES: Low prevalence: Low pre-test probability for positives. The test will pick up more false positives.
YES: High prevalence: High pre-test probability for positives. The test will pick up more true positives.

35. Positive predictive value of a test according to prevalence and specificity
PPV (%)

36. Predictive value of a negative test
The predictive value of a negative test is the probability that an individual testing negative is truly non-affected
Proportion of non-affected persons among those testing negative

37. Negative predictive value (NPV) of a test
Status of persons
Affected
Non-affected
Test
Positive A B
A+B
Negative C D
C+D
A+C
B+D
A+C+B+D
NPV = D / (C+D)
This is only valid for the sample of specimens tested 37

38. What factors influence the negative predictive value of a test?
Status of persons
Affected
Non-affected
Test
Positive A B
A+B
Negative C D
C+D
A+C
B+D
A+C+B+D
Sensitivity?
YES: The more the test is sensitive, the more it captures affected persons (less false negatives).
Specificity?
YES: But to a lesser extend.
Prevalence of the disease?
YES: Low prevalence: High pre-test probability for negatives. The test will pick up more true negatives.
YES: High prevalence: Low pre-test probability for negatives. The test will pick up more false negatives.

39. Negative predictive value of a test according to prevalence and sensitivity
NPV (%)

40. Relation between predictive values and sensitivity (Se), specificity (Sp), prevalence (Pr)
(1-Se)Pr + Sp(1-Pr)
Disease
(1-Sp)(1-Pr)
Se Pr No
Yes
Se Pr + (1-Sp)(1-Pr) Pr
1-Pr
Sp(1-Pr)
(1-Se)Pr
Test + -

41. Calculate PPV and NPV Pr) Sp)(1 (1 Pr Se PPV - + = Pr Se) (1 Pr) -

42. Relation between predictive values and sensitivity / specificity(1 Pr Se
PPV - + =
Increasing specificity  increasing PPV Pr
Se) (1
Pr) -
Sp(1
NPV + =
Increasing sensitivity  increasing NPV

43. Relation between predictive values and prevalence
Sp)(1 (1 Pr Se
PPV - + =
Increasing prevalence  increasing PPV Pr
Se) (1
Pr) -
Sp(1
NPV + =
Decreasing prevalence  increasing NPV

44. Example: Screening for acute Q-fever in two settings
ELISA IgM test
Sensitivity = 98%
Specificity = 95%
Population in low endemic area
Prevalence = 0.5%
Patients with atypical pneumonia
Prevalence = 20%
10,000 tests performed in each group

45. Example: Screening for acute Q-fever in a population in a low endemic area
IgM ELISA test sensitivity = 98%
IgM ELISA test specificity = 95%
Prevalence = 0.5%
Q-fever
Yes No
Total
IgM ELISA + 49
497
546 ­- 1
9,453
9,454 50
9,950
10,000
PPV = %
NPV = %

46. Example: Screening for acute Q-fever in patients with atypical pneumonia
IgM ELISA test sensitivity = 98%
IgM ELISA test specificity = 95%
Prevalence = 20%
Q-fever
Yes No
Total
IgM ELISA +
1,960
400
2,360 ­- 40
7,600
7,640
2,000
8,000
10,000
PPV = %
NPV = %

47. To whom predictive values matters most?
Look at denominators!
Persons testing positive
Persons testing negative
► To clinicians
probability that a individual with a positive test is really sick?
probability that a individual with a negative test is really healthy?
► To epidemiologists!
proportion of positive tests corresponding to true patients?
proportion of negative tests corresponding to healthy subjects?

48. Summary Sensitivity and specificity matter to laboratory specialists
Studied on panels of positives and negatives
Intrinsic characteristics of a test
Capacity to identify the affected
Capacity to identify the non-affected
Predictive values matter to clinicians and epidemiologists
Studied on homogeneous populations
Dependent on the disease prevalence
Performance of a test in real life
How to interpret a positive test
How to interpret a negative test

49. Where will you do your rain dance?
There?
Here?