Evaluating Diagnostic Tests Payam Kabiri, MD. PhD. Clinical Epidemiologist Tehran University of Medical Sciences

Seven question to evaluate the utility of a diagnostic test Can the test be reliably performed? Was the test evaluated on an appropriate population? Was an appropriate gold standard used? Was an appropriate cut-off value chosen to optimize sensitivity and specificity?

What are the positive and negative likelihood ratios? How well does the test perform in specific populations? What is the balance between cost of the disease and cost of the test? Seven question to evaluate the utility of a diagnostic test

4 Which one of these test is the best for SLE Dx? TestSensitivity%Specificity% ANA9980 dsDNA7095 ssDNA8050 Histone Nucleoprotein5850 Sm2599 RNP PCNA595

5 Diagnostic Tests Characteristics Sensitivity Specificity Predictive Value Likelihood Ratio

6 Validity of Screening Tests a d c b True Disease Status + - Results of Screening Test + - Sensitivity: The probability of testing positive if the disease is truly present Sensitivity = a / (a + c)

7 Validity of Screening Tests a d c b True Disease Status + - Results of Screening Test + - Specificity: The probability of screening negative if the disease is truly absent Specificity = d / (b + d)

Two-by-two tables can also be used for calculating the false positive and false negative rates. The false positive rate = false positives / (false positives + true negatives). It is also equal to 1- specificity.

The false negative rate = false negatives / (false negatives + true positives). It is also equal to 1 – sensitivity. An ideal test maximizes both sensitivity and specificity, thereby minimizing the false positive and false negative rates.

10 Validity of Screening Tests Breast Cancer + - Physical Exam and Mammo- graphy + - Sensitivity: a / (a + c) Sensitivity = Specificity: d / (b + d) Specificity =

11 Validity of Screening Tests Breast Cancer + - Physical Exam and Mammo- graphy + - Sensitivity:a / (a + c) Sensitivity = 132 / ( ) = 74.6% Specificity: d / (b + d) Specificity = / ( ) = 98.5%

2 X 2 table Disease Test Sensitivity Positive predictive value

Natural Frequencies Tree Population 100

In Every 100 People, 4 Will Have The Disease Disease + 4 Disease - 96 Population 100 If these 100 people are representative of the population at risk, the assessed rate of those with the disease (4%) represents the PREVALENCE of the disease – it can also be considered the PRE-TEST PROBABILITY of having the disease

OF THE 4 PEOPLE WITH THE DISEASE, THE TEST WILL DETECT 3 Disease + 4 Disease - 96 Test + 3 Test - 1 Population 100 In other words, the sensitivity is 75%

AMONG THE 96 PEOPLE WITHOUT THE DISEASE, 7 WILL TEST POSITIVE Disease + 4 Disease - 96 Test + 7 Test - 89 Test + 3 Test - 1 Population 100 In other words, the specificity is 93%

POSITIVE PREDICTIVE VALUE = 30% AMONG THOSE WHO TEST POSITIVE, 3 IN 10 WILL ACTUALLY HAVE THE DISEASE Disease + 4 Disease - 96 Test + 7 Test - 89 Test + 3 Test - 1 Population 100 This is also the POST-TEST PROB- ABILITY of having the disease

NEGATIVE PREDICTIVE VALUE = 99% AMONG THOSE WHO TEST NEGATIVE, 89 OF 90 WILL NOT HAVE THE DISEASE Disease + 4 Disease - 96 Test + 7 Test - 89 Test + 3 Test - 1 Population 100

CONVERSELY, IF SOMEONE TESTS NEGATIVE, THE CHANCE OF HAVING THE DISEASE IS ONLY 1 IN 90 Disease + 4 Disease - 96 Test + 7 Test - 89 Test + 3 Test - 1 Population 100

PREDICTIVE VALUES AND CHANGING PREVALENCE Disease + 4 Disease Population 1000 Prevalence reduced by an order of magnitude from 4% to 0.4%

PREDICTIVE VALUE AND CHANGING PREVALENCE Disease + 4 Disease Test + 70 Test Test + 3 Test - 1 Population 1000 Sensitivity and Specificity unchanged

POSITIVE PREDICTIVE VALUE = 4% POSITIVE PREDICTIVE VALUE AT LOW PREVALENCE Disease + 4 Disease Test + 70 Test Test + 3 Test - 1 Population 1000 Previously, PPV was 30%

NEGATIVE PREDICTIVE VALUE >99% NEGATIVE PREDICTIVE VALUE AT LOW PREVALENCE Disease + 4 Disease Test + 70 Test Test + 3 Test - 1 Population 1000 Previously, NPV was 99%

Prediction Of Low Prevalence Events Even highly specific tests, when applied to low prevalence events, yield a high number of false positive results Because of this, under such circumstances, the Positive Predictive Value of a test is low However, this has much less influence on the Negative Predictive Value

Relationship Between Prevalence and Predictive Value Based on a test with 90% sensitivity and 82% specificity Difference between PPV and NPV relatively small Difference between PPV and NPV relatively large

Relationship Between Prevalence And Predictive Value Based on a test with 75% sensitivity and 93% specificity Prevalence Predictive Value

Performance of A Test With Changing Prevalence A : Sensitivity = Specificity = 0.9 LR+ = 9.0 B : Sensitivity = Specificity = 0.7 LR+ = 3.0 C : Sensitivity = Specificity = 0.5 LR+ = 1.0 POST-TEST PROBABILITY

2 X 2 table

Sensitivity The proportion of people with the diagnosis (N=4) who are correctly identified (N=3) Sensitivity = a/(a+c) = 3/4 = 75% FALSE NEGATIVES

Specificity The proportion of people without the diagnosis (N=96) who are correctly identified (N=89) Specificity = d/(b+d) = 89/96 = 93% FALSE POSITIVES

31 Value of a diagnostic test depends on the prior probability of disease Prevalence (Probability) = 5% Sensitivity = 90% Specificity = 85% PV+ = 24% PV- = 99% Test not as useful when disease unlikely Prevalence (Probability) = 90% Sensitivity = 90% Specificity = 85% PV+ = 98% PV- = 49% Test not as useful when disease likely

A Test With Normally Distributed Values Negative Positive Degree of ‘positivity’ on test % of Group DISEASED NON-DESEASED Test cut-off Assessing the performance of the test assumes that these two distributions remain constant. However, each of them will vary (particularly through spectrum or selection bias)

CASESNON-CASES Performance of A Diagnostic Test Negative Positive Degree of ‘positivity’ on test % of Group DISEASED NON-DESEASED Test cut-off FALSE NEGATIVES FALSE POSITIVES

Minimising False Negatives: A Sensitive Test Negative Positive Degree of ‘positivity’ on test % of Group DISEASED NON- DESEASED Test cut-off Cut-off shifted to minimise false negatives ie to optimise sensitivity CONSEQUENCES: - Specificity reduced - A Negative result from a seNsitive test rules out the diagnosis - snNout CASESNON-CASES

Minimising False Positives: A Specific Test Negative Positive Degree of ‘positivity’ on test % of Group DISEASED NON-DESEASED Test cut-off Cut-off shifted to minimise false positives ie to optimise specificity CONSEQUENCES: - Sensitivity reduced - A Positive result from a sPecific test rules in the diagnosis - spPin

Non-diseasedDiseased Evaluation Result Value Or Subjective Judgment Of Likelihood That Case Is Diseased Threshold Receiver Operating Characteristics (ROC)

Non-diseased Centers Diseased Centers Test result value or subjective judgment of likelihood that case is diseased Threshold

Non-diseased Centers Diseased Centers Cutoff point more typically:

Non-diseased cases Diseased cases FP rate more typically:

Non-diseased Centers Diseased Centers TP rate more typically:

Threshold TPF, sensitivity FPF, 1-specificity less aggressive mindset Non-diseased Centers Diseased Centers

Threshold TPF, sensitivity FPF, 1-specificity moderate mindset Non-diseased cases Diseased cases

Threshold TPF, sensitivity FPF, 1-specificity more aggressive mindset Non-diseased cases Diseased cases

Threshold Non-diseased cases Diseased cases TPF, sensitivity FPF, 1-specificity Entire ROC curve

TPF, sensitivity FPF, 1-specificity Entire ROC curve chance line

Check this out: c/Findex.htm c/Findex.htm

Likelihood Ratios

Pre-test & post-test probability Pre-test probability of disease can be compared with the estimated later probability of disease using the information provided by a diagnostic test. The difference between the previous probability and the later probability is an effective way to analyze the efficiency of a diagnostic method.

It tells you how much a positive or negative result changes the likelihood that a patient would have the disease. The likelihood ratio incorporates both the sensitivity and specificity of the test and provides a direct estimate of how much a test result will change the odds of having a disease

The likelihood ratio for a positive result (LR+) tells you how much the odds of the disease increase when a test is positive. The likelihood ratio for a negative result (LR-) tells you how much the odds of the disease decrease when a test is negative.

Positive & Negative Likelihood Ratios We can judge diagnostic tests: positive and negative likelihood ratios. Like sensitivity and specificity, are independent of disease prevalence.

52 Likelihood Ratios (Odds) The probability of a test result in those with the disease divided by the probability of the result in those without the disease. How many more times (or less) likely a test result is to be found in the disease compared with the non-diseased.

Positive Likelihood Ratios This ratio divides the probability that a diseased patient will test positive by the probability that a healthy patient will test positive. The positive likelihood ratio +LR = sensitivity/(1 – specificity)

False Positive Rate The false positive rate = false positives / (false positives + true negatives). It is also equal to 1- specificity. The false negative rate = false negatives / (false negatives + true positives). It is also equal to 1 – sensitivity.

Positive Likelihood Ratios It can also be written as the true positive rate/false positive rate. Thus, the higher the positive likelihood ratio, the better the test (a perfect test has a positive likelihood ratio equal to infinity).

Negative Likelihood Ratio This ratio divides the probability that a diseased patient will test negative by the probability that a healthy patient will test negative. The negative likelihood ratio –LR = (1 – sensitivity)/specificity.

False Negative Rate The false negative rate = false negatives / (false negatives + true positives). It is also equal to 1 – sensitivity.

Negative Likelihood Ratio It can also be written as the false negative rate/true negative rate. Therefore, the lower the negative likelihood ratio, the better the test (a perfect test has a negative likelihood ratio of zero).

Positive & Negative Likelihood Ratios Although likelihood ratios are independent of disease prevalence, their direct validity is only within the original study population.

60 Probability of Disease Pre-test probability of disease = disease prevalence Post-test probability of disease =  If normal, c/(c+d)  If negative, a/(a+b) Disease present, gold standard Disease absent, gold standard False positives (b)True positives (a)Test result positive True negatives (d)False negatives (c)Test result negative

Bayes Theorem Post-test Odds = Likelihood Ratio X Pre-test Odds

62 Using Likelihood Ratios to Determine Post- Test Disease Probability

Pre-test & post-test probability “Post-test probability” depends on the accuracy of the diagnostic test and the pre-test probability of disease A test result cannot be interpreted without some knowledge of the pre-test probability

Where does “pre-test probability” come from? Clinical experience Epidemiological data “Clinical decision rules” Guess

what is the likelihood that this patient has the disease? A disease with a prevalence of 30% must be diagnosed. There is a test for this disease. It has a sensitivity of 50% and a specificity of 90%.

Likelihood Ratios Sensitivity 1 – Specificity = 0.88 / (1 – 0.82) = 4.89 This means that Anne’s positive FNA biopsy will be approx. 5 times as likely to be seen with, as opposed to without, thyroid cancer. SensitivitySpecificity FNA Biopsy88%82% From: J Clin End & Metab. 2006; 91(11):

Prevalence of 30% Sensitivity of 50% Specificity of 90% – 63 = positive tests in total of which 15 have the disease About 70% Disease +ve Disease -ve 63 15

Likelihood Disease + 4 Test + 3 Test - 1 Population 100 The likelihood that someone with the disease will have a positive test is ¾ or 75% This is the same as the sensitivity

Likelihood II Disease - 96 Test + 7 Test - 89 Population 100 The likelihood that someone without the disease will have a positive test is 7/96 or 7% This is the same as the (1-specificity)

Likelihood Ratio Likelihood of Positive Test in the Absence of the Disease Sensitivity 1- Specificity = = 10.7 Likelihood of Positive Test Given The Disease = Likelihood Ratio A Likelihood Ratio of 1.0 indicates an uninformative test (occurs when sensitivity and specificity are both 50%) The higher the Likelihood Ratio, the better the test (other factors being equal) =

Diagnostic Odds Ratio The Diagnostic Odds Ratio is the ratio of odds of having the diagnosis given a positive test to those of having the diagnosis given a negative test Potentially useful as an overall summary measure, but only in conjunction with other measures (LR, sensitivity, specificity)

Is there an easier way?

Likelihood Ratio And Pre- And Post-test Probabilities For a given test with a given likelihood ratio, the post-test probability will depend on the pre-test probability (that is, the prevalence of the condition in the sample being assessed)

Sensitivity Analysis of A Diagnostic Test Value95% CI Pre-test probability 35% 26% to 44%

Sensitivity Analysis of A Diagnostic Test Applying the 95% confidence intervals above to the nomogram, the post-test probability is likely to lie in the range 55-85% Value95% CI Pre-test probability 35%26% to 44% Likelihood ratio to 8.5

Applying A Diagnostic Test In Different Settings  The Positive Predictive Value of a test will vary (according to the prevalence of the condition in the chosen setting)  Sensitivity and Specificity are usually considered properties of the test rather than the setting, and are therefore usually considered to remain constant  However, sensitivity and specificity are likely to be influenced by complexity of differential diagnoses and a multitude of other factors (cf spectrum bias)

77 Likelihood Ratios (Odds) This is an alternative way of describing the performance of a diagnostic test. Similar to S and S, and can be used to calculate the probability of disease after a positive or negative test (predictive value). Advantage of this is that it can be used at multiple levels of test results.

78 What is this second fraction? Likelihood Ratio Positive Multiplied by any patient’s pretest odds gives you their posttest odds. Comparing LR+ of different tests is comparing their ability to “rule in” a diagnosis. As specificity increases LR+ increases and PPV increases (Sp P In)

79 Clinical interpretation of post- test probability Disease ruled out Disease ruled in If you are here, Test will help you to go toward one end of this probability, either 0 or 1 to get the final decision.

Values of Positive and Negative Likelihood Ratios (LR) LR Poor-fairGoodExcellent Positive likelihood ratio >10 Negative likelihood ratio <0.1

Likelihood Ratios & You Allows us to determine the accuracy with which a test identifies the target disorder As the LR becomes larger, the likelihood of the target disease increases: Likelihood ratioInterpretation >10Strong evidence to rule in disease 5-10Moderate evidence to rule in disease 2-5Weak evidence to rule in disease 0.5-2No significant change in the likelihood of disease Weak evidence to rule out disease Moderate evidence to rule out disease <0.1Strong evidence to rule out disease

82 Advantages of LRs The higher or lower the LR, the higher or lower the post-test disease probability Which test will result in the highest post-test probability in a given patient? The test with the largest LR+ Which test will result in the lowest post-test probability in a given patient? The test with the smallest LR-

83 Advantages of LRs Clear separation of test characteristics from disease probability.

84 Likelihood Ratios - Advantage Provide a measure of a test’s ability to rule in or rule out disease independent of disease probability Test A LR+ > Test B LR+  Test A PV+ > Test B PV+ always! Test A LR- < Test B LR-  Test A PV- > Test B PV- always!

85 Predictive Values Alternate formulations:Bayes’ Theorem PV+ = Se  Pre-test Prevalence Se  Pre-test Prevalence + (1 - Sp)  (1 - Pre-test Prevalence) High specificity to “rule-in” disease PV- = Sp  (1 - Pre-test Prevalence) Sp  (1 - Pre-test Prevalence) + (1 - Se)  Pre-test Prevalence High sensitivity to “rule-out” disease

86 Clinical Interpretation: Predictive Values

87 If Predictive value is more useful why not reported? Should they report it? Only if everyone is tested. And even then. You need sensitivity and specificity from literature. Add YOUR OWN pretest probability.

88 So how do you figure pretest probability? Start with disease prevalence. Refine to local population. Refine to population you serve. Refine according to patient’s presentation. Add in results of history and exam (clinical suspicion). Also consider your own threshold for testing.

89 Pretest Probability: Clinical Significance Expected test result means more than unexpected. Same clinical findings have different meaning in different settings (e.g.scheduled versus unscheduled visit). Heart sound, tender area. Neurosurgeon. Lupus nephritis.

90 What proportion of all patients will test positive? Diseased X sensitivity + Healthy X (1-specificity) Prevalence X sensitivity + (1-prevalence)(1-specificity) We call this “test prevalence” i.e. prevalence according to the test.

Some Examples Diabetes mellitus (type 2) Check out this:

Some Examples from Essential Evidence Plus DiseaseLink Address Diabetes Mellitus (type 2) Deep Vein Thrombosis Arrhythmia (Atrial Fibrillation & Flutter)

93 Which one of these test is the best for SLE Dx? TestSensitivitySpecificityLR(+) ANA dsDNA ssDNA Histone Nucleoprotein Sm RNP PCNA5951

Was it clear enough !

Key References Sedlmeier P and Gigerenzer G. Teaching Bayesian reasoning in less than two hours. Journal of Experimental Psychology: General. 130 (3): , Knotternus JA (ed). The Evidence Base of Clinical Diagnosis. London: BMJ Books, Sackett DL, Haynes RB, Guyatt G, and Tugwell P. Clinical Epidemiology : A Basic Science for Clinical Medicine. Boston, Mass: Little, Brown & Co, Loong TW. Understanding sensitivity and specificity with the right side of the brain. BMJ 2003: 327:

بزنید ! اگر میل داشتید