10 May Understanding diagnostic tests Evan Sergeant AusVet Animal Health Services

10 May Welcome and Introductions welcome housekeeping Introductions –Who are you, where from/role –Expectations from course – what would you like to learn –How you would expect to apply what you learn in your current/future work –Specific topics of interest to you

10 May Workshop outcomes At the end of the workshop, participants should be able to: Describe and interpret measures of test precision and validity; Interpret test results based on the use of imperfect tests; Describe common methods for evaluating tests; and Plan and implement surveillance activities

10 May Course Outline Day 1 Introductions, Expectations and Outcomes, Course overview EpiTools and USB stick Understanding test characteristics Sampling populations  Day 2 Field trip – sampling in practice Aggregate (herd) test performance Application of tests Approaches to test evaluation

10 May Course Outline  Day 3 Introduction to animal health information and surveillance Planning an effective surveillance activity Measures of disease  Day 4 Surveys for freedom Risk-based surveillance for freedom Prevalence surveys

Course outline  Day 5 Prevalence surveys (cont) Case detection surveillance Data management and analysis Revisit field trip – results Wrap-up and close

EpiTools and USB Web-based epidemiological calculator and utilities Available at: Provided on USB stick Instructions in word document in root directory of stick USB also has “Resources” folder, containing materials for case studies and activities during workshop

10 May Diagnosis and screening What is a test? Why do we use tests? What is the difference between using tests for diagnosis and screening? How do we measure test performance? –What is a good test?

10 May Test performance Accuracy vs precision How do we measure precision? How do we measure accuracy?

Valid Precise

Valid Imprecise

Invalid Precise

Invalid Imprecise

Assessing test precision  Repeatability tests performed under conditions that are as constant as possible in the one laboratory by one operator using the same equipment over a short period of time

Assessing test precision  Reproducibility tests performed under widely varying conditions in different laboratories at different times by different operators

Assessing test precision  Robustness a measure of an assay’s capacity to remain unaffected by small, but deliberate, variations in method parameters, and provides an indication of its reliability during normal usage.

Assessing test precision  Coefficient of variation The ratio of the standard deviation of a series of values to its mean (usually for repeated testing on the same sample)  Correlation coefficient Correlation of results of duplicate testing of large numbers of individual samples

Statistical Control Chart

Cusum chart

Regression Analysis

Measures of accuracy  How do we measure test accuracy?

Diagnostic Sensitivity  The proportion of animals with the disease of interest who test positive. the probability that a test will correctly identify those animals that are infected (Pr T+|D+) True Positive Rate = 1 – false negative rate

Diagnostic Specificity  The proportion of animals without the disease of interest that test negative. the probability that a test will correctly identify those animals that are not infected (Pr T-|D-). True Negative Rate = 1 – false positive rate

Diagnostic or Analytical  What is the difference between: diagnostic and analytical sensitivity? diagnostic and analytical specificity?

Exercise  What is the sensitivity of a test in following situations? 10 infected animals tested, 9 positive; 100 infected animals tested, 90 positive; 75 infected, 73 positive  What is the specificity of a test in following situations? 100 uninfected animals tested, 99 negative 1000 uninfected animals tested, 990 negative 453 uninfected, 420 negative  How confident are you in each case?

 Calculate confidence intervals using epitools: Application of diagnostic tests > Test evaluation against gold standard Disease status = reference test

Small sample size (10 & 100) Point Estimate Lower 95% CL Upper 95% CL Sensitivity Specificity Large sample size (100 & 1000) Point Estimate Lower 95% CL Upper 95% CL Sensitivity Specificity

How do we set a cut-off?

Relationship between Se & Sp

ROC Curve

Exercise  Data from ELISA_ROC_Data.xls EpiTools > application of diagnostic tests > ROC analysis Paste data to calculate Se/Sp for a range of cut-off values What is a reasonable cut-off? Why?

Two-graph ROC curve

Factor to consider  Use of test  Which is more important: high Se or high Sp  Multiple cut-off values: Positive vs inconclusive/suspect For different purposes

10 May Interpreting Individual tests Testing for brucellosis in cattle –Testing a single animal –You know that on average 1% of animals in the area are infected –Test Se = 99%, Sp = 95% –You get a positive test result – what is your interpretation? –What about for a negative result? –What difference would it make if the animal was from a herd that you knew had a 20% infection rate Work it out using EpiTools (Positive and Negative predictive values)

10 May Results Positive predictive value Negative predictive value Prior probability = 1% 16.7%99.99% Prior probability = 20% 83.2%99.7%  Se = 99%  Sp = 95%  Try experimenting with different values for Se and Sp and see which they affect more

10 May Calculating PPV/NPV Scenario tree to calculate Formulae in book What sort of test do we need for high PPV? How can we achieve this? What about for NPV?

10 May Combining tests What is testing in series and parallel? How are they interpreted? What effect does this have on Se and Sp overall? What are some examples of where we use series or parallel testing? Try and use unrelated tests

Series and parallel  Calculate Se and Sp: Test 1: Se = 95%, Sp = 95% Test 2: Se = 60%, Sp = 99%

 Series: Se = 57%, Sp = 1  Parallel: Se = 98%, Sp = 94%