1. 10 May 20101 Understanding diagnostic tests Evan Sergeant AusVet Animal Health Services

2. 10 May 20102 Welcome and Introductions welcome housekeeping Introductions –Who are you, where from/role –Expectations from course – what would you like to learn –How you would expect to apply what you learn in your current/future work –Specific topics of interest to you

3. 10 May 20103 Workshop outcomes At the end of the workshop, participants should be able to: Describe and interpret measures of test precision and validity; Interpret test results based on the use of imperfect tests; Describe common methods for evaluating tests; and Plan and implement surveillance activities

4. 10 May 20104 Course Outline Day 1 Introductions, Expectations and Outcomes, Course overview EpiTools and USB stick Understanding test characteristics Sampling populations  Day 2 Field trip – sampling in practice Aggregate (herd) test performance Application of tests Approaches to test evaluation

5. 10 May 20105 Course Outline  Day 3 Introduction to animal health information and surveillance Planning an effective surveillance activity Measures of disease  Day 4 Surveys for freedom Risk-based surveillance for freedom Prevalence surveys

6. Course outline  Day 5 Prevalence surveys (cont) Case detection surveillance Data management and analysis Revisit field trip – results Wrap-up and close

7. EpiTools and USB Web-based epidemiological calculator and utilities Available at: http://epitools.ausvet.com.au http://epitools.ausvet.com.au Provided on USB stick Instructions in word document in root directory of stick USB also has “Resources” folder, containing materials for case studies and activities during workshop

8. 10 May 20108 Diagnosis and screening What is a test? Why do we use tests? What is the difference between using tests for diagnosis and screening? How do we measure test performance? –What is a good test?

9. 10 May 20109 Test performance Accuracy vs precision How do we measure precision? How do we measure accuracy?

10. Valid Precise

11. Valid Imprecise

12. Invalid Precise

13. Invalid Imprecise

14. Assessing test precision  Repeatability tests performed under conditions that are as constant as possible in the one laboratory by one operator using the same equipment over a short period of time

15. Assessing test precision  Reproducibility tests performed under widely varying conditions in different laboratories at different times by different operators

16. Assessing test precision  Robustness a measure of an assay’s capacity to remain unaffected by small, but deliberate, variations in method parameters, and provides an indication of its reliability during normal usage.

17. Assessing test precision  Coefficient of variation The ratio of the standard deviation of a series of values to its mean (usually for repeated testing on the same sample)  Correlation coefficient Correlation of results of duplicate testing of large numbers of individual samples

18. Statistical Control Chart

19. Cusum chart

20. Regression Analysis

21. Measures of accuracy  How do we measure test accuracy?

22. Diagnostic Sensitivity  The proportion of animals with the disease of interest who test positive. the probability that a test will correctly identify those animals that are infected (Pr T+|D+) True Positive Rate = 1 – false negative rate

23. Diagnostic Specificity  The proportion of animals without the disease of interest that test negative. the probability that a test will correctly identify those animals that are not infected (Pr T-|D-). True Negative Rate = 1 – false positive rate

24. Diagnostic or Analytical  What is the difference between: diagnostic and analytical sensitivity? diagnostic and analytical specificity?

25. Exercise  What is the sensitivity of a test in following situations? 10 infected animals tested, 9 positive; 100 infected animals tested, 90 positive; 75 infected, 73 positive  What is the specificity of a test in following situations? 100 uninfected animals tested, 99 negative 1000 uninfected animals tested, 990 negative 453 uninfected, 420 negative  How confident are you in each case?

26.  Calculate confidence intervals using epitools: Application of diagnostic tests > Test evaluation against gold standard Disease status = reference test

27. Small sample size (10 & 100) Point Estimate Lower 95% CL Upper 95% CL Sensitivity 0.90.5550.9975 Specificity 0.990.94550.9997 Large sample size (100 & 1000) Point Estimate Lower 95% CL Upper 95% CL Sensitivity 0.90.82380.951 Specificity 0.990.98170.9952

28. How do we set a cut-off?

29. Relationship between Se & Sp

30. ROC Curve

31. Exercise  Data from ELISA_ROC_Data.xls EpiTools > application of diagnostic tests > ROC analysis Paste data to calculate Se/Sp for a range of cut-off values What is a reasonable cut-off? Why?

32. Two-graph ROC curve

33. Factor to consider  Use of test  Which is more important: high Se or high Sp  Multiple cut-off values: Positive vs inconclusive/suspect For different purposes

34. 10 May 201034 Interpreting Individual tests Testing for brucellosis in cattle –Testing a single animal –You know that on average 1% of animals in the area are infected –Test Se = 99%, Sp = 95% –You get a positive test result – what is your interpretation? –What about for a negative result? –What difference would it make if the animal was from a herd that you knew had a 20% infection rate Work it out using EpiTools (Positive and Negative predictive values)

35. 10 May 201035 Results Positive predictive value Negative predictive value Prior probability = 1% 16.7%99.99% Prior probability = 20% 83.2%99.7%  Se = 99%  Sp = 95%  Try experimenting with different values for Se and Sp and see which they affect more

36. 10 May 201036 Calculating PPV/NPV Scenario tree to calculate Formulae in book What sort of test do we need for high PPV? How can we achieve this? What about for NPV?

37. 10 May 201037 Combining tests What is testing in series and parallel? How are they interpreted? What effect does this have on Se and Sp overall? What are some examples of where we use series or parallel testing? Try and use unrelated tests

38. Series and parallel  Calculate Se and Sp: Test 1: Se = 95%, Sp = 95% Test 2: Se = 60%, Sp = 99%

39.  Series: Se = 57%, Sp = 1  Parallel: Se = 98%, Sp = 94%