1. 10 May 20101 Approaches to test evaluation Evan Sergeant AusVet Animal Health Services

2. Comparing tests  Kappa – how well tests agree  McNemar’s chi-sq – are tests significantly different?

3. Kappa  Expected no. both +ve = (157 x 155)/1122 = 21.7  Expected no. both -ve = (965 x 967)/1122 = 831.6  Total Agreement = 1052  Chance Agreement = 853.4  K=(1052-853.4)/(1122-853.4) = 0.739

4. McNemar Chi-Squared McNemar's Chi-squared test with continuity correction McNemar's chi-squared = 22.881, df = 1, p-value = 1.724e-06

5. OJD AGID and ELISA ELISA AGID+–Total +342155 –15154169 Total49175224  Enter data into epitools Application of diagnostic tests > compare 2 tests see kappa, McNemar’s and level of agreement

6. Kappa0.5496 SE for kappa = 00.0666 Z(kappa)8.25 p(kappa) - one-tailed0 Proportion positive agreement0.6538 Proportion negative agreement0.8953 Overall proportion agreement0.8393 McNemar's Chi sq0.6944 p(Chi sq)0.4

7. Gold Standard Tests  Use tests with perfect sensitivity and/or specificity to identify the true disease status of the individual from which the samples were taken.  What are the advantages and disadvantages of this approach?

8. Gold Standards Tests  Advantages Known disease status, Relatively simple calculations  Disadvantages May not exist, or be prohibitively expensive Rare diseases may only allow small sample size Disease may not be present in the country? Difficult to get representative (or even comparable) samples of diseased/non-diseased individuals

9. Exercises  Calculate Se and Sp for OJD AGID using data provided in OJD_AGID_Data.xls Calculate confidence limits using epitools

10. Non-gold standard methods  Do not depend on determining true infection status of individual.  Rely on statistical approaches to calculate best fit values for Se and Sp.  Tests must satisfy some important assumptions.

11. Comparison with a known reference test  Assumptions Independence of tests Se/Sp of reference test is known.  For ~100% specific reference test, Se(new test) = Number positive both tests / Total number positive to the reference test

12. Culture vs Serology  Estimate sensitivity of culture and serology (as flock tests)  Serology followed-up by histopathology to confirm flock status  Both tests 100% specificity (as flock tests)  How would you estimate sensitivity for these test(s)  Which test has better Se? Is the difference significant? All FlocksSerology +ve-veTotal PFC+ve583795 -ve5196201 Total63233296

13. Example  Se (PFC) = 58/63 = 92% (83% - 97%)  Se (Serology) = 58/95 = 61% (51% - 70%) Value Kappa0.6427 SE for kappa = 00.0559 Z(kappa)11.49 p(kappa) - one-tailed0 Proportion positive agreement0.7342 Proportion negative agreement0.9032 Overall proportion agreement0.8581 McNemar's Chi sq22.881 p(Chi sq)0

14. Estimation from routine testing data  test-positives are subject to follow-up and truly infected animals are identified and removed from the population  Can be used to estimate specificity when the disease is rare in the population of interest.  Sp = 1 – (Number of reactors / Total number tested)

15. Se and Sp of equine influenza ELISA  During the equine influenza outbreak in Australia, horses were tested by PCR and serology: to confirm infection; to demonstrate seroconversion and/or absence of infection >30 days later; As part of random and targeted surveillance for case detection, to confirm area status and for zone progression in presumed “EI free” areas.  How could you use the resulting data to estimate sensitivity and specificity of the ELISA?

16. Equine influenza ELISA  475 PCR-positive horses, 471 also positive on ELISA  1323 horses from properties in areas with no infection, 1280 ELISA negative  Analyse in Epitools Application of diagnostic tests> test evaluation against gold standard  Sergeant, E. S. G., Kirkland, P. D. & Cowled, B. D. 2009. Field Evaluation of an equine influenza ELISA used in New South Wales during the 2007 Australian outbreak response. Preventive Veterinary Medicine, 92, 382-385.

17. Point Estimate Lower 95% CL Upper 95% CL Sensitivity0.99160.97860.9977 Specificity0.96750.95650.9764

18. Mixture modelling  Assumptions observed distribution of test results (for a test with a continuous outcome reading such as an ELISA) is actually a mixture of two frequency distributions, one for infected individuals and one for uninfected individuals  Opsteegh, M., Teunis, P., Mensink, M., Zuchner, L., Titilincu, A., Langelaar, M. & van der Giessen, J. 2010. Evaluation of ELISA test characteristics and estimation of Toxoplasma gondii seroprevalence in Dutch sheep using mixture models. Preventive Veterinary Medicine.

19. Latent Class Analysis  What is Latent Class Analysis?  Maximum Likelihood  Bayesian

20. Maximum likelihood estimation  Assumptions The tests are independent conditional on disease status (the sensitivity [specificity] of one test is the same, regardless of the result of the other test); The tests are compared in two or more populations with different prevalence between populations; Test sensitivity and specificity are constant across populations; and There are at least as many populations as there are tests being evaluated.  TAGS software Hui, S. L. & Walter, S. D. 1980. Estimating the error rates of diagnostic tests. Biometrics, 36, 167-171.

21. TAGS  Open R – shortcut in root directory of stick  Open tags.R in text editor or word  Select all and copy/paste into R console  Type TAGS() and to run  Hui Walter example 2 tests for TB Test 1 = Mantoux Test 2 = Tine test

22.  Follow the prompts to enter data: Data set = new Name = test Number of tests = 2, Number of populations = 2 Reference population? = No (0) Enter results for each population from table below Best guesses use defaults Bootstrap CI = Yes (1000 iterations) Test 1Test 2Population 1Population 2 00528367 10431 01937 1114887 Data

23.  $Estimations pre1 pre2 Sp1 Sp2 Se1 Se2 Est 0.0268 0.7168 0.9933 0.9841 0.9661 0.9688 CIinf 0.0159 0.6911 0.9797 0.9684 0.9495 0.9540 CIsup 0.0450 0.7412 0.9978 0.9921 0.9774 0.9790

24. Bayesian estimation  What is Bayesian estimation? Combines prior knowledge/belief (what you think you know) with data to give best estimate Incorporates existing knowledge on parameters (Se, Sp, prevalence) “Priors” entered as probability (usually Beta) distributions Uses Monte Carlo simulation to solve Outputs also as probability distributions Can get very complex  Assumptions Independence of the tests Appropriate prior distributions chosen. Need information on prior probabilities Some methods can adjust for correlated tests Multiple tests in multiple populations

25.  Methods EpiTools (only allows one population so must have good information on one or more test characteristics) WinBUGS models

26. Bayesian analysis surra data Test 2 Test 1ELISA CATT+ve-veTotal +ve039 -ve0251 Total0290 Inputs for Bayesian analysis for revised sensitivity and specificity estimates Prior distributions for Bayesian analysis xnalphabeta Prev 11 Se_CATT (81%)100818220 Sp_CATT (99.4%)1601591602 Se_ELISA_2 (75%)100757626 Sp_ELISA_2 (97.5%)1201171184

27. EpiTools  Run EpiTools > Estimating true prevalence > Bayesian estimation with two tests  Enter parameters: Data from 2x2 table: 0, 39, 0, 251 Prevalence = Beta(1,1) (uniform = don’t know) Test 1 (CATT): Se = Beta(82, 20), Sp = Beta(160, 2) Test 2 (ELISA): Se = Beta(76, 26), Sp = Beta(118, 4) Starting values: 0, 38, 0, 245 Other values as defaults and click submit

28. Prevalence Sensitivity-1 Specificity-1 Sensitivity-2 Specificity-2 Minimum<0.00010.62190.85350.54750.9554 2.5%0.00010.72100.88180.65100.9789 Median0.00380.80640.91090.74180.9910 97.5%0.02010.87490.93540.82170.9973 Maximum0.05670.93700.95170.88910.9998 Mean0.00550.80440.91030.74060.9903 SD0.00550.03930.01360.04360.0048 Iterations20000