10 May Approaches to test evaluation Evan Sergeant AusVet Animal Health Services

Comparing tests  Kappa – how well tests agree  McNemar’s chi-sq – are tests significantly different?

Kappa  Expected no. both +ve = (157 x 155)/1122 = 21.7  Expected no. both -ve = (965 x 967)/1122 =  Total Agreement = 1052  Chance Agreement =  K=( )/( ) = 0.739

McNemar Chi-Squared McNemar's Chi-squared test with continuity correction McNemar's chi-squared = , df = 1, p-value = 1.724e-06

OJD AGID and ELISA ELISA AGID+–Total – Total  Enter data into epitools Application of diagnostic tests > compare 2 tests see kappa, McNemar’s and level of agreement

Kappa SE for kappa = Z(kappa)8.25 p(kappa) - one-tailed0 Proportion positive agreement Proportion negative agreement Overall proportion agreement McNemar's Chi sq p(Chi sq)0.4

Gold Standard Tests  Use tests with perfect sensitivity and/or specificity to identify the true disease status of the individual from which the samples were taken.  What are the advantages and disadvantages of this approach?

Gold Standards Tests  Advantages Known disease status, Relatively simple calculations  Disadvantages May not exist, or be prohibitively expensive Rare diseases may only allow small sample size Disease may not be present in the country? Difficult to get representative (or even comparable) samples of diseased/non-diseased individuals

Exercises  Calculate Se and Sp for OJD AGID using data provided in OJD_AGID_Data.xls Calculate confidence limits using epitools

Non-gold standard methods  Do not depend on determining true infection status of individual.  Rely on statistical approaches to calculate best fit values for Se and Sp.  Tests must satisfy some important assumptions.

Comparison with a known reference test  Assumptions Independence of tests Se/Sp of reference test is known.  For ~100% specific reference test, Se(new test) = Number positive both tests / Total number positive to the reference test

Culture vs Serology  Estimate sensitivity of culture and serology (as flock tests)  Serology followed-up by histopathology to confirm flock status  Both tests 100% specificity (as flock tests)  How would you estimate sensitivity for these test(s)  Which test has better Se? Is the difference significant? All FlocksSerology +ve-veTotal PFC+ve ve Total

Example  Se (PFC) = 58/63 = 92% (83% - 97%)  Se (Serology) = 58/95 = 61% (51% - 70%) Value Kappa SE for kappa = Z(kappa)11.49 p(kappa) - one-tailed0 Proportion positive agreement Proportion negative agreement Overall proportion agreement McNemar's Chi sq p(Chi sq)0

Estimation from routine testing data  test-positives are subject to follow-up and truly infected animals are identified and removed from the population  Can be used to estimate specificity when the disease is rare in the population of interest.  Sp = 1 – (Number of reactors / Total number tested)

Se and Sp of equine influenza ELISA  During the equine influenza outbreak in Australia, horses were tested by PCR and serology: to confirm infection; to demonstrate seroconversion and/or absence of infection >30 days later; As part of random and targeted surveillance for case detection, to confirm area status and for zone progression in presumed “EI free” areas.  How could you use the resulting data to estimate sensitivity and specificity of the ELISA?

Equine influenza ELISA  475 PCR-positive horses, 471 also positive on ELISA  1323 horses from properties in areas with no infection, 1280 ELISA negative  Analyse in Epitools Application of diagnostic tests> test evaluation against gold standard  Sergeant, E. S. G., Kirkland, P. D. & Cowled, B. D Field Evaluation of an equine influenza ELISA used in New South Wales during the 2007 Australian outbreak response. Preventive Veterinary Medicine, 92,

Point Estimate Lower 95% CL Upper 95% CL Sensitivity Specificity

Mixture modelling  Assumptions observed distribution of test results (for a test with a continuous outcome reading such as an ELISA) is actually a mixture of two frequency distributions, one for infected individuals and one for uninfected individuals  Opsteegh, M., Teunis, P., Mensink, M., Zuchner, L., Titilincu, A., Langelaar, M. & van der Giessen, J Evaluation of ELISA test characteristics and estimation of Toxoplasma gondii seroprevalence in Dutch sheep using mixture models. Preventive Veterinary Medicine.

Latent Class Analysis  What is Latent Class Analysis?  Maximum Likelihood  Bayesian

Maximum likelihood estimation  Assumptions The tests are independent conditional on disease status (the sensitivity [specificity] of one test is the same, regardless of the result of the other test); The tests are compared in two or more populations with different prevalence between populations; Test sensitivity and specificity are constant across populations; and There are at least as many populations as there are tests being evaluated.  TAGS software Hui, S. L. & Walter, S. D Estimating the error rates of diagnostic tests. Biometrics, 36,

TAGS  Open R – shortcut in root directory of stick  Open tags.R in text editor or word  Select all and copy/paste into R console  Type TAGS() and to run  Hui Walter example 2 tests for TB Test 1 = Mantoux Test 2 = Tine test

 Follow the prompts to enter data: Data set = new Name = test Number of tests = 2, Number of populations = 2 Reference population? = No (0) Enter results for each population from table below Best guesses use defaults Bootstrap CI = Yes (1000 iterations) Test 1Test 2Population 1Population Data

 $Estimations pre1 pre2 Sp1 Sp2 Se1 Se2 Est CIinf CIsup

Bayesian estimation  What is Bayesian estimation? Combines prior knowledge/belief (what you think you know) with data to give best estimate Incorporates existing knowledge on parameters (Se, Sp, prevalence) “Priors” entered as probability (usually Beta) distributions Uses Monte Carlo simulation to solve Outputs also as probability distributions Can get very complex  Assumptions Independence of the tests Appropriate prior distributions chosen. Need information on prior probabilities Some methods can adjust for correlated tests Multiple tests in multiple populations

 Methods EpiTools (only allows one population so must have good information on one or more test characteristics) WinBUGS models

Bayesian analysis surra data Test 2 Test 1ELISA CATT+ve-veTotal +ve039 -ve0251 Total0290 Inputs for Bayesian analysis for revised sensitivity and specificity estimates Prior distributions for Bayesian analysis xnalphabeta Prev 11 Se_CATT (81%) Sp_CATT (99.4%) Se_ELISA_2 (75%) Sp_ELISA_2 (97.5%)

EpiTools  Run EpiTools > Estimating true prevalence > Bayesian estimation with two tests  Enter parameters: Data from 2x2 table: 0, 39, 0, 251 Prevalence = Beta(1,1) (uniform = don’t know) Test 1 (CATT): Se = Beta(82, 20), Sp = Beta(160, 2) Test 2 (ELISA): Se = Beta(76, 26), Sp = Beta(118, 4) Starting values: 0, 38, 0, 245 Other values as defaults and click submit

Prevalence Sensitivity-1 Specificity-1 Sensitivity-2 Specificity-2 Minimum< % Median % Maximum Mean SD Iterations20000