Evaluating the Efficacy and Equivalence of Phytosanitary Risk Reduction Measures Mark Powell U.S. Department of Agriculture, Office of Risk Assessment and Cost-Benefit Analysis Washington, DC, USA International Plant Health Risk Analysis Workshop October 24-28, 2005 Niagara Falls, Ontario, Canada

Powell - Efficacy and Equivalence2 Motivation Efficacy - ISPM 2 (Guidelines for Pest Risk Analysis) Article 3.2 (Efficacy and Impact of the Options) - Calls for consideration of several factors in evaluating phytosanitary risk management options, including their “biological effectiveness.” Equivalence - SPS Agreement (Article 4): “Members shall accept the sanitary or phytosanitary measures of other Members as equivalent, even if these measures differ from their own or from those used by other Members trading in the same product, if the exporting Member objectively demonstrates to the importing Member that its measures achieve the importing Member's appropriate level of sanitary or phytosanitary protection.”

Powell - Efficacy and Equivalence3 Outline Describe three analytical methods to recurrent challenges faced by plant health risk analysts in evaluating the efficacy and equivalence of phytosanitary risk reduction measures: 1. Define the desired level of treatment efficacy as a performance standard 2. Combine information from a series of replicated treatment trials 3. Account for uncertainty in control mortality

Powell - Efficacy and Equivalence4 Define the desired level of treatment efficacy as a performance standard Process standards: –Zero survivors out of ~30,000 treated –Zero survivors out of 90,000 + treated Performance standards: –95% confidence of 1 x survival (0.0001) –95% confidence of 3.2 x survival (Probit 9) Couey & Chew Confidence Limits and Sample Size in Quarantine Research. J. Econ. Entomol. 79:

Powell - Efficacy and Equivalence5 Binomial Distribution Random number of successes (s) of n independent trials: s ~ Binomial(n,p) where: f(x) = Binomial probability mass function F(x) = Binomial cumulative distribution function p = probability of survival (define “success” = survivor) x = number of survivors (s) n = sample size C = confidence level (more precisely, prob(s>x|n,p))

Powell - Efficacy and Equivalence6 Calculate Sample Size from Binomial Distribution Special case where s = 0, simplifies to: C = prob(x>0|n,p) =1-(1-p) n n=[log(1-C)/log(1-p)] Set p = 1 x and C = 0.95: solve for n = 29,956 (~30,000) Set p = 3.2 x and C = 0.95: solve for n = 93,615 (90,000 + )

Powell - Efficacy and Equivalence7 Poisson Approximation to Binomial where: f(x) = Poisson probability mass function F(x) = Poisson cumulative distribution function x = number of survivors (s) C = confidence level (prob(s>x|λ)) λ ≈ np (for large n, small p, where p is binomial probability)

Powell - Efficacy and Equivalence8 Calculate Sample Size from Look-up Table C = 0.95Set p at desired level sλ=npSolve for n Source: Couey & Chew (1986, Table 1)

Powell - Efficacy and Equivalence9 C = 0.95p = sλ=npn , , , , , , , , , , ,600 Calculate Sample Size from Look-up Table

Powell - Efficacy and Equivalence10 C = 0.95p = sλ=npn , , , , , , , , , , ,600 Equivalent Treatment Efficacy Illustrates that different combinations of s and n are equivalent in terms of demonstrating the efficacy of phytosanitary treatment.

Powell - Efficacy and Equivalence11 Spreadsheet Snapshot Where: n 1 = sample size from look-up table for p = and C 1 = 0.95; n 2 = precise sample size for p = , C 2 = C is calculated in Excel © as: C = 1-BINOMDIST(s, n, p, 1). Sample size (n 2 ) is calculated numerically to desired precision using the Excel © add-in tool Solver ©. (Calculated values confirmed using SAS ©, where C = 1-CDF(‘BINOMIAL’,s, p, n).) Spreadsheet model provides greater numerical precision than look-up table for calculating sample size requirements (n), although need to consider measurement error for large counts. Provides greater flexibility to calculate sample size (n) required for any desired treatment efficacy (p) and confidence level (C), where C = p(s>x|n,p). sn1n1 C1C1 n2n2 C2C2 n 1 -n 2 030, , , , , ,

Powell - Efficacy and Equivalence12 Combining Confidence Levels in Assessing Treatment Efficacy One or more initial rounds of testing may be analyzed to determine how much more data is required to meet a defined treatment performance standard. Combining confidence levels. If an overall confidence level C is required, and the first round of testing provides C 1, then the level of confidence C 2 required for the second round is: C 2 = (C-C 1 )/(1-C 1 ) (Source: Cannon, R.M Demonstrating disease freedom – combining confidence levels. Preventive Veterinary Medicine. 52: ) Example: Design standard is 95% confidence that survival is no more than p. An initial round of testing provides 86% confidence that survival is no more than p. Applying the equation for combining confidence levels, the second round of testing has to provide 64% confidence that the survival is no more than p. Won’t find that value in a sample size look-up table. (Note: this equation also useful for pest prevalence surveillance.)

Powell - Efficacy and Equivalence13 Beta Distribution - Uncertainty about Binomial Probability (p) Recall that the binomial distribution applies to the random variable s: s ~ Binomial(n,p); where n & p are fixed (C = 1- F(x) = p(s>x|n,p)). To assess the efficacy and equivalence of phytosanitary treatments, we’d like to draw inferences about p, where p = prob(plant pest survival). p is a distribution of values, not a point estimate, with 0≤p≤1. Given observed data (n & s), the binomial probability (p): p ~ Beta(α,β) where: μ = α/(α+β) σ 2 = αβ/[(α+β) 2 (α+β+1)]

Powell - Efficacy and Equivalence14 Beta Parameter Estimation Different methods for estimating the parameters of Beta(α,β) Method 1: p prior ~ Uniform(0,1) α est = s+1 β est = n-s+1 For n = s = 0 (no data), Beta(1,1) ~ Uniform(0,1) Advantage: can handle case where s = 0 Disadvantage: estimate of binomial probability (p) is biased toward 0.5. (Probability of s in “future” trials is )

Powell - Efficacy and Equivalence15 Method 2: Method of Matching Moments (MoMM) Disadvantage: can’t handle case where s = 0 Advantage: unbiased estimate of binomial probability (p) Beta Parameter Estimation

Powell - Efficacy and Equivalence16 Beta – Uniform Prior n 1 = sample size from look-up table for p = (1E-04) and C = 0.95 s = number of survivors, α est = s+1, β est = n-s+1 p u = BETAINV(C, α, β) = 95% confidence level for p (prob(survival)) n1n1 30,00047,40063,00077,50091,500105,100 s α est β est 30,00147,40062,99977,49891,497105,096 mean est 3.33E E E E E E-05 var est 1.11E E E E E E-10 pupu 9.98E E E E-04

Powell - Efficacy and Equivalence17 Beta – Uniform Prior n 1 = sample size from look-up table for p = (1E-04) and C = 0.95 s = number of survivors, α est = s+1, β est = n-s+1 p u = BETAINV(C, α, β) = 95% confidence level for p (prob(survival)) n1n1 30,00047,40063,00077,50091,500105,100 s α est β est 30,00147,40062,99977,49891,497105,096 mean est 3.33E E E E E E-05 var est 1.11E E E E E E-10 pupu 9.98E E E E-04 Equivalent to the results calculated from the sample size look-up table.

Powell - Efficacy and Equivalence18 Beta – MoMM n2n2 30,000 47,40063,00077,50091,500 s Sample pn/a3.33E E E E E-05 Sample s 2 n/a1.11E-098.9E E E E-10 α est * β est 30,001*29,99747,39662,99577,49491,493 pupu 9.98E E E E E-04 n 2 = sample size s = number of survivors, α* = s+1, β* = n-s+1 p u = BETAINV(C, α, β) = 95% confidence level for p (survival)

Powell - Efficacy and Equivalence19 Beta – MoMM n2n2 30,000 47,40063,00077,50091,500 s Sample pn/a3.33E E E E E-05 Sample s 2 n/a1.11E-098.9E E E E-10 α est * β est 30,001*29,99747,39662,99577,49491,493 pupu 9.98E E E E E-04 n 2 = sample size s = number of survivors, α* = s+1, β* = n-s+1 p u = BETAINV(C, α, β) = 95% confidence level for p (survival) MoMM yields unbiased estimate of prob(survival)

Powell - Efficacy and Equivalence20 Comparison of Results For n = 47,400 s = 1 (s/n = 2 x ) Uniform prior ~ Beta(2, 47400) MoMM ~ Beta(1, 47397) μ = 2 x μ = 4 x 10 -5

Powell - Efficacy and Equivalence21 Comparison of Results Let n = 30,000 s = 0 Uniform prior ~ Beta(1, 30001) Let n = 30,000 s= 1 MoMM ~ Beta(0.9999, 29997) 95%ile =

Powell - Efficacy and Equivalence22 Simulation to Confirm MoMM Results p = s/n s~binomial(n,p) For SAS © users - DATA; DO I=1 TO 1E5; /* 1E5 = no. iterations */ P =(RANBIN(SEED,N,S/N))/N; OUTPUT; END; /* SEED = random number seed */ RUN; PROC UNIVARIATE; HISTOGRAM P; RUN; SN95%ILE(P) E E E E E-05

Powell - Efficacy and Equivalence23 Combine information from a series of replicated trials Bayes Rule implies: Likelihood function (in Excel © ) = BINOMDIST(s,n,p,0). Beginning with a suitable starting prior (e.g., p prior1 = uniform(0,1)), we can use Bayes rule to sequentially update the estimated treatment effect (p = prob(survival)) from a series of replicated trials. The posterior distribution of p after considering results of the i th trial becomes the prior distribution of p for considering the results of the (i+1) th trial.

Powell - Efficacy and Equivalence24 Sequential Updating TRIAL 1TRIAL 2TRIAL 3 survivors (s)s000 sample size (n)n10000 p (prop. survivors)priorlikproductposterior1likproductposterior2likproductposterior E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E-16 sum9.994E-021sum4.998E-011sum6.663E-011 Illustrative Example: 3 Replicate Trials of n = 10,000 with s = 0 survivors

Powell - Efficacy and Equivalence25 Sequential Updating TRIAL 1 TRIAL 2TRIAL 3 survivors (s)s 0 00 sample size (n)n p (prop. survivors)prior lik productposterior1likproductposterior2likproductposterior E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E-16 sum9.994E-021sum4.998E-011sum6.663E-011 Initial prior – all values of p are equally likely

Powell - Efficacy and Equivalence26 Sequential Updating TRIAL 1 TRIAL 2TRIAL 3 survivors (s)s0 00 sample size (n)n10000 p (prop. survivors)priorlikproductposterior1 likproductposterior2likproductposterior E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E-16 sum9.994E-021 sum4.998E-011sum6.663E-011 Update the estimated distribution of p based a series of trials, considering the new information sequentially.

Powell - Efficacy and Equivalence27 Sequential Updating TRIAL 1 TRIAL 2 TRIAL 3 survivors (s)s sample size (n)n p (prop. survivors)priorlikproductposterior1likproductposterior2 likproductposterior E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E-16 sum9.994E-021sum4.998E-011 sum6.663E-011

Powell - Efficacy and Equivalence28 Sequential Updating TRIAL 1TRIAL 2 TRIAL 3 survivors (s)s 00 0 sample size (n)n p (prop. survivors)priorlikproductposterior1likproductposterior2likproductposterior E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E-16 sum9.994E-021sum4.998E-011sum6.663E-011

Powell - Efficacy and Equivalence29 Sequential Updating Spreadsheet Sequential UpdatingTRIAL 1TRIAL 2TRIAL 3 survivors (s)s000 sample size (n)n10000 p (prop. survivors)priorlikproductposterior1likproductposterior2likproductposterior E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E-16 sum9.994E-021sum4.998E-011sum6.663E-011 Zoom in to take a closer look

Powell - Efficacy and Equivalence30 Sequential Updating Spreadsheet BDCFE Sum(E7:E10006) Sequential UpdatingTRIAL 1 survivors (s)s0 sample size (n)n10000 p (prop. survivors)priorlikproductposterior E =BINOMDIST(D$4,D$5,$B7,0)=C7*D7=E7/$E$ E =BINOMDIST(D$4,D$5,$B8,0)=C8*D8=E8/$E$10007 Discretized values of p in very small increments (1 x )

Powell - Efficacy and Equivalence31 Sequential Updating Spreadsheet BDCFE Sum(E7:E10006) Sequential UpdatingTRIAL 1 survivors (s)s0 sample size (n)n10000 p (prop. survivors)priorlikproductposterior E =BINOMDIST(D$4,D$5,$B7,0)=C7*D7=E7/$E$ E =BINOMDIST(D$4,D$5,$B8,0)=C8*D8=E8/$E$10007 Discretized values of p in very small increments (1 x ) Initial prior is that all values of p are equally likely

Powell - Efficacy and Equivalence32 Sequential Updating Spreadsheet BDCFE Sum(E7:E10006) Sequential UpdatingTRIAL 1 survivors (s)s0 sample size (n)n10000 p (prop. survivors)priorlikproductposterior E =BINOMDIST(D$4,D$5,$B7,0)=C7*D7=E7/$E$ E =BINOMDIST(D$4,D$5,$B8,0)=C8*D8=E8/$E$10007 Likelihood function in Excel © : BINOMDIST(s,n,p,0)

Powell - Efficacy and Equivalence33 Sequential Updating Spreadsheet BDCFE Sum(E7:E10006) Sequential UpdatingTRIAL 1 survivors (s)s0 sample size (n)n10000 p (prop. survivors)priorlikproductposterior =BINOMDIST(D$4,D$5,$B7,0)=C7*D7=E7/$E$ =BINOMDIST(D$4,D$5,$B8,0)=C8*D8=E8/$E$10007 s = D4 Likelihood function in Excel © : BINOMDIST(s,n,p,0)

Powell - Efficacy and Equivalence34 Sequential Updating Spreadsheet BDCFE Sum(E7:E10006) Sequential UpdatingTRIAL 1 survivors (s)s0 sample size (n)n10000 p (prop. survivors)priorlikproductposterior =BINOMDIST(D$4,D$5,$B7,0)=C7*D7=E7/$E$ =BINOMDIST(D$4,D$5,$B8,0)=C8*D8=E8/$E$10007 s = D4 n = D5 Likelihood function in Excel © : BINOMDIST(s,n,p,0)

Powell - Efficacy and Equivalence35 Sequential Updating Spreadsheet BDCFE Sum(E7:E10006) Sequential UpdatingTRIAL 1 survivors (s)s0 sample size (n)n10000 p (prop. survivors)priorlikproductposterior =BINOMDIST(D$4,D$5,$B7,0)=C7*D7=E7/$E$ =BINOMDIST(D$4,D$5,$B8,0)=C8*D8=E8/$E$10007 s = D4 n = D5 p values in Col B Likelihood function in Excel © : BINOMDIST(s,n,p,0)

Powell - Efficacy and Equivalence36 Sequential Updating Spreadsheet BDCFE Sum(E7:E10006) Sequential UpdatingTRIAL 1 survivors (s)s0 sample size (n)n10000 p (prop. survivors)priorlikproductposterior =BINOMDIST(D$4,D$5,$B7,0)=C7*D7=E7/$E$ =BINOMDIST(D$4,D$5,$B8,0)=C8*D8=E8/$E$10007 posterior is proportional to prior * lik

Powell - Efficacy and Equivalence37 Sequential Updating Spreadsheet BDCFE Sum(E7:E10006) Sequential UpdatingTRIAL 1 survivors (s)s0 sample size (n)n10000 p (prop. survivors)priorlikproductposterior =BINOMDIST(D$4,D$5,$B7,0)=C7*D7=E7/$E$ =BINOMDIST(D$4,D$5,$B8,0)=C8*D8=E8/$E$10007 posterior is proportional to prior * lik normalized posterior sums to 1

Powell - Efficacy and Equivalence38 Sequential Updating Spreadsheet BDCFE Sum(E7:E10006) Sequential UpdatingTRIAL 1 survivors (s)s0 sample size (n)n10000 p (prop. survivors)priorlikproductposterior =BINOMDIST(D$4,D$5,$B7,0)=C7*D7=E7/$E$ =BINOMDIST(D$4,D$5,$B8,0)=C8*D8=E8/$E$10007 posterior1 based on TRIAL 1 becomes prior for updating based on TRIAL 2 Prior2

Powell - Efficacy and Equivalence39 Sequential Updating

Powell - Efficacy and Equivalence40 Sequential Updating

Powell - Efficacy and Equivalence41 Sequential Updating

Powell - Efficacy and Equivalence42 Sequential Updating

Powell - Efficacy and Equivalence43 Comparison of Sequential Updating v. Beta Distribution Fit to Cumulative Data Sequential: 3 replicate trials of n =10,000 with s = 0 Cumulative: n = 30,000; s = 0; s ~ Beta(s+1,n-s+1) = Beta (1, 30001)

Powell - Efficacy and Equivalence44 Combine information from a series of replicated trials As evidence accumulates, the influence of the prior diminishes, and the data dominate the posterior distribution. Conveniently, when the cumulative sample size is large (n ~ 10,000), we can obtain a good approximation of the posterior distribution of p by simply analyzing the combined data from replicate trials to estimate the Beta distribution parameters: p posterior → p ~ Beta(α est, β est ) where: n = ∑n i s = ∑s i i = 1 to j trials

Powell - Efficacy and Equivalence45 Account for uncertainty in control mortality Abbott’s Formula to correct for control mortality: Conventionally, p control treated as a deterministic value (point estimate). Fails to account for uncertainty in control mortality. Source: Abbott, W.S A method of computing the effectiveness of an insecticide. J Econ Entomol 18:

Powell - Efficacy and Equivalence46 Account for uncertainty in control mortality p control ~ Beta(α 1, β 1 ) p treat ~ Beta(α 2, β 2 ) p(survival, corrected for control mortality and accounting for uncertainty):

Powell - Efficacy and Equivalence47 Account for uncertainty in control mortality: empirical example Ex: 90% survival in control trial with n = 100 Control trial Treatment trial n10033,500 s900 sample p0.9n/a sample s n/a α est β est 9.833,501 p corrected estimated by Monte Carlo Simulation, © Summary of Results: 95th %ile = 9.95 x Extra treatment trial sample size ~ 3,500 required to demonstrate p ≤ with 95% confidence.

Powell - Efficacy and Equivalence48 Disclaimers The opinions expressed herein are the views of the author and do not necessarily reflect the official policy or position of the United States Department of Agriculture. Reference herein to any specific commercial products, process, or service by trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government.