A Nonparametric Tolerance Interval Approach to Immunogenicity Assay Cut Point Determination Jason Zhang, Senior Principal Statistician Joint work with Wenjia Li, Harry Yang and Lorin Roskos Nonclinical Biostatistics Conference, 2017 Rutgers University

Immunogenicity Immunogenicity (IM) Unwanted immune response to large-molecule therapeutics, called anti-drug antibody (ADA) Potentially affect drug efficacy, safety Clinical effect: from no effect at all to extreme harmful effects A concern for clinicians, manufacturers and regulatory agencies Well developed and validated assay is needed for ADA detection, characterization

A Multiple-Tiered Approach for Assay Validation

MSD Streptavidin-Coated Plate Assay Format Example Universal assay capable of detecting most immunoglobulin isotypes Species and matrix independent High sensitivity and good tolerance for drug Fairly good ability of detecting low affinity antibodies ECL Signal ECL-Tag Drug Anti-Drug Antibody Drug Biotin MSD Streptavidin-Coated Plate

Cut Point Design At least 50 samples be tested on 3 different days by at least two analyst (FDA, 2016). Balanced design is usually adopted to minimize the impact of possible confounding factors (see, Shankar, et al. 2008; Yang, et al. (2016))

Screening Assay Cut Point Is the level of assay signal at or above which a sample is defined to be potentially positive (reactive) and below which it is defined to be negative Is determined statistically from drug-naïve samples FDA (2009 guidance) “recommends that the cut point have an upper negative limit of approximately 95 percent” to yield 5% false positive rate on average

Screening Assay Cut Point(Cont’d) In 2016 guidance, FDA “recommends that the cut point for screening assays be determined by a 90% one-sided lower confidence interval for the 95th percentile of the negative control population”. Under the new guidance, the lower 90% confidence limit of 95% quantile is nothing but the lower 5%-content, 90%-confidence tolerance limit. So that with 90% confidence, 5% false positive rate is guaranteed.

Screening Assay Cut Point(Cont’d) The idea is initially proposed by Shen, et al. (2015) with a motivation to reserve a certain percentage of false positives and minimize the false negative rate There is a potential impact that the proposed cut point is too low such that the low-signal positives can affect the relationship between ADA positivity and other variables such as PK, efficacy, etc. It is therefore critical to use the appropriate statistical method for cut point determination under the new FDA guidance and evaluate its impact.

Statistical Models In a typical cut point evaluation, the vast majority of variability comes from samples(subjects), not from the analysts, or plates. A one-component random effect model is the simplest model to capture the data structure of the cut point design. Traditional random effect model often assumes that the random effect (and error term) has normal distribution. The patient population from which the samples are obtained, however, are heterogeneous and typically has non-normal distribution, especially for screening assays.

Statistical Models (cont’d) Consider the balanced one-way random effect model, where it is assumed that there are I subjects with each subjects being tested J times, possibly by different analysts on different days. Let   𝑌 𝑖𝑗 =𝜇+ 𝛼 𝑖 + 𝜀 𝑖𝑗 , 𝑖=1,…, 𝐼;𝑗=1,…, 𝐽 (1) where 𝑌 𝑖𝑗 𝑠 are the (normalized) sample responses, which are identically distributed with a continuous cumulative distribution function 𝐹(𝑥).

Statistical Models (Cont’d) When the random effect model and error term are independently normally distributed, this is the conventional one-way random effect model. Methods for finding tolerance interval under normally distributed one-way random effect model are well-studied in the literature, see the book by Krishnamoorthy and Mathew (2004).

Non-normal Parametric Models When the random effect is non-normal, Zhang, et al. (2015) proposed some non-normal random effect models for cut point determination. These models include a skew-t random effect and a log-gamma random effect model Tolerance interval for these models have not been developed but is doable However, in practice, it is often difficult to verify whether the assay validation data does follow a specified parametric model.

Nonparametric Tolerance Limit Olsson and Rootzen (1996) proposed quantile estimators and their asymptotic properties for nonparametric one-way random effect model These results can be used to construct confidence limit for quantile estimator and be directly applied to cut point determination In this presentation, we considered the balanced one-way random effect model Unbalanced one-way random effect model can be developed similarly, see, Olsson and Rootzen (1996).

Assumption and Notation Denote 𝐼 𝑖𝑗 𝑥 =1( 𝑌 𝑖𝑗 ≤𝑥), where 1(∙) is the indicator function; 𝜌 𝑥 be the correlation function between any two indicators 𝐼 𝑖𝑗 𝑥 and 𝐼 𝑖𝑘 𝑥 , 𝑗≠𝑘; 𝜎 2 𝑥 =𝐹 𝑥 1−𝐹(𝑥) be the common variance function. For balanced design, the cumulative distribution function 𝐹(𝑥) can be estimated as 𝐹 𝑥 = 1 𝐼𝐽 𝑖,𝑗 𝐼 𝑖𝑗 𝑥 . The variance function 𝜎 2 𝑥 can be estimated as 𝜎 2 𝑥 = 1 𝐼𝐽 𝑖,𝑗 𝐼 𝑖𝑗 𝑥 − 𝐹 𝑥 2 . And the correlation function 𝜌 𝑥 can be estimated as 𝜌 𝑥 = 1 𝐼 𝜎 2 (𝑥) 𝑖 1 𝐽(𝐽−1) 𝑗≠𝑘 𝐼 𝑖𝑗 𝑥 − 𝐹 𝑥 𝐼 𝑖𝑘 𝑥 − 𝐹 𝑥 .

Cut Point /Tolerance Limit Estimation Define the qth quantile by 𝜉 𝑞 = inf 𝑥:𝐹 𝑥 ≥𝑞 . The quantile can be estimated by 𝜉 𝑞 = inf 𝑥: 𝐹 𝑥 ≥𝑞 . 𝜉 𝑞 has asymptotic normal distribution as I goes to infinity, 𝜉 𝑞 − 𝜉 𝑞 𝑠 𝑛 ( 𝜉 𝑞 )𝑓( 𝜉 𝑞 ) 𝑑 𝑁(0,1) The screening cut point can be estimated as 𝜉 0.95 − 𝑧 0.9 𝑠 𝑛 𝑓 where 𝑠 𝑛 𝜉 𝑞 = 1 𝐼𝐽 1+ 𝐽−1 𝜌 𝜉 𝑞 𝑞 1−𝑞 is the variance of 𝐹 𝜉 𝑞 and 𝑓( 𝜉 𝑞 ) is the density function at 𝜉 𝑞 , 𝑠 𝑛 is the estimator of 𝑠 𝑛 𝜉 𝑞 , and 𝑓 is the estimator of 𝑓, using kernel smoothing method.

Cut Point Estimator/Tolerance Limit(Cont’d) An alternative estimator is 𝜉 (0.95− 𝑧 0.9 𝑠 𝑛 ) , which is shown to have coverage probability of 0.9 asymptotically. We denote 𝜉 0.95 − 𝑧 0.9 𝑠 𝑛 𝑓 as NonPar1 and 𝜉 (0.95− 𝑧 0.9 𝑠 𝑛 ) as NonPar2. The advantage of NonPar2 is the avoidance of estimation of 𝑓 . Estimation of 𝑓 at tail of distribution is imprecise.

Existing Methods for Comparison Shen et al. (2015) considered exact lower confidence limit of normal percentile for IID data: 𝜃 ≡ 𝑌 +𝐶 𝑧 𝑝 𝑆−𝑏𝑆 1+𝑛 𝑧 𝑝 2 𝐶 2 −1 / 𝑛 The method is derived from the pivotal quantity Thus, the resulting tolerance limit is exact in terms of coverage probability under IID normal.

Tolerance Limit for One-way Random Effect Model Hoffman and Berger (2011) considered to use confidence-interval type of cut point and proposed to use 𝑌 + 𝑡 𝛼,𝑑𝑓,𝛿 𝜎 𝑇 𝑁 𝑒 under one-way random effect model The coverage probability appears to be satisfactory under correct assumption There are actually more than a dozen of tolerance interval methods for normally-distributed one-way random effect model.

Nonparametric method for IID data Non-parametric tolerance limit can be used when the data are IID. Assume 𝑋 1 , 𝑋 2 ,⋯, 𝑋 𝑛 are IID random sample from a continuous distribution and 𝑋 (1) , 𝑋 (2) ,⋯, 𝑋 (𝑛) are the corresponding order statistics Suppose a random variable Z follows a Binomial distribution Z ∼ Bin(n, 1-p) and let k be the largest integer such that Pr(Z ≥ k) ≥ 1-α, then 𝑋 (𝑘) is the desired p-content, 1-α confidence lower tolerance limit. Young and Mathew (2014) proposed an improved version of nonparametric tolerance limit by using linear interpolation. We denote Young and Mathew’s method as Nonpar3.

Simulation 1: Coverage Probability under Normal Distribution

Simulation 2: Coverage Probability under Student’s t-Distribution

Simulation 3: Coverage Probability under Gamma Distribution

Discrete Nature of Coverage Probability Coverage Probabilities of Screening Cut Point for Different Number of Subjects under Student’s t-Distribution

Discrete Nature of Coverage Probability Coverage Probabilities of Confirmatory Cut Point for Different Number of Subjects under Student-t Distribution

Conclusion/Remarks The nonparametric method is showed to be more appropriate compared to other methods for one-way random effect model with non-normal distribution Is computationally simple compared with smoothing/bootstrapping method Nonpar 2 method is simpler and performs well for cut point validation data where biological variability is usually dominant. The impact of adopting confidence-interval type of cut point should be carefully assessed using appropriate statistical method

References FDA. (2009). Guidance for industry: Assay development for immunogenicity testing of therapeutic proteins. FDA. (2016). Draft Guidance for industry: Assay development for immunogenicity testing of therapeutic proteins. Hoffman, D., & Berger, M. (2011). Statistical considerations for calculation of immunogenicity screening assay cut points. Journal of immunological methods, 373(1), 200-208. Olsson, J., & Rootzén, H. (1996). Quantile estimation from repeated measurements. Journal of the American Statistical Association, 91(436), 1560-1565. Shankar, G., Devanarayan, V., Amaravadi, L., Barrett, Y.C., Bowsher, R., Finco-Kent, D., Fiscella, M., Gorovits, B., Kirschner, S., Moxness, M. and Parish, T. (2008). Recommendations for the validation of immunoassays used for detection of host antibodies against biotechnology products. Journal of pharmaceutical and biomedical analysis, 48(5), pp.1267-1281. Shen, M., Dong, X., & Tsong, Y. (2015). Statistical Evaluation of Several Methods for Cut-Point Determination of Immunogenicity Screening Assay. Journal of biopharmaceutical statistics, 25(2), 269-279. Yang, H., Zhang, J., Yu, B., & Zhao, W. (2015). Statistical Methods for Immunogenicity Assessment (Vol. 85). CRC Press. Zhang, J., Yu, B., Zhang, L., Roskos, L., Richman, L., & Yang, H. (2015). Non-normal random effects models for immunogenicity assay cut point determination. Journal of biopharmaceutical statistics, 25(2), 295-306. Young, D.S. and Mathew, T., 2014. Improved nonparametric tolerance intervals based on interpolated and extrapolated order statistics. Journal of Nonparametric Statistics, 26(3), pp.415-432.