A Bayesian Approach to Parallelism Testing in Bioassay Steven Novick, GlaxoSmithKline Harry Yang, MedImmune LLC

John Peterson, Director of statistics, GlaxoSmithKline Manuscript co-author John Peterson, Director of statistics, GlaxoSmithKline

Warm up exercise

When are two lines parallel? Parallel: Being everywhere equidistant and not intersecting Slope Horizontal shift places one line on top of the other.

When are two curves parallel? Parallel: Being everywhere equidistant and not intersecting? Can you tell by checking model parameters? Horizontal shift places one curve on top of the other.

Where is parallelism important? Gottschalk and Dunn (2005) Determine if biological response(s) to two substances are similar Determine if two different biological environments will give similar dose–response curves to the same substance. Compound screening Assay development / optimization Bioassay standard curve

Screening for compound similar to “gold-standard” E.g., seeking new HIV compound with AZT-like efficacy, but different viral-mutation profile. Desire for dose-response curves to be parallel. Parallel indicates a change in potency only.

Change to assay procedure E.g., change from fresh to frozen cells. Want to provide same assay signal window. Desire for control curves to be parallel. Fresh Frozen Day 0 Day 5 Day 10 CD8 T cells, HIV ELISPOT assay

Assess validity of bioassay used for relative potency Dilution must be parallel to original. Callahan and Sajjadi (2003) Original Dilution

Replacing biological materials used in standard curve E.g., ELISA (enzyme-linked immunosorbent) assay to measure protein expression. Recombinant proteins used to make standard curve. Testing clinical sample New lot of recombinant proteins for standards. Check curves are parallel Calibrate new curve to match the old curve.

Potency often determined relative to a reference standard such as ratio of EC50 Only meaningful if test sample behaves as a dilution or concentration of reference standard Testing parallelism is required by revised USP Chapter <111> and European Pharmacopeia

Linear: Two lots of Protein “A” Estimated Concentrations Lot 2 is 1.4-fold higher than Lot 1 Log10 Signal =0.14

If the lines are parallel… Shift “Lot 2” line to the left by a calibration constant .  is log relative potency of Lot 2. USP = United States Pharmacopeia USP sets standards for the quality, purity, strength, and consistency of products–critical to the public health. Draft USP <1034> 2010

Testing for Parallelism in bioassay

Typical experimental design Serial dilutions of each lot Several replicates Fit on single plate (no plate effect) Log10 Signal

Tests for parallel curves Linear model Hauck et al, 2005; Gottschalk and Dunn, 2005 H0: b1 = b2 H1: b1 ≠ b2 ANOVA: T-test F goodness of fit test 2 goodness of fit test May lack power with small sample size Might be too powerful for large sample size

A better idea Callahan and Sajjadi 2003; Hauck et al. 2005 Slopes are equivalent H0: | b1 − b2 |≥  H1: | b1 − b2 |< 

Nonlinear: Two lots of protein “B” Estimated Concentrations Lot 2 is 1.6-fold higher than Lot 1 Depending on the response to back-calculate, the fold-change between lots 1 and 2 can range from as little as 1.5-fold to as much as 5.5-fold. =0.21

Tests for parallel curves 4-parameter logistic (FPL) model Jonkman and Sidik 2009 F-test goodness of fit statistic H0: A1 = A2 and B1 = B2 and D1 = D2 H1: At least one parameter not equal May lack power with small sample size Might be too powerful for large sample size

Calahan and Sajjadi 2003; Hauck et al. 2005; Jonkman and Sidik (2009) Equivalence test for each parameter = intersection-union test H0: |1 / 2|  1 or |1 / 2|  2 H1: 1 <|1 / 2| < 2 i = Ai, Bi, Di , i=1,2 Equiv. of params does not provide assurance of parallelism (except for linear) May lack power with small sample size Forces a hyper-rectangular acceptance region

“Parallel Equivalence” Our proposal “Parallel Equivalence”

Definition of Parallel Two curves and are parallel if there exists a real number ρ such that for all x.

Definition of Parallel Equivalence Two curves and are parallel equivalent if there exists a real number ρ such that for all x  [xL, xU].

It follows that two curves are parallel equivalent if there exists a real number ρ such that It also follows that

Are these two lines parallel enough when xL < x < xU ? < ?

Linear-model solution Just check the endpoints

Parallel equivalence = slope equivalence wlog Same as testing: | b1 − b2 |< 

Parallel Equivalence FPL model: No closed-form solution. Simple two-dimensional minimax procedure.

Are these two curves parallel enough when xL < x < xU ? < ?

Testing for parallel equivalence H0: H1: Proposed metric (Bayesian posterior probability):

Computing the Bayesian posterior probability For each curve, assume Data distribution: i =1, 2 = reference or sample j = 1, 2, …, N = observations Prior distribution: Posterior distribution proportional to:

Draw a random sample of the i of size K from the posterior distribution (e.g., using WinBugs). The posterior probability is estimated by the proportion (out of K) that the posterior distribution of:

= 0.07  90% probability to call parallel equivalent If we examine the difference in estimated concentration for lot 2 from the x-range, we get a delta.x = 0.07 -> 1.17-fold maximum shift in estimated conc. Frequentist model point-estimate maximum shift = 1.07-fold. = 0.14 (100.14=1.4-fold shift) = 0.07  90% probability to call parallel equivalent

= 0.52  90% probability to call parallel equivalent If we examine the difference in estimated concentration for lot 2 from the x-range, we get a delta.x = 0.35 -> 2.15-fold maximum shift in estimated conc. Frequentist model point-estimate maximum shift = 1.67-fold. = 0.33 (100.33=2.15-fold shift) = 0.52  90% probability to call parallel equivalent

Simulation: FPL Model Based on protein “B” data

Simulation: FPL model Similar to Protein “B” protein-chip data. Concentrations (9-point curve + 0): 0, 102(=100), 102.5625, 103.125, …, 106.5(=3,200,000) Three replicates xL = 3.5=log10(3162) & xU = 5=log10(100,000) δ = 0.2.  = 0.02, 0.04, 0.11, and 0.21 (%CV = 5, 10, 25, 50) For each Monte Carlo run, I computed:

Scenario Original New Max Diff A1 B1 C1 D1 A2 B2 C2 D2 1 2 4.5 4 -1 4.7 3 2.445 4.945 0.15 -1.312 5 2.61 5.11 =0.20 6 2.25 4.75 -1.5 0.30 5,000 Monte Carlo Replicates

Example data (CV=10%) Diff: 0 0 0.15 0.15 0.20 0.30

Diff: 0 0 0.15 0.15 0.20 0.30

Frequentist statistical power Scenario Frequentist statistical power Max Diff %CV=5 %CV=10 %CV=25 %CV=50 1 1.00 0.50 2 3 0.15 (shape 1) 0.99 0.28 < 0.01 4 0.15 (shape 2) 0.88 0.36 0.14 5 δ = 0.20 0.10 0.02 6 0.30 0.00

Summary Straight-forward and simple test method to assess parallelism. Yields the log-relative potency factor. Easily extended.

Extensions Instead of f(, x), could use f(, x) /  x = instantaneous slope f-1(, y) = estimated concentrations

What’s next? Head-to-head comparison with existing methods Choosing test level and , possibly based on ROC curve? – Harry Yang paper Guidance for prior distribution of 1 and 2.

References Callahan, J. D. and Sajjadi, N. C. (2003), “Testing the Null Hypothesis for a Specified Difference - The Right Way to Test for Parallelism”, Bioprocessing Journal Mar/Apr 1-6. Gottschalk P.J. and Dunn J.R. (2005), “Measuring Parallelism, Linearity, and Relative Potency in Bioassay and Immunoassay Data”, Journal of Biopharmaceutical Statistics, 15: 3, 437 - 463. Hauck W.W., Capen R.C., Callahan J.D., Muth J.E.D., Hsu H., Lansky D., Sajjadi N.C., Seaver S.S., Singer R.R. and Weisman D. (2005), “Assessing parallelism prior to determining relative potency”, Journal of pharmaceutical science and technology, 59, 127-137. Jonkman J and Sidik K (2009), “Equivalence Testing for Parallelism in the Four- Parameter Logistic Model”, Journal of Biopharmaceutical Statistics, 19: 5, 818 - 837.

Thank you! Questions?