Adaptive non-inferiority margins under observable non-constancy Brett Hanscom, PhD HIV Prevention Trials Network Seattle, WA 5/9/2017

Introduction Non-inferiority design Determine whether an experimental product is not meaningfully worse than an active-control therapy. Define the “NI Margin” Must first know how effective the active-control therapy is Meta-analysis, meta-regression As a refresher for those who aren’t familiar with non-inferiority trials, a non-inferiority trial is a controlled trial with an active-control comparison group instead of a placebo control. The active control therapy is often the current standard of care. A non-inferiority trial is designed to determine whether an experimental product is not meaningfully worse than this current standard. Last year at this meeting I talked about the design of HPTN 083, a randomized controlled trial of long-acting injectable cabotegravir used for PrEP, with oral TDF/FTC as an active control. HPTN 083 isa non-inferiority trial, which means that the study is designed to determine whether Cabotegravir not meaningfully worse than TDF/FTC.

Example Trial HPTN 083 Randomized trial of injectable Cabotegravir as long-acting PrEP Enrolling 4,500 participants in the US, S America, Asia, and S Africa Active-control: Oral TDF/FTC (Truvada) Shown to be effective in multiple RCTs The numerical threshold beyond which a new product could be considered acceptable is called the non-inferiority margin This margin is generally calculated based on the results of existing placebo-controlled trials. As I mentioned in my talk last year, strong predictors of active-control effectiveness can make the margin more precise.

META-REGRESSION RESULTS – Prior Trials - One example is the case where adherence to daily oral PrEP has been shown to be a strong predictor of effectiveness. - In this plot adherence is on the x-axis and relative risk is on the y-axis, with results for men shown in blue and results for women shown in red. -The size of the circles is proportional to the size of the trial,. - The solid and dashed blue lines show the results of fitting a meta-regression model to the collected trial results among MEN, and the corresponding 95% confidence bands. -The model results can be used directly to compute an NI margin for a planned trial

Planned adherence For example, if we anticipate adherence at around 60%, we would identify the lower bound of the relative estimate and use that as a conservative estimate for TDF/FTC effectiveness.

Estimated effectiveness 1.5 For example, if we anticipate adherence at around 60%, we would identify the lower bound of the relative estimate and use that as a conservative estimate for TDF/FTC effectiveness.

PLANNED Non-Inferiority Margin – 60% Adherence Planned Margin Active-Control Benefit (Prior Trials) Experimental vs. Active-Control (Planned Trial) For 60% adherence that lower bound is a relative risk of 1.5. Relative risk is now shown here on the x-axis. To preserve 50% of this benefit on the log scale, we compute the square root of 1.5, which gives a NI margin of 1.23 We want to be sure that the Cab arm does not have more that 23% more infections than the TDF/FTC arm. This is the current planned margin for HPTN 083. If the relative risk comparing the experimental therapy to the active control, and it’s 95% confidence interval (shown here as the grey circle with error bars) fall entirely to the left of the margin, then we conclude that the experimental treatment is non inferior, and we have “Positive Trial” 1.0 1.0 1.23 1.50 Relative Risk relative to the Active Control (log scale) Relative Risk relative to the Active Control (log scale)

Non-constancy What happens if adherence is not as planned? TDF/FTC adherence is notoriously variable. Effectiveness of TDF/FTC will not be as planned either, and the selected NI margin will be invalid. Type-I error and power will not be as expected -One of the primary assumptions in a non-inferiority trial is that the active control will have the same effect in the new study population is it had in the prior trials from which the margin was derived. -One important way that the constancy assumption could be violated is if adherence were different than in prior trials, and prior experience with PrEP trials tells that this can easily happen.

Type-I error and power, wrong margin Observed Adherence Estimated TDF/FTC Benefit NI Margin Preserving 50% Benefit Type-I Error Power** 0.50 1.17 1.08 0.13 0.98 0.55 1.33 1.15 0.06 0.95 0.60* 1.5 1.23 0.025 0.90 0.65 1.69 1.30 0.01 0.82 0.70 1.89 1.37 0.004 0.71 Type-I error and power for various scenarios are listed here in this table. The middle row shows the planned level of adherence, 60%, and here the corresponding type-I error is 2.5% with power equal to 90%, as planned. Power is computed in relation to the alternative hypothesis that the experimental versus control relative risk is 0.75. In row 1 we have the case where adherence is 50%, giving a margin of 1.08, and type-I error elevated to 13%. If adherence is high, say 70% as shown in the last row, power is diminished from 90% down to 71%. * Planned level of adherence ** Assuming RR = 0.75

Proposal – Adaptive Margins Measure drug adherence in the active-control arm during/after the trial Re-compute the NI Margin based on the observed population values Apply this adapted margin to the final endpoint results

Adapting the Null Hypothesis The adapted margin becomes the adapted null hypothesis In a superiority H0: RR = 1.0 In a NI trial H0: RR = NI Margin In an adaptive margin NI trial H0: RR = “Preserve 50% of Benefit” Ruling out 1.0 implies that the experimental treatment is superior to control, as compared to being equally effective to the control. Ruling out the NI margin means establishing that the new treatment is not worse than the control by the amount of the specific margin And in an adaptive-margin trial, ruling out the margin means establishing that at least 50% of the active-control benefit is preserved. This specific value of the margin is variable, but the concept remains the same.

Corrected Type-I Error Observed Adherence Estimated TDF/FTC Benefit Adapted NI Margin Type-I Error 0.50 1.17 1.08 0.025 0.55 1.33 1.15 0.60* 1.50 1.23 0.65 1.69 1.30 0.70 1.89 1.37 When we adjust the margin to the level appropriate to the observed level of adherence, we correct the type-I error to the planned level 2.5%. This results from the fact the new margin is now our null hypothesis – and under the null a 95% confidence interval will rule out the null on the low end 2.5% of the time. * Planned level of adherence

Alternative Hypothesis UNDER A FIXED SAMPLE SIZE (n=172) Planned Alt Planned Margin Effect Size This trial was powered to be able to detect a relative risk equal to 0.75 (or stronger), and to achieve that power 172 events must be observed. Because sample size determines the effect size, once the desired number of events have been observed, and the margin has been appropriately adjusted, we can compute the alternative we are now fully powered to detect. 0.75 1.0 1.0 1.23 Relative Risk relative to the Active Control (log scale) Relative Risk relative to the Active Control (log scale)

Alternative Hypothesis UNDER A FIXED SAMPLE SIZE (n=172) Adapted Margin Adapted Alt Effect Size In this case, if the margin is shifted down to 1.15, the alternative shifts down to 0.7, which maintains a constant ration between the margin and the alternative. 0.7 0.75 1.0 1.0 1.15 1.23 Relative Risk relative to the Active Control (log scale) Relative Risk relative to the Active Control (log scale)

New Alternative Hypothesis Observed Adherence Estimated TDF/FTC Benefit Adapted NI Margin Type-I Error Alternative with 90% Power 0.50 1.17 1.08 0.025 0.66 0.55 1.33 1.15 0.70 0.60* 1.50 1.23 0.75 0.65 1.69 1.30 0.79 1.89 1.37 0.83 Here are a few more examples. With 50% adherence, we have 90% power to detect a true experimental vs control relative risk of 0.66. This makes intuitive sense in that if the adherence is low and the active-control does not work well, we would want the experimental agent to perform better than planned in relation to the control. If adherence is 70% * Planned level

Pre-specification The meta-regression model used for adapting the margin is based entirely on external trials The adapted NI margin depends only on adherence observed in the active control arm, and not on the observed effect size Procedures for measuring adherence, should be carefully pre-specified.

Cautions The meta-regression model is not perfect Assessment of adherence is not perfect, and may not be identical to the way adherence was measured previously These methods are in development and not yet approved by the FDA for HPTN 083

Summary Non-inferiority trials are increasingly common, and we expect to see this trend in HIV prevention Adaptive NI margins provide useful tool when active-control effectiveness is not as planned and the constancy assumption fails

Deborah Donnell, Jim Hughes Collaborators: Deborah Donnell, Jim Hughes HPTN SDMC / SCHARP Brian Williamson University of Washington

OBSERVE 55% Adherence – MARGIN IS TOO HIGH – INFLATED TYPE-I ERROR Correct Margin Estimated Active-Control Benefit Experimental vs. Active-Control (Planned Trial) The following schematics are designed to give an intuitive sense for what can happen when adherence is not as planned and the NI margin is not correct. This slide shows the scenario where adherence is observed to be 55%, or 5% less than planned. In this case the estimated active-control benefit is 1.33 and the optimal margin would be 1.15, which is lower than the pre-specified value 1.23 (represented by the solid red line). If the final results of the trial were as shown by point estimate and confidence interval shown here, using the planned margin we would conclude that the experimental intervention was non inferior, when in fact these results do not exclude the correct margin and therefor do not clearly preserve 50% of the active control benefit. In other words, it is too easy to declare non-inferiority, and hence we have inflated type-I error. 1.0 1.0 1.15 1.33 Relative Risk relative to the Active Control (log scale) Relative Risk relative to the Active Control (log scale)

OBSERVE 70% Adherence – MARGIN IS TOO HIGH – LOW POWER Correct Margin Observed Active-Control Benefit Experimental vs. Active-Control (Planned Trial) This slide shows the scenario where adherence is observed to be 70%, or 10% higher than planned. In this case the estimated active-control benefit is 1.89 and the optimal margin would be 1.37, substantially higher than the pre-specified value . If the final results of the trial were as shown by point estimate and confidence interval shown here, using the planned margin we would not conclude that the experimental intervention was non inferior, when in fact these results exclude the correct margin and therefor do clearly preserve 50% of the active control benefit. In other words, it is too difficult to declare non-inferiority, and hence we have loss of power. 1.0 1.0 1.37 1.89 Relative Risk relative to the Active Control (log scale) Relative Risk relative to the Active Control (log scale)