1. Adaptive non-inferiority margins under observable non-constancy
Brett Hanscom, PhD
HIV Prevention Trials Network
Seattle, WA
5/9/2017

2. Introduction Non-inferiority design
Determine whether an experimental product is not meaningfully worse than an active-control therapy.
Define the “NI Margin”
Must first know how effective the active-control therapy is
Meta-analysis, meta-regression
As a refresher for those who aren’t familiar with non-inferiority trials, a non-inferiority trial is a controlled trial with an active-control comparison group instead of a placebo control. The active control therapy is often the current standard of care. A non-inferiority trial is designed to determine whether an experimental product is not meaningfully worse than this current standard.
Last year at this meeting I talked about the design of HPTN 083, a randomized controlled trial of long-acting injectable cabotegravir used for PrEP, with oral TDF/FTC as an active control.
HPTN 083 isa non-inferiority trial, which means that the study is designed to determine whether Cabotegravir not meaningfully worse than TDF/FTC.

3. Example Trial
HPTN 083
Randomized trial of injectable Cabotegravir as long-acting PrEP
Enrolling 4,500 participants in the US, S America, Asia, and S Africa
Active-control: Oral TDF/FTC (Truvada)
Shown to be effective in multiple RCTs
The numerical threshold beyond which a new product could be considered acceptable is called the non-inferiority margin
This margin is generally calculated based on the results of existing placebo-controlled trials.
As I mentioned in my talk last year, strong predictors of active-control effectiveness can make the margin more precise.

4. META-REGRESSION RESULTS – Prior Trials
- One example is the case where adherence to daily oral PrEP has been shown to be a strong predictor of effectiveness.
- In this plot adherence is on the x-axis and relative risk is on the y-axis, with results for men shown in blue and results for women shown in red.
-The size of the circles is proportional to the size of the trial,.
- The solid and dashed blue lines show the results of fitting a meta-regression model to the collected trial results among MEN, and the corresponding 95% confidence bands.
-The model results can be used directly to compute an NI margin for a planned trial

5. Planned adherence
For example, if we anticipate adherence at around 60%, we would identify the lower bound of the relative estimate and use that as a conservative estimate for TDF/FTC effectiveness.

6. Estimated effectiveness
1.5
For example, if we anticipate adherence at around 60%, we would identify the lower bound of the relative estimate and use that as a conservative estimate for TDF/FTC effectiveness.

7. PLANNED Non-Inferiority Margin – 60% Adherence
Planned Margin
Active-Control Benefit (Prior Trials)
Experimental vs.
Active-Control
(Planned Trial)
For 60% adherence that lower bound is a relative risk of Relative risk is now shown here on the x-axis.
To preserve 50% of this benefit on the log scale, we compute the square root of 1.5, which gives a NI margin of 1.23
We want to be sure that the Cab arm does not have more that 23% more infections than the TDF/FTC arm.
This is the current planned margin for HPTN If the relative risk comparing the experimental therapy to the active control, and it’s 95% confidence interval (shown here as the grey circle with error bars) fall entirely to the left of the margin, then we conclude that the experimental treatment is non inferior, and we have “Positive Trial”
1.0
1.0
1.23
1.50
Relative Risk relative to the Active Control (log scale)
Relative Risk relative to the Active Control (log scale)

8. Non-constancy
What happens if adherence is not as planned? TDF/FTC adherence is notoriously variable.
Effectiveness of TDF/FTC will not be as planned either, and the selected NI margin will be invalid.
Type-I error and power will not be as expected
-One of the primary assumptions in a non-inferiority trial is that the active control will have the same effect in the new study population is it had in the prior trials from which the margin was derived.
-One important way that the constancy assumption could be violated is if adherence were different than in prior trials, and prior experience with PrEP trials tells that this can easily happen.

9. Type-I error and power, wrong margin
Observed Adherence
Estimated TDF/FTC Benefit
NI Margin Preserving 50% Benefit
Type-I Error
Power**
0.50
1.17
1.08
0.13
0.98
0.55
1.33
1.15
0.06
0.95
0.60*
1.5
1.23
0.025
0.90
0.65
1.69
1.30
0.01
0.82
0.70
1.89
1.37
0.004
0.71
Type-I error and power for various scenarios are listed here in this table. The middle row shows the planned level of adherence, 60%, and here the corresponding type-I error is 2.5% with power equal to 90%, as planned. Power is computed in relation to the alternative hypothesis that the experimental versus control relative risk is In row 1 we have the case where adherence is 50%, giving a margin of 1.08, and type-I error elevated to 13%. If adherence is high, say 70% as shown in the last row, power is diminished from 90% down to 71%.
* Planned level of adherence
** Assuming RR = 0.75

10. Proposal – Adaptive Margins
Measure drug adherence in the active-control arm during/after the trial
Re-compute the NI Margin based on the observed population values
Apply this adapted margin to the final endpoint results

11. Adapting the Null Hypothesis
The adapted margin becomes the adapted null hypothesis
In a superiority
H0: RR = 1.0
In a NI trial
H0: RR = NI Margin
In an adaptive margin NI trial
H0: RR = “Preserve 50% of Benefit”
Ruling out 1.0 implies that the experimental treatment is superior to control, as compared to being equally effective to the control.
Ruling out the NI margin means establishing that the new treatment is not worse than the control by the amount of the specific margin
And in an adaptive-margin trial, ruling out the margin means establishing that at least 50% of the active-control benefit is preserved. This specific value of the margin is variable, but the concept remains the same.

12. Corrected Type-I Error
Observed Adherence
Estimated TDF/FTC Benefit
Adapted NI Margin
Type-I Error
0.50
1.17
1.08
0.025
0.55
1.33
1.15
0.60*
1.50
1.23
0.65
1.69
1.30
0.70
1.89
1.37
When we adjust the margin to the level appropriate to the observed level of adherence, we correct the type-I error to the planned level 2.5%. This results from the fact the new margin is now our null hypothesis – and under the null a 95% confidence interval will rule out the null on the low end 2.5% of the time.
* Planned level of adherence

13. Alternative Hypothesis UNDER A FIXED SAMPLE SIZE (n=172)
Planned Alt
Planned Margin
Effect Size
This trial was powered to be able to detect a relative risk equal to 0.75 (or stronger), and to achieve that power 172 events must be observed. Because sample size determines the effect size, once the desired number of events have been observed, and the margin has been appropriately adjusted, we can compute the alternative we are now fully powered to detect.
0.75
1.0
1.0
1.23
Relative Risk relative to the Active Control (log scale)
Relative Risk relative to the Active Control (log scale)

14. Alternative Hypothesis UNDER A FIXED SAMPLE SIZE (n=172)
Adapted Margin
Adapted Alt
Effect Size
In this case, if the margin is shifted down to 1.15, the alternative shifts down to 0.7, which maintains a constant ration between the margin and the alternative.
0.7
0.75
1.0
1.0
1.15
1.23
Relative Risk relative to the Active Control (log scale)
Relative Risk relative to the Active Control (log scale)

15. New Alternative Hypothesis
Observed Adherence
Estimated TDF/FTC Benefit
Adapted NI Margin
Type-I Error
Alternative with
90% Power
0.50
1.17
1.08
0.025
0.66
0.55
1.33
1.15
0.70
0.60*
1.50
1.23
0.75
0.65
1.69
1.30
0.79
1.89
1.37
0.83
Here are a few more examples. With 50% adherence, we have 90% power to detect a true experimental vs control relative risk of This makes intuitive sense in that if the adherence is low and the active-control does not work well, we would want the experimental agent to perform better than planned in relation to the control. If adherence is 70%
* Planned level

16. Pre-specification
The meta-regression model used for adapting the margin is based entirely on external trials
The adapted NI margin depends only on adherence observed in the active control arm, and not on the observed effect size
Procedures for measuring adherence, should be carefully pre-specified.

17. Cautions The meta-regression model is not perfect
Assessment of adherence is not perfect, and may not be identical to the way adherence was measured previously
These methods are in development and not yet approved by the FDA for HPTN 083

18. Summary
Non-inferiority trials are increasingly common, and we expect to see this trend in HIV prevention
Adaptive NI margins provide useful tool when active-control effectiveness is not as planned and the constancy assumption fails

19. Deborah Donnell, Jim Hughes
Collaborators:
Deborah Donnell, Jim Hughes
HPTN SDMC / SCHARP
Brian Williamson
University of Washington

20. OBSERVE 55% Adherence – MARGIN IS TOO HIGH – INFLATED TYPE-I ERROR
Correct Margin
Estimated Active-Control Benefit
Experimental vs.
Active-Control
(Planned Trial)
The following schematics are designed to give an intuitive sense for what can happen when adherence is not as planned and the NI margin is not correct.
This slide shows the scenario where adherence is observed to be 55%, or 5% less than planned. In this case the estimated active-control benefit is 1.33 and the optimal margin would be 1.15, which is lower than the pre-specified value 1.23 (represented by the solid red line). If the final results of the trial were as shown by point estimate and confidence interval shown here, using the planned margin we would conclude that the experimental intervention was non inferior, when in fact these results do not exclude the correct margin and therefor do not clearly preserve 50% of the active control benefit. In other words, it is too easy to declare non-inferiority, and hence we have inflated type-I error.
1.0
1.0
1.15
1.33
Relative Risk relative to the Active Control (log scale)
Relative Risk relative to the Active Control (log scale)

21. OBSERVE 70% Adherence – MARGIN IS TOO HIGH – LOW POWER
Correct Margin
Observed Active-Control Benefit
Experimental vs.
Active-Control
(Planned Trial)
This slide shows the scenario where adherence is observed to be 70%, or 10% higher than planned. In this case the estimated active-control benefit is 1.89 and the optimal margin would be 1.37, substantially higher than the pre-specified value . If the final results of the trial were as shown by point estimate and confidence interval shown here, using the planned margin we would not conclude that the experimental intervention was non inferior, when in fact these results exclude the correct margin and therefor do clearly preserve 50% of the active control benefit. In other words, it is too difficult to declare non-inferiority, and hence we have loss of power.
1.0
1.0
1.37
1.89
Relative Risk relative to the Active Control (log scale)
Relative Risk relative to the Active Control (log scale)