Remaining Challenges in Assessing Non-Inferiority Steven Snapinn DIA Statistics Community Virtual Journal Club December 16, 2014 Based on Paper with Qi Jiang: TIRS 2014, Vol 48(I) 62-67

Outline Specification of the Null Hypothesis Assessment of Preservation of Effect

FDA Draft Non-Inferiority Guidance Document and Indirect Comparison to Placebo Line 457 – “A successful non-inferiority study shows rigorously that the test drug has an effect greater than zero if it excludes an NI margin of M 1, so long as M 1 is well chosen and represents an effect that the control drug actually would have had (versus a placebo, had there been a placebo group).”

FDA Draft Non-Inferiority Guidance Document and Indirect Comparison to Placebo (2) Line 69 – “What non-inferiority trials seek to show is that any difference between the two treatments is small enough to allow a conclusion that the new drug has at least some effect ….”

FDA Draft Non-Inferiority Guidance Document and Indirect Comparison to Placebo (3) Line 567 – “M 1 … addresses the question of whether the test drug has any effect.” Line 1476 – “M 1 is used to determine whether the NI study shows that the test drug has any effect at all.”

FDA Draft Non-Inferiority Guidance Document and Indirect Comparison to Placebo (4) Line 138 – “… a finding of non-inferiority means that the test drug has an effect greater than 0.” Line 258 – “Because the consequence of choosing a margin greater than the actual treatment effect of the active control in the study is the false conclusion that a new drug is effective (a very bad public health outcome)…” Line 270 – “The 95% CI upper bound for C-T is used to provide a reasonably high level of assurance that the test drug does, in fact, have an effect greater than zero (i.e., that it has not lost all of the effect of the active control).” Line 1238 – “A successful NI conclusion, ruling out a difference > M 1, shows that the test drug is effective (just as a superiority study showing a significant effect at p ≤ 0.05 does) ….

What Is the Corresponding Hypothesis? Should It Be This (From FDA Draft Guidance)? – “the null hypothesis is that the degree of inferiority of the new drug (T) to the control (C), C-T, is greater than the non-inferiority margin M 1, where M 1 represents what is thought to be the whole effect of the active control (C) relative to placebo in the NI study. Ho: C – T ≥ M 1 (T is inferior to the control by M 1 or more) Ha: C – T < M 1 (T is inferior to the control by less than M 1 )” Or This? – H 0 : T ≤ 0 vs H a : T > 0

Alternate Hypothesis Definitions Based on the Margin H(M) 0 : μ T – μ C ≤ -δ H(M) A : μ T – μ C > -δ Based on an Indirect Comparison H(I) 0 : μ T – μ P ≤ 0 H(I) A : μ T – μ P > 0

Comparison of Hypotheses H(M) 0 and H(I) 0 Are Identical When δ = μ C – μ P Tests of H(M) 0 and H(I) 0 Are Generally Consistent, But Not Under Two Circumstances 1.When Margin Is Too Small (δ < μ C – μ P ) And The Treatment Is Superior To Placebo By A Small Amount (0 < (μ T – μ P ) < δ) 2.When Margin Is Too Large (δ > μ C – μ P ) And The Treatment Is Inferior To Placebo By A Small Amount (0 > (μ T – μ P ) > δ)

Regions of Inconsistency

Different Conclusions for Different Hypotheses Case 1: Concluding Non-Inferiority Is a Type 1 Error Using H(M) But Not Using H(I) – H(M) Seems Inconsistent With FDA Guidance, Since Test Drug Has Some Effect – “M 1 … addresses the question of whether the test drug has any effect.” Case 2: Concluding Non-Inferiority Is a Type 1 Error Using H(I) But Not Using H(M) – H(M) Seems Particularly Problematic, Since Test Drug Is Worse Than Placebo

Analysis Methods for the Alternate Hypotheses Fixed-Margin Method – Estimate μ T – μ C and Compare Lower End of CI to δ – Natural Approach for Testing H(M) and Can Be Used for Testing H(I) Synthesis Method – Combine (Synthesize) Information from NI Trial and Historical Trial to Estimate μ T – μ P – Natural Approach for Testing H(I), But Cannot Be Used for Testing H(M) Hypothesis Should Determine the Analysis Method, Not Vice Versa

Summary Non-Inferiority Hypothesis Can Be Specified Using a Margin (H(M)) or Based on an Indirect Comparison to Placebo (H(I)) Conclusions Using the Two Hypotheses Are Generally Consistent, But When They Differ H(M) Seems Inappropriate: – Concluding Non-Inferiority Is a Type 1 Error When Test Drug Has Some Effect – Concluding Non-Inferiority Is Not a Type 1 Error When Test Drug Worse Than Placebo

Assessment of Preservation of Effect Simply Demonstrating Efficacy May Not Be Sufficient for Regulatory Approval; May Be Necessary to Demonstrate “Preservation of Effect” Not to Be Confused With “Discounting” Define f (0 ≤ f ≤ 1) as “Preservation Factor” Can Be Applied to Either Hypothesis – H(M) 0 : μ T – μ C ≤ -(1 – f) * δ – H(I) 0 : μ T – μ P ≤ f * (μ C – μ P )

Preservation of Effect vs Preference Criteria Suppose Treatment C Has Been Approved Based on a Placebo-Controlled Trial, And Treatment T Has Been Evaluated Based on a Non-Inferiority Trial vs Treatment C Questions: – What Would Be the Criterion a Reasonable Person Would Use to Prefer T to C? – Is This Consistent With the Criterion for Concluding Non-Inferiority Using Preservation of Effect?

Example Historical Trial: C Significantly Superior to P – Estimate = 5; SD = 2 Non-Inferiority Trial: T Nominally Superior to C – Estimate = 3, SD = 2 Without Considering Preservation of Effect, T Meets Non-Inferiority Criterion – Fixed-Margin Method Let δ = 5 – 1.96 * 2 = 1.08 LB of 95% CI = 3 – 2 * 1.96 = > -δ = – Also Holds for Synthesis Method

Example (continued) With Considering Preservation of Effect, T May Not Meet Non-Inferiority Criterion – Fixed-Margin Method LB of 95% CI = > -(1 – f) * 1.08 Only When f < 0.15 – Synthesis Method Result Also Depends on f Effect of T Relative to P Estimated Through Synthesis – Estimate = = 8 – SD = sqrt( ) = 2.8

T Appears Preferable to C

Summary When Applying Preservation of Effect, Criterion for Concluding Non-Inferiority May Be Inconsistent With Reasonable Criterion for Preferring New Treatment to Control Example – T Was Not Non-Inferior to C for f > 0.15 – T Was Preferable to C Based On Demonstrated Efficacy (Through NI Analysis) Superior Point Estimate Superior LB of 95% CI Does Current Preservation of Effect Approach Lead to Logical Non-Inferiority Conclusions?