Issues of Simultaneous Tests for Non-Inferiority and Superiority Tie-Hua Ng*, Ph. D. U.S. Food and Drug Administration Presented at MCP 2002 August 5-7, 2002 Bethesda, Maryland _______ * The views expressed in this presentation are not necessarily of the U.S. Food and Drug Administration.

2 Simultaneous Tests for Non-Inferiority and Superiority Multiplicity adjustment is not necessary –Intersection-union principle (IU) Dunnett and Gent (1996) –Closed testing procedure (CTP) Morikawa and Yoshida (1995) Indisputable

3 A Big Question Is Multiplicity Adjustment Necessary?

4 IsMultiplicityAdjustmentNecessary?

5 Outline Assumptions and Notations Switching between Superiority and Non-Inferiority Is Simultaneous Testing Acceptable? Use of Confidence Interval in Hypothesis Testing --- Pitfall Problems of Simultaneous Testing Conclusion

6 Assumptions/Notations Normality and larger is better T: Test/Experimental treatment (  t ) S: Standard therapy/Active control (  s )  : Non-Inferiority Margin (> 0) For a given d (real number), define –Null: H 0 (d): T  S - d –Alternative: H 1 (d): T > S - d Non-Inferiority: d =  Superiority: d = 0

7 Non-Inferiority (d =  ) H 0 (  ): T  S -  against H 1 (  ): T > S -  H0()H0() H1()H1() ° T Boundary WorseBetter Mean Response S 

8 Superiority (d = 0) H 0 (0): T  S against H 1 (0): T > S H 0 (0) H 1 (0) ° T Boundary WorseBetter Mean Response S

9 Switching between Superiority and Non-Inferiority CPMP (Committee for Proprietary Medicinal Products), European Agency for the Evaluation of Medicinal Products Points to Consider on Switching Between Superiority and Non-Inferiority,

10 Switching between Superiority and Non-Inferiority (2) Non-Inferiority Trial –If H 0 (  ) is rejected, proceed to test H 0 (0) –No multiplicity issue, closed testing procedure Superiority Trial –Fail to reject H 0 (0), proceed to test H 0 (  ) –No multiplicity issue –Post hoc specification of 

11 Switching between Superiority and Non-Inferiority (3) Non-inferiority Trial –Intention-to-treat (ITT) –Per protocol (PP) Superiority Trial –Primary: Intention-to-treat (ITT) –Supportive: Per protocol (PP) Assume ITT = PP

12 Simultaneous Testing One-sided 100(1 -  )% lower Confidence Interval for T - S Test is worse Test is better Mean Difference (T – S) 0 -- Superiority Non-inferiority Neither

13 Simultaneous Testing (2) Multiplicity adjustment is not necessary –Dunnett and Gent (1996) Intersection-Union (IU): Superiority: Both H 0 (  ) and H 0 (0) are rejected –Morikawa and Yoshida (1995) Closed Testing Procedure (CTP): Test H 0 (0) when H 0 (  )  H 0 (0) is rejected

14 Simultaneous Testing (3) Discussion Forum (October 1998) –London –PSI (Statisticians in Pharmaceutical Industry) Is Simultaneous Testing of Equivalence [Non- Inferiority] and Superiority Acceptable? –Superiority trial: Fail to reject H 0 (0) No equivalence/non-inferiority claim –Ok: Morikawa and Yoshida (1995) Ref: Phillips et al (2000), DIJ

15 Is Simultaneous Testing Acceptable?

16 Use of Confidence Interval in Hypothesis Testing H 0 (d): T  S - d (at significance level  ) One-sided 100(1-  )% lower CI for T-S Reject H 0 (d) if and only if the CI excludes -d Test is worseTest is better Mean Difference (T – S) -d Reject H 0 (d) Do not reject H 0 (d)

17 Use of Confidence Interval in Hypothesis Testing (2) If CI = (L,  ), then H 0 (d) will be rejected for all -d < L. A Tricky Question –Suppose CI = (-1.999,  ), L = H 0 (2): T  S - 2 is rejected (d=2) since -d < L Can we conclude that T > S - 2? Yes, if H 0 (2) is prespecified. No, otherwise.

18 Use of Confidence Interval in Hypothesis Testing (3) Post hoc specification of H 0 (d) is a No

19 Simultaneous Testing: Problems H 0 (d 1 ) and H 0 (d 2 ), for d 1 > d 2 One-sided (1 -  )100% lower CI for T - S Test is worse Test is better Mean Difference (T – S) -d 2 -d 1 Reject H 0 (d 2 ) Reject H 0 (d 1 ) Neither

20 Simultaneous Testing: Problems (2) H 0 (d 1 ), H 0 (d 2 ) and H 0 (d 3 ), for d 1 > d 2 > d 3 One-sided (1 -  )100% lower CI for T - S Test is worse Test is better Mean Difference (T – S) -d 3 -d 1 Reject H 0 (d 3 ) Reject H 0 (d 2 ) None -d 2 Reject H 0 (d 1 )

21 Simultaneous Testing: Problems (3) H 0 (d 1 ), H 0 (d 2 ),…, H 0 (d k ), for d 1 > d 2 > … > d k One-sided (1 -  )100% lower CI for T - S Test is worse Test is better Mean Difference (T – S) -d k -d 1 Reject H 0 (d k ) Reject H 0 (d 2 ) None -d 2 Reject H 0 (d 1 ) … … d 3

22 Simultaneous Testing: Problems (4) Choose k large enough  Pr[-d 1 < Lower limit < -d k ] close to 1  Max |d k - d k-1 | < a given small number  Simultaneous testing of H 0 (d i ), i = 1,…, k  Post hoc specification of H 0 (d)

23 Simultaneous Testing: Problems (5) Number of Nested hypotheses Exploratory (many H 0 (d)) Confirmatory (one H 0 (d)) …………. k ………… Simultaneous H 0 (  ) and H 0 (0)

24 Simultaneous Testing: Problems (6) What is wrong with IU and CTP? Nothing Pr[Rejecting at least one true null]   What kind of problems?

25 Simultaneous Testing: Problems (7) Post hoc specification of H 0 (d)  Let -d 0 = 100(1 -  )% lower limit -   Reject H 0 (d 0 ), since -d 0 < lower limit  Repeat the same trial independently  Pr[Rejecting H 0 (d 0 )] = 0.5 +

26 Simultaneous Testing: Problems (8) Simultaneous testing of many H 0 (d) –Repeat the same trial independently –Low probability of confirming the finding 1 st trial: Reject H 0 (d j ) but not H 0 (d j+1 ) 2 nd trial: Pr[Rejecting H 0 (d j )] is low (e.g., 0.5+)

27 Simultaneous Testing: Problems (9) Simultaneous testing of H 0 (  ) and H 0 (0)? Confirm the finding   = 2  Known variance  Let   T - S  Significance level  =  80% power for H 0 (  ) (at  = 0)

28 Simultaneous Testing: Problems (10) f  (  ) = Pr[Rejecting H 0 (  ) |  ] f 0 (  ) = Pr[Rejecting H 0 (0) |  ]

29 Simultaneous Testing: Problems (11) Test one null hypothesis H 0 (  ) Suppose that H 0 (  ) is rejected Repeat the same trial independently Pr[Rejecting H 0 (  ) again] = f  (  )

30 Simultaneous Testing: Problems (12) Test H 0 (  ) and H 0 (0) simultaneously Suppose that H 0 (  ) or H 0 (0) is rejected Repeat the same trial independently Pr[Rejecting the same null hypothesis again] = [1 - w(  )] · f  (  ) + w(  ) · f 0 (  ) = f  (  ) - f 0 (  ) [1 – f 0 (  )/f  (  )], where w(  ) = f 0 (  )/f  (  )

31 Simultaneous Testing: Problems (13) [1 - w(  )] · f  (  ) + w(  ) · f 0 (  ) where w(  ) = f 0 (  )/f  (  ) Simultaneous tests in the 2 nd trial

32 Simultaneous Testing: Problems (14) Ratio: 1 – [f 0 (  )/f  (  )] [1 – f 0 (  )/f  (  )] Ratio may be as low as 0.75

33 Conclusion Many H 0 (d): Problematic Not type I error rate H 0 (  ) and H 0 (0): Acceptable? If “zero tolerance policy”: No If 25% reduction cannot be tolerated: No If 25% reduction can be tolerated: Yes

34 Is Simultaneous Testing of H 0 (  ) and H 0 (0) Acceptable?

35 You be the judge

36 References Dunnett and Gent (1976), Statistics in Medicine, 15, Committee for Proprietary Medicinal Products (CPMP; 2002). Points to Consider on Switching Between Superiority and Non-Inferiority. Morikawa T, Yoshida M. (1995), Journal of Biopharmaceutical Statistics, 5: Phillips et al., (2000), Drug Information Journal, 34:

37