1. Proprietary and Confidential © AstraZeneca 2009 FOR INTERNAL USE ONLY 1 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29 Alpha Recycling in Confirmatory Clinical Trials Unifies, Simplifies, Extends many common MTPs Olivier Guilbaud Senior Principal Scientist AstraZeneca R&D, Södertälje, Sweden

2. Proprietary and Confidential © AstraZeneca 2009 FOR INTERNAL USE ONLY 2 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29 Outline Background Splitting, Recycling, Adding, (parts of alpha) Graphs and Default Graphs for MTPs Rejection Algorithm Holm’s MTP for groups of Hs, and a problem to discuss Improvements through extensions of the Default Graph Summary, and further results: (Simultaneous Confidence regions, CTP-formulation of MTPs)

3. Proprietary and Confidential © AstraZeneca 2009 FOR INTERNAL USE ONLY 3 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29 Background: Regulatory environment Citation from EMEA/CPMP’s (2002) Points to Consider on Multiplicity Issues in Clinical Trials : “ … multiplicity can have a substantial influence on the rate of false positive conclusions …” ”As a general rule it can be stated that control of the family-wise type-I error rate in the strong sense … is a minimal prerequisite for confirmatory claims”

4. Proprietary and Confidential © AstraZeneca 2009 FOR INTERNAL USE ONLY 4 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29 Background: Multiple Confirmatory Comparisons Several Efficacy Variables Several Tolerability Variables Several Comparators (e.g. Placebo, Active 1, Active 2, …) Several Doses Delta-Noninferiority / Superiority / Delta-Superiority Several Subgroups/Kinds of Subjects Several kinds of Administration (e.g. once daily, twice daily, …) Confirmatory: Pre-specified family of multiple comparisons (i.e. statistical tests). Risk of getting any ”false positive” result/conclusion is   (no matter how many or which H 0 s are true – Strong Control)

5. Proprietary and Confidential © AstraZeneca 2009 FOR INTERNAL USE ONLY 5 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29 Background: Abbreviations/Terminology MTP : Multiple Testing Procedure MCP : Multiple Comparison Procedure FWER : Family-Wise Error Rate (type-I errors) = Pr[at least 1 true H i in the Family is rejected by the MTP] (to be controlled to be   in the strong sense)  , no matter how many, or which, H i s in the family are true In contrast to the weak sense (old) :  , if all H i s in the family are true

6. Proprietary and Confidential © AstraZeneca 2009 FOR INTERNAL USE ONLY 6 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29 Background: brief refresher about three basic MTPs Three basic MTPs for Family {H 1, H 2, H 3 } based on raw p-values p 1, p 2, p 3 Holm MTP (1979) p (1)   /3, p (2)   /2, p (3)   /1 Fixed-Sequence MTP (old) p 1  , p 2  , p 3   Fallback MTP (2003/2005) weights w 1, w 2, w 3 p 1  w 1 , p 2  (w 2 +) , p 3  (w 3 +)  Now, to the Alpha-Recycling framework

7. Proprietary and Confidential © AstraZeneca 2009 FOR INTERNAL USE ONLY 7 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29 Two articles in Statistics in Medicine, February 2009 Only this now

8. Proprietary and Confidential © AstraZeneca 2009 FOR INTERNAL USE ONLY 8 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29 Idea: Splitting and Recycling test-mass (= part of  ) Bonferroni: split of  Fixed-Sequence: recycling of  Combination of split & recycling of  Throughout: Raw p-value available for each null hypothesis H

9. Proprietary and Confidential © AstraZeneca 2009 FOR INTERNAL USE ONLY 9 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29 Idea: Adding test-mass from different paths Parallel-gatekeeping MTP: 2 graphs for same MTP Default graph Important for rejection Algorithm Its sequences reflect the possible rejection paths

10. Proprietary and Confidential © AstraZeneca 2009 FOR INTERNAL USE ONLY 10 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29 Idea: Adding test-mass from different paths Fallback MTP for 3 Hs: 2 graphs for same MTP Default graph Important for rejection Algorithm Its sequences reflect the possible rejection paths

11. Proprietary and Confidential © AstraZeneca 2009 FOR INTERNAL USE ONLY 11 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29 Idea: Adding test-mass from different paths Holm’s MTP for 3 Hs: 2 graphs for same MTP Default graph Important for rejection Algorithm Its sequences reflect the possible rejection paths

12. Proprietary and Confidential © AstraZeneca 2009 FOR INTERNAL USE ONLY 12 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29 Rejection Algorithm - based on Default-Graph Seqs This MTP controls (in the strong sense) the FWER to be   ! Step 1. For each ”first” H in Sequences, (a) add-upp test mass from ”its” Sequences (b) test H at this added-up level If nothing is rejected, then Stop; otherwise: Reduce Sequences by deleting all rejected H from Sequences, and go to Step 2 with remaining reduced Sequences Step r = 2, 3, …. For each ”first” H in remaining reduced Sequences, (a) add-upp test mass from ”its” Sequences (b) test H at this added-up level If nothing is rejected, then Stop; otherwise: Reduce Sequences by deleting all rejected H from Sequences, and go to Step r + 1 with remaining reduced Sequences

13. Proprietary and Confidential © AstraZeneca 2009 FOR INTERNAL USE ONLY 13 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29 Comments number of sequences can be increased (no restriction) length of sequences can be increased (no restriction) if not all sequences contain all Hs, then improvements are possible (by increasing length of relevant sequences) interpretability of MTP is important (also when improvements are considered)

14. Proprietary and Confidential © AstraZeneca 2009 FOR INTERNAL USE ONLY 14 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29 A refresher – Bonferroni & Holm for groups of Hs Step1: Bonferroni-version as above Step2: If one group is entirely rejected, but not the other, then try again in the other at increased level  12 34 Bonferroni for 2 groups of 2 Hs, with Fixed-Seq testing within group  /2 Holm for 2 groups of 2 Hs, with Fixed-Seq testing within group: 12 34  /2 112 34  1

15. Proprietary and Confidential © AstraZeneca 2009 FOR INTERNAL USE ONLY 15 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29 Problem to discuss with your neighbour 12 34 Bonferroni for 2 groups of 2 Hs, with Fixed-Seq testing within group  /2 (is already in default-graph form): Extend sequences of default graph to get recycling improvement equivalent to Holm for 2 groups of 2 Hs, with Fixed-Seq testing within group 12 34  /2

16. Proprietary and Confidential © AstraZeneca 2009 FOR INTERNAL USE ONLY 16 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29 Recent Example discussed by FDA statisticians Confirmatory family with 4 null hypotheses : H1H1 H2H2 H3H3 H4H4 ww (1-w)  

17. Proprietary and Confidential © AstraZeneca 2009 FOR INTERNAL USE ONLY 17 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29 Recent Example discussed by FDA statisticians Alpha-recycling applied to show Non-inf & Superiority for Primary & Secondary Variables H1H1 H2H2 H3H3 H4H4 ww (1-w)   Basic version H1H1 H2H2 H3H3 ww (1-w)  H1H1 H4H4 Reformulation in Recycling terms Very easy to test with rejection algorithm ! But why not recycle also after the last H in each sequence ?

18. Proprietary and Confidential © AstraZeneca 2009 FOR INTERNAL USE ONLY 18 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29 Recent Example discussed by FDA statisticians Alpha-recycling applied to show Non-inf & Superiority for Primary & Secondary Variables - Simple improvement ImprovementReformulation in Recycling terms H1H1 H2H2 H3H3 ww (1-w)  H1H1 H4H4 H3H3 H2H2 H4H4 H1H1 H2H2 H3H3 H4H4 ww  Very easy to test with rejection algorithm ! This improved MTP is ”  -exhaustive” ! May be questioned by some (not me)!

19. Proprietary and Confidential © AstraZeneca 2009 FOR INTERNAL USE ONLY 19 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29 Sequences can be extended/added for more rejections Improved Fallback MTP for 3 Hs: 2 graphs for same MTP Default graph This improved MTP is ”  -exhaustive” !

20. Proprietary and Confidential © AstraZeneca 2009 FOR INTERNAL USE ONLY 20 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29 Sequences can be extended/added for more rejections Improved Parallel-gatekeeping : 2 graphs for same MTP Default graph This improved MTP is ”  -exhaustive” !

21. Proprietary and Confidential © AstraZeneca 2009 FOR INTERNAL USE ONLY 21 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29 XXX example: 2 primary & 2 second vars, 2 age stata 1 23 4 56 8 911 7 1012  Original proposal Simple improvement possible (  Holm instead of Bonf after H 8 ): Add sub-seq (11, 12) to Default-graph sequences ending with (9, 10) Add sub-seq (9, 10) to Default-graph sequences ending with (11, 12) 1 2 7 8 9 10  /8 1 3 7 8 9 10  /8 4 5 7 8 9 10  /8 4 6 7 8 9 10  /8 1 2 7 8 11 12  /8 1 3 7 8 11 12  /8 4 5 7 8 11 12  /8 4 6 7 8 11 12  /8 Default graph Reformulation in Recycling terms to show strong control of FWER

22. Proprietary and Confidential © AstraZeneca 2009 FOR INTERNAL USE ONLY 22 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29 Compact notation for MTP in XXX example 1 23 4 56 8 911 7 1012  Original proposal Compact notation for this recycling MTP (B here is Bonf-operator): B(H 1 B(H 2, H 3 ), H 4 B(H 5, H 6 )) H 7 H 8 B(H 9 H 10, H 11 H 12 ) Can be expanded (using certain rules) to 8 sequences of Hs above 1 2 7 8 9 10  /8 1 3 7 8 9 10  /8 4 5 7 8 9 10  /8 4 6 7 8 9 10  /8 1 2 7 8 11 12  /8 1 3 7 8 11 12  /8 4 5 7 8 11 12  /8 4 6 7 8 11 12  /8 Default graph Reformulation in Recycling terms

23. Proprietary and Confidential © AstraZeneca 2009 FOR INTERNAL USE ONLY 23 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29 YYY example: 3 doses vs. 0, 1 primary & 1 second var 1 3 1 5 2 5 2 4 66  /4 Reformulation in Recycling terms Default graph 12 345 6 Basic version  /2 Idea: To give lowest dose a chance if at least 1 of the 2 larger doses works for the primary variable

24. Proprietary and Confidential © AstraZeneca 2009 FOR INTERNAL USE ONLY 24 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29 YYY example – Improvement 1 1 3 1 5 2 5 2 4 66  /4 Reformulation in Recycling terms Default graph 5 6 5 6 12 345 6 Improvement 1  /2 Increases level for H 5 and H 6 if H 3 and/or H 4 are rejected

25. Proprietary and Confidential © AstraZeneca 2009 FOR INTERNAL USE ONLY 25 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29 YYY example – Improvement 2 12 345 6 Improvement 2 1/2  Gives H 1 or H 2 a (Holm-type) second opportunity to be rejected if not rejected initially 1 3 1 5 2 5 2 4 66  /4  /8  /4 Reformulation in Recycling terms Default graph 5 6 5 6 1 5 6  /8 2 5 6 2 4 22 4 11 31 3 3 43 4 34

26. Proprietary and Confidential © AstraZeneca 2009 FOR INTERNAL USE ONLY 26 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29 Summary, and further results Easy to formulate (cf. Closed-Testing Pocedures) Easy to perform (even manually) Easy to calculate multiplicity-adjusted p-values (algorithm similar to that for rejections) Easy to obtain weights of Bonferroni test for any intersection hypothesis H I of corresponding CTP (useful for contruction of Conf regions) Easy to construct new MTPs (for new problems) Unifies many common MTPs Compact algebraic notation available (to describe and derive default graphs)

27. Proprietary and Confidential © AstraZeneca 2009 FOR INTERNAL USE ONLY 27 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29 Corresponding Simultaneous Confidence Regions ??? Holm MTP (1979) p (1)   /3, p (2)   /2, p (3)   /1 Fixed-Sequence MTP (old) p 1  , p 2  , p 3   Fallback MTP (2003/2005) weights w 1, w 2, w 3 p 1  w 1 , p 2  (w 2 +) , p 3  (w 3 +)  Solution in JASA 1999 Open until 2007 Three basic MTPs for Family {H 1, H 2, H 3 } based on raw p-values p 1, p 2, p 3

28. Proprietary and Confidential © AstraZeneca 2009 FOR INTERNAL USE ONLY 28 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29 Corresponding Simultaneous Confidence Regions !!! Amazing : Simultanously presented at MCP 2007 in Vienna 28 years after Holm (1979) Extensions and Relations Confidence regions require weights that can be obtained directly from rejection algorithm of recycling approach

29. Proprietary and Confidential © AstraZeneca 2009 FOR INTERNAL USE ONLY 29 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29 Alternative formulation of basic MTPs as CTPs Basic MTPs as Closed-Testing Procedures for Family {H 1, H 2, H 3 } based on raw p-values Holm MTP (1979) p (1)   /3, p (2)   /2, p (3)   /1 Fixed-Sequence MTP (old) p 1  , p 2  , p 3   Fallback MTP (2003/2005) weights w 1, w 2, w 3 p 1  w 1 , p 2  (w 2 +) , p 3  (w 3 +) 

30. Proprietary and Confidential © AstraZeneca 2009 FOR INTERNAL USE ONLY 30 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29 Holm – Closed-Test formulation Reject H i iff each H I with i  I is rejected by weighted Bonf-test that rejects an H I =  k  I H k iff p k   w k (I) for some k  I ( weights: w k (I)  0,  k  I w k (I)  1 ) H1H1 H2H2 H3H3 H 12 H 13 H 23 H 123 E.g.: H 1 is rejected by Holm iff each H 123, H 12, H 13, and H 1 is rejected by its weighted Bonf-test

31. Proprietary and Confidential © AstraZeneca 2009 FOR INTERNAL USE ONLY 31 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29 Fixed-Sequence – Closed-Test formulation Reject H i iff each H I with i  I is rejected by weighted Bonf-test that rejects an H I =  k  I H k iff p k   w k (I) for some k  I ( weights: w k (I)  0,  k  I w k (I)  1 ) H1H1 H2H2 H3H3 H 12 H 13 H 23 H 123 E.g.: H 1 is rejected by F-S iff each H 123, H 12, H 13, and H 1 is rejected by its weighted Bonf-test, i.e. iff p 1  

32. Proprietary and Confidential © AstraZeneca 2009 FOR INTERNAL USE ONLY 32 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29 Fallback – Closed-Test formulation (2005) Reject H i iff each H I with i  I is rejected by weighted Bonf-test that rejects an H I =  k  I H k iff p k   w k (I) for some k  I ( weights: w k (I)  0,  k  I w k (I)  1 ) H1H1 H2H2 H3H3 H 12 H 13 H 23 H 123 E.g.: H 1 is rejected by Fallback iff each H 123, H 12, H 13, and H 1 is rejected by its weighted Bonf-test, i.e. iff p 1   w 

33. Proprietary and Confidential © AstraZeneca 2009 FOR INTERNAL USE ONLY 33 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29 Weights for intersection H I s in XXX example 1 23 4 56 8 911 7 1012  Original proposal The CTP formulation involves 2 12 - 1 = 4095 different H I =  k  I H k. But the  w k (I) for k  I to be used in the Bonf-test of a H I =  k  I H k can easily be obtained from”first” H i s remaining after deleting all indexes in I c from seq’s. E.g. with I = {1, 5, 11}, what are  w 1 (I),  w 5 (I),  w 11 (I) ? 1 2 7 8 9 10  /8 1 3 7 8 9 10  /8 4 5 7 8 9 10  /8 4 6 7 8 9 10  /8 1 2 7 8 11 12  /8 1 3 7 8 11 12  /8 4 5 7 8 11 12  /8 4 6 7 8 11 12  /8 Default graph Reformulation in Recycling terms

34. Proprietary and Confidential © AstraZeneca 2009 FOR INTERNAL USE ONLY 34 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010-11-29 Outline Background Splitting, Recycling, Adding, (parts of alpha) Graphs and Default Graphs for MTPs Rejection Algorithm Holm’s MTP for groups of Hs, and a problem to discuss Improvements through extensions of the Default Graph Summary, and further results: (Simultaneous Confidence regions, CTP-formulation of MTPs)