Balancing the practical implications of adaptive designs with the statistical benefits Mahesh Parmar MRC Clinical Trials Unit at UCL

Rationale for adaptive trials Our adaptive trials and their setting Overview Rationale for adaptive trials Our adaptive trials and their setting Discuss some broad practical and statistical challenges Offer some solutions and thoughts Mention some other relevant presentations on these trials at this meeting However, adaptive design comes with a (sometimes significant) cost, at least from a methodological point of view. Because we need to plan possible adaptations in advance (prospectively), a trial can become so complex that the operating characteristics become difficult to control. Adaptations can introduce bias, and not only statistical bias. There can also be operational bias, for example if an adaptation suggests that the results of a trial go in a certain direction. Any systematic tendency or bias can compromise the integrity and the validity of the trial, or can make it very difficult to interpret (and explain) the results. We may improve cost efficiency at the level of the trial and the patients in the trial, however if not properly conducted, there is a high risk that such a trial can result in clinical results which are difficult to interpret or translate into daily practice.

Adaptive trials are increasing - why? The current trials process is: Too slow - > 10 years Too costly ~ $1 Billion Too often show that new is not better than standard Not efficient when we have many therapies to test

Adaptive designs particularly gaining popularity at Early/Middle Phase Bayesian Adaptive Randomisation Designs ISPY ISPY2 BATTLE Aim is to evaluate many therapies in many biomarker defined groups and pick the treatment/biomarker combinations which are ready for evaluation at phase III However, adaptive design comes with a (sometimes significant) cost, at least from a methodological point of view. Because we need to plan possible adaptations in advance (prospectively), a trial can become so complex that the operating characteristics become difficult to control. Adaptations can introduce bias, and not only statistical bias. There can also be operational bias, for example if an adaptation suggests that the results of a trial go in a certain direction. Any systematic tendency or bias can compromise the integrity and the validity of the trial, or can make it very difficult to interpret (and explain) the results. We may improve cost efficiency at the level of the trial and the patients in the trial, however if not properly conducted, there is a high risk that such a trial can result in clinical results which are difficult to interpret or translate into daily practice.

Biggest Gains - Late Phase Trials Most costly stage Lengthiest stage However, relatively few examples However, adaptive design comes with a (sometimes significant) cost, at least from a methodological point of view. Because we need to plan possible adaptations in advance (prospectively), a trial can become so complex that the operating characteristics become difficult to control. Adaptations can introduce bias, and not only statistical bias. There can also be operational bias, for example if an adaptation suggests that the results of a trial go in a certain direction. Any systematic tendency or bias can compromise the integrity and the validity of the trial, or can make it very difficult to interpret (and explain) the results. We may improve cost efficiency at the level of the trial and the patients in the trial, however if not properly conducted, there is a high risk that such a trial can result in clinical results which are difficult to interpret or translate into daily practice.

An Adaptive Design at Phase III Multi-arm, Multi-stage Platform trials However, adaptive design comes with a (sometimes significant) cost, at least from a methodological point of view. Because we need to plan possible adaptations in advance (prospectively), a trial can become so complex that the operating characteristics become difficult to control. Adaptations can introduce bias, and not only statistical bias. There can also be operational bias, for example if an adaptation suggests that the results of a trial go in a certain direction. Any systematic tendency or bias can compromise the integrity and the validity of the trial, or can make it very difficult to interpret (and explain) the results. We may improve cost efficiency at the level of the trial and the patients in the trial, however if not properly conducted, there is a high risk that such a trial can result in clinical results which are difficult to interpret or translate into daily practice.

What we already know: insufficient progress Most trials use traditional 2-arm design <50% superiority trials show ‘new’ better? Source: Clinicaltrials.gov Added 01/2010-07/2012 Clinical trial Randomised Interventional Superiority N=632

Traditional vs Multi-arm, Multi-stage (MAMS)

Traditional vs MAMS Multi-arm Many research arms Multi-stage Is there a reason to continue randomising to a specific research arm? Opportunity cost in continuing to assess something that isn’t going to work well enough

Assume half of treatments are “better” -Djulbegovic, Nature, 2013

Assume half of treatments are “better” -Djulbegovic, Nature, 2013 P(>1 success) increases with more comparisons 2 Two 2-arm trials I

Assume half of treatments are “better” -Djulbegovic, Nature, 2013 P(>1 success) increases with more comparisons 2 One 3-arm trial I Some correlation from common control arm 0.5

Assume half of treatments are “better” -Djulbegovic, Nature, 2013 P(>1 success) increases with more comparisons 3 2 One 4-arm trial I 0.5

Assume half of treatments are “better” -Djulbegovic, Nature, 2013 4 P(>1 success) increases with more comparisons 3 2 One 5-arm trial I 0.5

Assume half of treatments are “better” -Djulbegovic, Nature, 2013 5 4 P(>1 success) increases with more comparisons 3 2 One 6-arm trial I 0.5

 extra correlation of treatment effect of similar research treatments 5 4 3 2 I 0.5

5 4 3 2 I 0.5

5 4 3 5 4 2 3 2 I I

5 4 3 5 4 2 3 2 I I

STAMPEDE – MAMS platform trial in prostate cancer

STAMPEDE – MAMS platform trial in prostate cancer

STAMPEDE – MAMS trial in prostate cancer

STAMPEDE – MAMS trial in prostate cancer 1184 A Standard-of-care (SOC) 593 B SOC + zoledronic acid (ZA) 592 C SOC + docetaxel (Doc) 593 E SOC + ZA + D 2962 Oct-2005 to Mar-2013

Results changed practice Improved survival with chemotherapy Immediately changed practice SOC+Doc 4 weeks SOC Death Pts SOC 415 1184 SOC+Doc 175 592 HR (95%CI) 0.78 (0.66, 0.93) P-value 0.006 SOC alone - median OS 71m SOC+Doc – median OS 81m

MAMS trials as a platform for new questions What if there is a new treatment worthy of testing while another trial is recruiting? Motivating example: abiraterone Poor options ? Not assess ? Set-up competing trial ? Wait till after current trial Better option √ Amend current trial to incorporate testing Efficient use of volunteers Efficient use of resources Quicker time to top speed

STAMPEDE – MAMS trial in prostate cancer Amend further because Conditional approval from Cancer Research UK to assess metformin which looks very interesting in this patient group Repurposing a simple cheap diabetic treatment Report at least 5 years earlier and at two thirds costs of standalone trial [Next slide] [Ties in with Ruth’s talk]

STAMPEDE – MAMS trial in prostate cancer Amend further because Conditional approval from Cancer Research UK to assess metformin which looks very interesting in this patient group Repurposing a simple cheap diabetic treatment Report at least 5 years earlier and at two thirds costs of standalone trial [Next slide] [Ties in with Ruth’s talk]

STAMPEDE – MAMS trial in prostate cancer Amend further because Conditional approval from Cancer Research UK to assess metformin which looks very interesting in this patient group Repurposing a simple cheap diabetic treatment Report at least 5 years earlier and at two thirds costs of standalone trial [Next slide] [Ties in with Ruth’s talk]

Some Practical/Statistical Issues Chief Investigators Type I error rate Funding and approvals – especially new arms Doctors/patients views Dropping and Adding new arms Operational/Database Issues In the traditional approach, modifications during the course of the trial to make the approach more applicable (to reality) are not allowed. There are many reasons to use adaptive designs (also known as adaptive pathways). In an environment subject to economical challenges, adaptive designs appear to be appealing for pharmaceutical industry, academic institutions, clinicians, and also for patients.

Chief Investigators For all new research arms Spreads workload New Comparison Chief Investigators Spreads workload Spreads opportunity Anyone can propose a new research arm Clear criteria for assessment and prioritisation of new research arms However, adaptive design comes with a (sometimes significant) cost, at least from a methodological point of view. Because we need to plan possible adaptations in advance (prospectively), a trial can become so complex that the operating characteristics become difficult to control. Adaptations can introduce bias, and not only statistical bias. There can also be operational bias, for example if an adaptation suggests that the results of a trial go in a certain direction. Any systematic tendency or bias can compromise the integrity and the validity of the trial, or can make it very difficult to interpret (and explain) the results. We may improve cost efficiency at the level of the trial and the patients in the trial, however if not properly conducted, there is a high risk that such a trial can result in clinical results which are difficult to interpret or translate into daily practice.

Type I error rate Do we need to allow for the fact that we are testing multiple primary hypotheses Familywise error rate Many argue not David Cox However, may be challenged (by some) at peer review However, adaptive design comes with a (sometimes significant) cost, at least from a methodological point of view. Because we need to plan possible adaptations in advance (prospectively), a trial can become so complex that the operating characteristics become difficult to control. Adaptations can introduce bias, and not only statistical bias. There can also be operational bias, for example if an adaptation suggests that the results of a trial go in a certain direction. Any systematic tendency or bias can compromise the integrity and the validity of the trial, or can make it very difficult to interpret (and explain) the results. We may improve cost efficiency at the level of the trial and the patients in the trial, however if not properly conducted, there is a high risk that such a trial can result in clinical results which are difficult to interpret or translate into daily practice.

Type I error rate: STAMPEDE Reported the FWER for the original 5 research arms For a new research arm If only 1 control patient in common between this new arm and the original 5 research arms Almost independent trials No allowance for multiple hypotheses However, adaptive design comes with a (sometimes significant) cost, at least from a methodological point of view. Because we need to plan possible adaptations in advance (prospectively), a trial can become so complex that the operating characteristics become difficult to control. Adaptations can introduce bias, and not only statistical bias. There can also be operational bias, for example if an adaptation suggests that the results of a trial go in a certain direction. Any systematic tendency or bias can compromise the integrity and the validity of the trial, or can make it very difficult to interpret (and explain) the results. We may improve cost efficiency at the level of the trial and the patients in the trial, however if not properly conducted, there is a high risk that such a trial can result in clinical results which are difficult to interpret or translate into daily practice.

STAMPEDE – MAMS trial in prostate cancer Amend further because Conditional approval from Cancer Research UK to assess metformin which looks very interesting in this patient group Repurposing a simple cheap diabetic treatment Report at least 5 years earlier and at two thirds costs of standalone trial [Next slide] [Ties in with Ruth’s talk] Thinking of FWER, these comparisons have very few patients in common

Type I error rate: STAMPEDE Reported the FWER for the original 5 research arms For a new research arm If only 1 control patient in common between this new arm and the original 5 research arms Almost independent trials No allowance for multiple hypotheses Overlap given by Correlation between test statistics of new arm vs contemporaneous controls and 5 original research arms and their contemporaneous controls Correlation between arm G and Original Arms = 0.12 However, adaptive design comes with a (sometimes significant) cost, at least from a methodological point of view. Because we need to plan possible adaptations in advance (prospectively), a trial can become so complex that the operating characteristics become difficult to control. Adaptations can introduce bias, and not only statistical bias. There can also be operational bias, for example if an adaptation suggests that the results of a trial go in a certain direction. Any systematic tendency or bias can compromise the integrity and the validity of the trial, or can make it very difficult to interpret (and explain) the results. We may improve cost efficiency at the level of the trial and the patients in the trial, however if not properly conducted, there is a high risk that such a trial can result in clinical results which are difficult to interpret or translate into daily practice.

Funding and approvals State in the original protocol that there is plan to include further research arms Most organisations ok… Treat addition of new research arm as if starting a ‘new trial’ Scientific case Peer review Funding Add new arm as an amendment to protocol If overlap of control arm patients modest, no allowance of multiple testing needed However, adaptive design comes with a (sometimes significant) cost, at least from a methodological point of view. Because we need to plan possible adaptations in advance (prospectively), a trial can become so complex that the operating characteristics become difficult to control. Adaptations can introduce bias, and not only statistical bias. There can also be operational bias, for example if an adaptation suggests that the results of a trial go in a certain direction. Any systematic tendency or bias can compromise the integrity and the validity of the trial, or can make it very difficult to interpret (and explain) the results. We may improve cost efficiency at the level of the trial and the patients in the trial, however if not properly conducted, there is a high risk that such a trial can result in clinical results which are difficult to interpret or translate into daily practice.

Doctors and Patients views Patients have been universally supportive Patient charity groups have been the biggest supporters More than 9000 patients randomised to STAMPEDE Doctors now consider STAMPEDE to be part of routine care However, adaptive design comes with a (sometimes significant) cost, at least from a methodological point of view. Because we need to plan possible adaptations in advance (prospectively), a trial can become so complex that the operating characteristics become difficult to control. Adaptations can introduce bias, and not only statistical bias. There can also be operational bias, for example if an adaptation suggests that the results of a trial go in a certain direction. Any systematic tendency or bias can compromise the integrity and the validity of the trial, or can make it very difficult to interpret (and explain) the results. We may improve cost efficiency at the level of the trial and the patients in the trial, however if not properly conducted, there is a high risk that such a trial can result in clinical results which are difficult to interpret or translate into daily practice.

‘Dropping’ and Adding arms Done by an amendment to the protocol However, many aspects to ceasing randomisation to some arms All doctors and nurses have to be informed Patients in the arm stopped have to be informed Patients in the other arms should be informed For the celecoxib arms we allowed re-randomisation of very small number of patients who had not started treatment and remained eligible However, adaptive design comes with a (sometimes significant) cost, at least from a methodological point of view. Because we need to plan possible adaptations in advance (prospectively), a trial can become so complex that the operating characteristics become difficult to control. Adaptations can introduce bias, and not only statistical bias. There can also be operational bias, for example if an adaptation suggests that the results of a trial go in a certain direction. Any systematic tendency or bias can compromise the integrity and the validity of the trial, or can make it very difficult to interpret (and explain) the results. We may improve cost efficiency at the level of the trial and the patients in the trial, however if not properly conducted, there is a high risk that such a trial can result in clinical results which are difficult to interpret or translate into daily practice.

Operational and Database Issues This is a Platform Alteration: A Trial Management Perspective on the operational aspects of adaptive platform trials Schiavone F; Bathia R; Letchemanan K Tuesday, 3.15-4.15, room 11 Changing platforms without stopping the train: A Data Management Perspective on the operational aspects of adaptive platform trials Hague D; Townsend S; Masters L Poster session, Tuesday

Conclusions New adaptive study designs are needed: Because progress has been too slow STAMPEDE will address 8 major questions in 15 years More therapies than ever before More efficient use of resources Shorter development process Testing Many Primary Hypotheses FOCUS4 Add Aspirin Truncate TB Wound Infections Alzheimers Disease