1. Balancing the practical implications of adaptive designs with the statistical benefits
Mahesh Parmar
MRC Clinical Trials Unit at UCL

2. Rationale for adaptive trials Our adaptive trials and their setting
Overview
Rationale for adaptive trials
Our adaptive trials and their setting
Discuss some broad practical and statistical challenges
Offer some solutions and thoughts
Mention some other relevant presentations on these trials at this meeting
However, adaptive design comes with a (sometimes significant) cost, at least from a methodological point of view. Because we need to plan possible adaptations in advance (prospectively), a trial can become so complex that the operating characteristics become difficult to control.
Adaptations can introduce bias, and not only statistical bias. There can also be operational bias, for example if an adaptation suggests that the results of a trial go in a certain direction.
Any systematic tendency or bias can compromise the integrity and the validity of the trial, or can make it very difficult to interpret (and explain) the results.
We may improve cost efficiency at the level of the trial and the patients in the trial, however if not properly conducted, there is a high risk that such a trial can result in clinical results which are difficult to interpret or translate into daily practice.

3. Adaptive trials are increasing - why?
The current trials process is:
Too slow - > 10 years
Too costly ~ $1 Billion
Too often show that new is not better than standard
Not efficient when we have many therapies to test

4. Adaptive designs particularly gaining popularity at Early/Middle Phase
Bayesian Adaptive Randomisation Designs
ISPY
ISPY2
BATTLE
Aim is to evaluate many therapies in many biomarker defined groups and pick the treatment/biomarker combinations which are ready for evaluation at phase III
However, adaptive design comes with a (sometimes significant) cost, at least from a methodological point of view. Because we need to plan possible adaptations in advance (prospectively), a trial can become so complex that the operating characteristics become difficult to control.
Adaptations can introduce bias, and not only statistical bias. There can also be operational bias, for example if an adaptation suggests that the results of a trial go in a certain direction.
Any systematic tendency or bias can compromise the integrity and the validity of the trial, or can make it very difficult to interpret (and explain) the results.
We may improve cost efficiency at the level of the trial and the patients in the trial, however if not properly conducted, there is a high risk that such a trial can result in clinical results which are difficult to interpret or translate into daily practice.

5. Biggest Gains - Late Phase Trials
Most costly stage
Lengthiest stage
However, relatively few examples
However, adaptive design comes with a (sometimes significant) cost, at least from a methodological point of view. Because we need to plan possible adaptations in advance (prospectively), a trial can become so complex that the operating characteristics become difficult to control.
Adaptations can introduce bias, and not only statistical bias. There can also be operational bias, for example if an adaptation suggests that the results of a trial go in a certain direction.
Any systematic tendency or bias can compromise the integrity and the validity of the trial, or can make it very difficult to interpret (and explain) the results.
We may improve cost efficiency at the level of the trial and the patients in the trial, however if not properly conducted, there is a high risk that such a trial can result in clinical results which are difficult to interpret or translate into daily practice.

6. An Adaptive Design at Phase III
Multi-arm, Multi-stage Platform trials
However, adaptive design comes with a (sometimes significant) cost, at least from a methodological point of view. Because we need to plan possible adaptations in advance (prospectively), a trial can become so complex that the operating characteristics become difficult to control.
Adaptations can introduce bias, and not only statistical bias. There can also be operational bias, for example if an adaptation suggests that the results of a trial go in a certain direction.
Any systematic tendency or bias can compromise the integrity and the validity of the trial, or can make it very difficult to interpret (and explain) the results.
We may improve cost efficiency at the level of the trial and the patients in the trial, however if not properly conducted, there is a high risk that such a trial can result in clinical results which are difficult to interpret or translate into daily practice.

7. What we already know: insufficient progress
Most trials use traditional 2-arm design
<50% superiority trials show ‘new’ better?
Source: Clinicaltrials.gov
Added 01/ /2012
Clinical trial
Randomised
Interventional
Superiority
N=632

8. Traditional vs Multi-arm, Multi-stage (MAMS)

9. Traditional vs MAMS Multi-arm Many research arms Multi-stage
Is there a reason to continue randomising to a specific research arm?
Opportunity cost in continuing to assess something that isn’t going to work well enough

10. Assume half of treatments are “better” -Djulbegovic, Nature, 2013

11. Assume half of treatments are “better” -Djulbegovic, Nature, 2013
P(>1 success) increases with more comparisons 2
Two 2-arm trials I

12. Assume half of treatments are “better” -Djulbegovic, Nature, 2013
P(>1 success) increases with more comparisons 2
One 3-arm trial I
Some correlation from common control arm
0.5

13. Assume half of treatments are “better” -Djulbegovic, Nature, 2013
P(>1 success) increases with more comparisons 3 2
One 4-arm trial I
0.5

14. Assume half of treatments are “better” -Djulbegovic, Nature, 20134
P(>1 success) increases with more comparisons 3 2
One 5-arm trial I
0.5

15. Assume half of treatments are “better” -Djulbegovic, Nature, 20135 4
P(>1 success) increases with more comparisons 3 2
One 6-arm trial I
0.5

16.  extra correlation of treatment effect of similar research treatments5 4 3 2 I
0.5

17. 5 4 3 2 I
0.5

18. 5 4 3 5 4 2 3 2 I I

19. 5 4 3 5 4 2 3 2 I I

20. STAMPEDE – MAMS platform trial in prostate cancer

21. STAMPEDE – MAMS platform trial in prostate cancer

22. STAMPEDE – MAMS trial in prostate cancer

23. STAMPEDE – MAMS trial in prostate cancer
1184 A Standard-of-care (SOC)
593 B SOC + zoledronic acid (ZA)
592 C SOC + docetaxel (Doc)
593 E SOC + ZA + D
2962 Oct-2005 to Mar-2013

24. Results changed practice
Improved survival with chemotherapy
Immediately changed practice
SOC+Doc
4 weeks
SOC
Death Pts
SOC
SOC+Doc
HR (95%CI) 0.78 (0.66, 0.93)
P-value
SOC alone - median OS 71m
SOC+Doc – median OS 81m

25. MAMS trials as a platform for new questions
What if there is a new treatment worthy of testing while another trial is recruiting?
Motivating example: abiraterone
Poor options
? Not assess
? Set-up competing trial
? Wait till after current trial
Better option
√ Amend current trial to incorporate testing
Efficient use of volunteers
Efficient use of resources
Quicker time to top speed

26. STAMPEDE – MAMS trial in prostate cancer
Amend further because
Conditional approval from Cancer Research UK to assess metformin which looks very interesting in this patient group
Repurposing a simple cheap diabetic treatment
Report at least 5 years earlier and at two thirds costs of standalone trial
[Next slide]
[Ties in with Ruth’s talk]

27. STAMPEDE – MAMS trial in prostate cancer
Amend further because
Conditional approval from Cancer Research UK to assess metformin which looks very interesting in this patient group
Repurposing a simple cheap diabetic treatment
Report at least 5 years earlier and at two thirds costs of standalone trial
[Next slide]
[Ties in with Ruth’s talk]

28. STAMPEDE – MAMS trial in prostate cancer
Amend further because
Conditional approval from Cancer Research UK to assess metformin which looks very interesting in this patient group
Repurposing a simple cheap diabetic treatment
Report at least 5 years earlier and at two thirds costs of standalone trial
[Next slide]
[Ties in with Ruth’s talk]

29. Some Practical/Statistical Issues
Chief Investigators
Type I error rate
Funding and approvals – especially new arms
Doctors/patients views
Dropping and Adding new arms
Operational/Database Issues
In the traditional approach, modifications during the course of the trial to make the approach more applicable (to reality) are not allowed.
There are many reasons to use adaptive designs (also known as adaptive pathways). In an environment subject to economical challenges, adaptive designs appear to be appealing for pharmaceutical industry, academic institutions, clinicians, and also for patients.

30. Chief Investigators For all new research arms Spreads workload
New Comparison Chief Investigators
Spreads workload
Spreads opportunity
Anyone can propose a new research arm
Clear criteria for assessment and prioritisation of new research arms
However, adaptive design comes with a (sometimes significant) cost, at least from a methodological point of view. Because we need to plan possible adaptations in advance (prospectively), a trial can become so complex that the operating characteristics become difficult to control.
Adaptations can introduce bias, and not only statistical bias. There can also be operational bias, for example if an adaptation suggests that the results of a trial go in a certain direction.
Any systematic tendency or bias can compromise the integrity and the validity of the trial, or can make it very difficult to interpret (and explain) the results.
We may improve cost efficiency at the level of the trial and the patients in the trial, however if not properly conducted, there is a high risk that such a trial can result in clinical results which are difficult to interpret or translate into daily practice.

31. Type I error rate
Do we need to allow for the fact that we are testing multiple primary hypotheses
Familywise error rate
Many argue not
David Cox
However, may be challenged (by some) at peer review
However, adaptive design comes with a (sometimes significant) cost, at least from a methodological point of view. Because we need to plan possible adaptations in advance (prospectively), a trial can become so complex that the operating characteristics become difficult to control.
Adaptations can introduce bias, and not only statistical bias. There can also be operational bias, for example if an adaptation suggests that the results of a trial go in a certain direction.
Any systematic tendency or bias can compromise the integrity and the validity of the trial, or can make it very difficult to interpret (and explain) the results.
We may improve cost efficiency at the level of the trial and the patients in the trial, however if not properly conducted, there is a high risk that such a trial can result in clinical results which are difficult to interpret or translate into daily practice.

32. Type I error rate: STAMPEDE
Reported the FWER for the original 5 research arms
For a new research arm
If only 1 control patient in common between this new arm and the original 5 research arms
Almost independent trials
No allowance for multiple hypotheses
However, adaptive design comes with a (sometimes significant) cost, at least from a methodological point of view. Because we need to plan possible adaptations in advance (prospectively), a trial can become so complex that the operating characteristics become difficult to control.
Adaptations can introduce bias, and not only statistical bias. There can also be operational bias, for example if an adaptation suggests that the results of a trial go in a certain direction.
Any systematic tendency or bias can compromise the integrity and the validity of the trial, or can make it very difficult to interpret (and explain) the results.
We may improve cost efficiency at the level of the trial and the patients in the trial, however if not properly conducted, there is a high risk that such a trial can result in clinical results which are difficult to interpret or translate into daily practice.

33. STAMPEDE – MAMS trial in prostate cancer
Amend further because
Conditional approval from Cancer Research UK to assess metformin which looks very interesting in this patient group
Repurposing a simple cheap diabetic treatment
Report at least 5 years earlier and at two thirds costs of standalone trial
[Next slide]
[Ties in with Ruth’s talk]
Thinking of FWER, these comparisons have very few patients in common

34. Type I error rate: STAMPEDE
Reported the FWER for the original 5 research arms
For a new research arm
If only 1 control patient in common between this new arm and the original 5 research arms
Almost independent trials
No allowance for multiple hypotheses
Overlap given by
Correlation between test statistics of new arm vs contemporaneous controls and 5 original research arms and their contemporaneous controls
Correlation between arm G and Original Arms = 0.12
However, adaptive design comes with a (sometimes significant) cost, at least from a methodological point of view. Because we need to plan possible adaptations in advance (prospectively), a trial can become so complex that the operating characteristics become difficult to control.
Adaptations can introduce bias, and not only statistical bias. There can also be operational bias, for example if an adaptation suggests that the results of a trial go in a certain direction.
Any systematic tendency or bias can compromise the integrity and the validity of the trial, or can make it very difficult to interpret (and explain) the results.
We may improve cost efficiency at the level of the trial and the patients in the trial, however if not properly conducted, there is a high risk that such a trial can result in clinical results which are difficult to interpret or translate into daily practice.

35. Funding and approvals
State in the original protocol that there is plan to include further research arms
Most organisations ok…
Treat addition of new research arm as if starting a ‘new trial’
Scientific case
Peer review
Funding
Add new arm as an amendment to protocol
If overlap of control arm patients modest, no allowance of multiple testing needed
However, adaptive design comes with a (sometimes significant) cost, at least from a methodological point of view. Because we need to plan possible adaptations in advance (prospectively), a trial can become so complex that the operating characteristics become difficult to control.
Adaptations can introduce bias, and not only statistical bias. There can also be operational bias, for example if an adaptation suggests that the results of a trial go in a certain direction.
Any systematic tendency or bias can compromise the integrity and the validity of the trial, or can make it very difficult to interpret (and explain) the results.
We may improve cost efficiency at the level of the trial and the patients in the trial, however if not properly conducted, there is a high risk that such a trial can result in clinical results which are difficult to interpret or translate into daily practice.

36. Doctors and Patients views
Patients have been universally supportive
Patient charity groups have been the biggest supporters
More than 9000 patients randomised to STAMPEDE
Doctors now consider STAMPEDE to be part of routine care
However, adaptive design comes with a (sometimes significant) cost, at least from a methodological point of view. Because we need to plan possible adaptations in advance (prospectively), a trial can become so complex that the operating characteristics become difficult to control.
Adaptations can introduce bias, and not only statistical bias. There can also be operational bias, for example if an adaptation suggests that the results of a trial go in a certain direction.
Any systematic tendency or bias can compromise the integrity and the validity of the trial, or can make it very difficult to interpret (and explain) the results.
We may improve cost efficiency at the level of the trial and the patients in the trial, however if not properly conducted, there is a high risk that such a trial can result in clinical results which are difficult to interpret or translate into daily practice.

37. ‘Dropping’ and Adding arms
Done by an amendment to the protocol
However, many aspects to ceasing randomisation to some arms
All doctors and nurses have to be informed
Patients in the arm stopped have to be informed
Patients in the other arms should be informed
For the celecoxib arms we allowed re-randomisation of very small number of patients who had not started treatment and remained eligible
However, adaptive design comes with a (sometimes significant) cost, at least from a methodological point of view. Because we need to plan possible adaptations in advance (prospectively), a trial can become so complex that the operating characteristics become difficult to control.
Adaptations can introduce bias, and not only statistical bias. There can also be operational bias, for example if an adaptation suggests that the results of a trial go in a certain direction.
Any systematic tendency or bias can compromise the integrity and the validity of the trial, or can make it very difficult to interpret (and explain) the results.
We may improve cost efficiency at the level of the trial and the patients in the trial, however if not properly conducted, there is a high risk that such a trial can result in clinical results which are difficult to interpret or translate into daily practice.

38. Operational and Database Issues
This is a Platform Alteration: A Trial Management Perspective on the operational aspects of adaptive platform trials
Schiavone F; Bathia R; Letchemanan K
Tuesday, , room 11
Changing platforms without stopping the train: A Data Management Perspective on the operational aspects of adaptive platform trials
Hague D; Townsend S; Masters L
Poster session, Tuesday

39. Conclusions New adaptive study designs are needed:
Because progress has been too slow
STAMPEDE will address 8 major questions in 15 years
More therapies than ever before
More efficient use of resources
Shorter development process
Testing Many Primary Hypotheses
FOCUS4
Add Aspirin
Truncate TB
Wound Infections
Alzheimers Disease