1. Valuing the effect sizes hypothesized in phase 3 trials of targeted therapies in advanced cancer
Dr Nicky Lawrence
Medial Oncologist, PhD Candidate
The University of Sydney, Australia
Co-authors: Dr F Roncolato, Dr A Martin, Prof M Stockler
Thank you for the introduction.

2. Introduction
The aim of a phase 3 trial is to provide reliable estimates of efficacy, safety, and net clinical benefit
Need sufficient power to identify clinically important effects on survival and/or aspects of health-related quality of life
Some clinical trials while statistically significant have questionable clinical significance
There is minimal published data regarding the magnitude of clinical benefit that has been used in the design of oncology phase 3 clinical trials
Point 1: Randomised phase 3 trials are the gold standard for determining efficacy and safety in comparison with a valid control treatment.
Point 2: In practice this means trials need to have sufficient power to identify clinically important effects on survival and/or aspects of health-related quality of life.
Point 3: In recent years trials have become larger.
This occurred as there was concern that small trials may miss clinically important effects.
These larger trials are powered to find smaller differences that are statistically significant.

3. In the cancer setting there are numerous examples in the media with sensationalised heading about the “war” on cancer, with new drugs seen as ‘gamechangers’ or ‘break-throughs’
There is now concern that trials are so large that they are detecting differences which are not clinically important.
Over the last 5 years, there has been increasing work in the oncology field regarding assessment of the results of trials.
This is due to increasing costs of anticancer drugs which may only offer a small benefit.
There is concern about the subsequent impact on the health care system.
This interest in assessing the results of clinicals trials are likely to have impact on the design of clinical trials but there is minimal published data regarding the magnitude of clinical benefit used in the design of oncology phase 3 clinical trials.

4. Aim
To determine the effect sizes hypothesised in phase 3 trials of targeted and immune therapy for advanced cancer and the effect sizes observed, and their relationship
To determine the effect sizes hypothesised in phase 3 trials of targeted and immune therapy for advanced cancer and the effect sizes observed, and their relationship

5. Terminology
Effect size hypothesised: from the statistical hypothesis used to justify the sample size Effect size observed: from the results Relative effect size: typically a hazard ratio Absolute effect size: typically a difference in median survival times or rates
For the purposes of this study when I use the term....
‘effect size hypothesised’ I am referring to the values used in the statistical hypothesis in the design of the trial.
‘effect size observed’ I am referring to actual results seen after the trial is completed.
In advanced cancer trials, the differences between treatment arms are often illustrated using Kaplan Meier Survival Curves.
The ‘relative effect size’ is often compared using hazard ratio’s.
The ‘absolute effect size’ is often measure in median survival times.

6. Methods We sought: Randomised controlled trial
Trial testing a targeted therapy, including immunotherapy and hormone therapy
Included direct measure of benefit (PFS or OS)
Metastatic solid organ malignancies
≥ 100 participants in each treatment arm
Adult population
Superiority hypothesis
Published 2005 to 2015
We performed a Medline search for randomised phase 3 trials.
Included trials were of targeted therapies including immunotherapy and hormone therapies.
Did not include vaccines or radioisotopes
No restriction on control arm (eg. could be targeted or compared to chemotherapy), excluded those evaluating dose or schedule

7. Data Extraction For each trial we determined: the primary endpoint
the statistical hypothesis: the effect size hypothesized in relative terms and in absolute terms, power, significance, sample size
the effect size observed
trial characteristics including tumour type, sponsor, experimental agent
For each trial we extracted data from the primary publication for each trial and the study protocol if this was publically available.
We extracted the primary endpoint,
the statistical hypothesis
the effect size observed
and characteristics of the trial

8. ASCO recommendations included:
The effect sizes hypothesised were compared to the American Society of Clinical Oncology (ASCO) recommendations for clinical trial design1
ASCO recommendations included:
All trials with OS as primary endpoint should be designed HR ≤ 0.8
Relative and absolute effect size recommendations for tumour types in specific clinical scenarios
The effect sizes hypothesised were compared to the American Society of Clinical Oncology (ASCO) recommendations for clinical trial design1
And were based on expert consensus
General recommendations were given for all trials with Overall Survival as a primary endpoint. This was that trials should be designed with a HR ≤ 0.8
More specific recommendations were given for particular tumour types in specific clinical scenarios, for
Ellis, L. M., et al. (2014). "American Society of Clinical Oncology perspective: Raising the bar for clinical trials by defining clinically meaningful outcomes." Journal of Clinical Oncology 32(12):

9. Characteristics of 216 included trials
Results
Characteristics of 216 included trials N
(%)
Experimental Agent
Tyrosine Kinase Inhibitor 85
(39)
Monoclonal Antibody 63
(29)
Immunotherapy 20
(9)
Hormonal Therapy 14
(6)
mTOR inhibitors 12
Other 22
(10)
Sponsor
Industry
160
(74)
Academic 52
(24)
Unspecified 4
(2)
The experimental agents evaluated included TKI 39%, monoclonal antibodies 30%, immunotherapies and 9% hormonal agents 6%
The majority of trials were commercially sponsored, and first authors were predominantly from North America and Europe.

10. Primary endpoint by tumour typeOS
PFS
Other
Total
Tumour Types N
(%)
Lung 34 21 55
(25)
Breast 3 1 38
(18)
Colorectal 14 2 30
(14)
Melanoma 15 5
(10)
Renal 7 10 17
(8)
Prostate 12
(6)
Gastric / GOJ 8 9
(4)
HCC
Pancreas 6
(3)
Head and Neck 4
(2)
Ovarian / Peritoneal
115
(53%) 97
(45%)
(2%)
216
(100)
The most common primary tumour types were lung, breast, colorectal and melanoma
4 trials either did not specify the endpoint or used endpoints other than PFS or OS
Endpoint used was Overall Survival in 53% of trials, and Progression Free Survival in 44%
Tumour types with longer expected survival times such as breast and ovarian cancer were more likely to use PFS as the primary endpoint.
Trials with a shorter expected survival time such as gastric and pancreatic cancer were more likely to use OS as the primary endpoint.

11. Insufficient information was available to determine both the relative and absolute effect sizes hypothesised in 21% of trials
Sufficient information to determine:
Phase 3 trials N
(%)
Both relative and absolute effect size
170
(79)
Relative effect size only 39
(18)
Absolute effect size only
(0)
Neither 7
(3)
Total
216
(100)
Information from both publications and publically available protocols
The number of trials with sufficient information to determine all critical elements of the design (relative and absolute effect sizes hypothesised, the estimated survival in the control arm, power and significance) was xx%

12. More extreme relative effect sizes hypothesised for PFS than OS
● PFS median HR 0.71 (IQR 0.67 to 0.75)
● OS median HR 0.75 (IQR 0.73 to 0.78)
difference p <
This graph shows the distribution of the relative effect sizes hypothesised in the phase 3 trials
More extreme relative effect sizes hypothesised for trials with PFS as the primary endpoint than OS.
The median relative effect size hypothesised was a HR of 0.71 for PFS and 0.75 for OS

13. Smaller absolute effect sizes hypothesised for PFS and OS
● PFS median 2.3 months (IQR 1.5 to 3.1)
● OS median 2.9 months (IQR 2.2 to 3.8)
difference p = 0.03
This graph shows the distribution of the absolute effect sizes hypothesised in the phase 3 trials shown as the difference in median survival times.
What was striking was how many of the trials hypothesised an absolute improvement of between 2 to 3 months for both OS and PFS

14. Hypothesised versus observed relative effect size for OS
The relationship was low correlation. With sensitivity analysis, excluding outlier, correlation was 0.13, p=0.20
Explain graph:
X axis
Y axis
HR 0.8 was the ASCO recommendation for minimum clinically meaningful benefit.
Black triangles the trials that were statistically significant
Grey triangles those that were not statistically significant
I will come to the coloured data points shortly
Three main points:
1. Just over half of the trials (56%) that were statistically significant found an observed effect size that was less extreme (closer to 1) than what was hypothesised, and were still statistically significant
For example this point which had a hypothesised HR of 0.67, an observed HR of 0.75 and was still statistically significant
2. The red dots show the trials that were statistically significant, but in which the observed HR was less extreme (closer to 1) than 0.8, and so may be of uncertain clinical importance. This occurred in a quarter of trials that were statistically significant.
3. The green dots show the trials that were not statistically significant, but in which the observed HR was more extreme than 0.8 (closer to 0), and so may be of clinical importance despite not being of clinical significance.

15. Hypothesised versus observed relative effect size for PFS
This graph shows the same relationship but for progression free survival as the primary endpoint. There was modest correlation.
There are no guidelines to recommendation the minimum clinically meaningful benefit for PFS. We would usually expect to see a bigger difference in PFS than OS, so it would be reasonable to assume that it would be a HR that is more extreme than 0.8.
57% (55 of 100 ) of trials with PFS as the primary outcome had a statistically significant observed effect size
Just over a third (20 (36%) ) of the statistically significant trials had an observed effect size that was less extreme (closer to 1) than what was hypothesised.
For example hypothesised HR 0.8, and was statistically significant for HR 0.9
There were 4 trials that had an observed effect size where the HR was less extreme (closer to 1) than 0.8, and so may be of uncertain clinical importance.

16. ASCO recommended that a HR ≤ 0
ASCO recommended that a HR ≤ 0.8 should be used for phase 3 trials with a primary outcome of overall survival This was achieved by most trials: 109/111 (98%)
5 trials did not specify a target HR (5/116)
2 trials > 0.8 (one was 0.82, one was 0.85)
22 trials exactly 0.8

17. Absolute Effect Size (months)
ASCO recommendations met for relative but not absolute effect sizes for specific disease types
Lung Cancer
Pancreatic Cancer
ASCO Proposal
Relative Effect Size
HR ≤ 0.8
Absolute Effect Size (months)
≥ 2.5 for squamous,
≥ 3.25 for non-squamous
HR ≤ 0.75
≥ 3
Met
22/23
(96%)
4/23
(17%)
5/6
(83%)
 (0%)
Not met 0 
10/23
(43%)
1/6
4/6
(67%)
Not able to assess
1/23
(4%)
9/23
(39%)  0
2/6
(33%)
ASCO provided recommendations for the relative effect size and absolute effect size in 4 clinical scenarios, all of which were in the first line metastatic setting with a primary outcome of overall survival.
The were not enough trials for the scenarios of breast and colon cancer, and so I have focussed on lung and pancreatic cancer.
In lung cancer ASCO recommended trials should be designed for a HR of 0.8 or less and an absolute improvement in median survival of around 2.5 months for SCC, and 3.25 months for non-squamous. The majority of trials met the recommendations for relative effect size, but far fewer met the recommendations for absolute effect size.
The same can be seen for trials assessing pancreatic cancer.

18. Conclusions
Insufficient information were available to determine both the relative and absolute effect sizes hypothesised in 21% of trials
Median effect sizes hypothesised were:
OS: HR 0.75, 2.9 months
PFS: HR 0.71, 2.3 months
Many trials were designed with observed survival benefits that could be both statistically significant and clinically unimportant

19. Implications for phase 3 trials
Trial reports and protocols should provide all critical elements of trial design
Relative and absolute benefits should both be considered and specified
Relative effects are best for depicting the activity of the treatments
Absolute benefits are best for assessing the benefits and harms of treatment for patients
Randomised phase 3 trials should be designed to provide statistically significant results for observed effects that are clinically important
Relative effects are best for depicting activity of treatment
Absolute benefits are best for depicting benefits and harms of treatment
Randomised phase 3 trials should be designed to provide statistically significant results for observed effects that are clinically important. If phase 3 trials are designed to look for bigger differences, this will require a smaller sample size.