Valuing the effect sizes hypothesized in phase 3 trials of targeted therapies in advanced cancer Dr Nicky Lawrence Medial Oncologist, PhD Candidate The University of Sydney, Australia Co-authors: Dr F Roncolato, Dr A Martin, Prof M Stockler Thank you for the introduction.

Introduction The aim of a phase 3 trial is to provide reliable estimates of efficacy, safety, and net clinical benefit Need sufficient power to identify clinically important effects on survival and/or aspects of health-related quality of life Some clinical trials while statistically significant have questionable clinical significance There is minimal published data regarding the magnitude of clinical benefit that has been used in the design of oncology phase 3 clinical trials Point 1: Randomised phase 3 trials are the gold standard for determining efficacy and safety in comparison with a valid control treatment. Point 2: In practice this means trials need to have sufficient power to identify clinically important effects on survival and/or aspects of health-related quality of life. Point 3: In recent years trials have become larger. This occurred as there was concern that small trials may miss clinically important effects. These larger trials are powered to find smaller differences that are statistically significant.

In the cancer setting there are numerous examples in the media with sensationalised heading about the “war” on cancer, with new drugs seen as ‘gamechangers’ or ‘break-throughs’ There is now concern that trials are so large that they are detecting differences which are not clinically important. Over the last 5 years, there has been increasing work in the oncology field regarding assessment of the results of trials. This is due to increasing costs of anticancer drugs which may only offer a small benefit. There is concern about the subsequent impact on the health care system. This interest in assessing the results of clinicals trials are likely to have impact on the design of clinical trials but there is minimal published data regarding the magnitude of clinical benefit used in the design of oncology phase 3 clinical trials.

Aim To determine the effect sizes hypothesised in phase 3 trials of targeted and immune therapy for advanced cancer and the effect sizes observed, and their relationship To determine the effect sizes hypothesised in phase 3 trials of targeted and immune therapy for advanced cancer and the effect sizes observed, and their relationship

Terminology Effect size hypothesised: from the statistical hypothesis used to justify the sample size Effect size observed: from the results Relative effect size: typically a hazard ratio Absolute effect size: typically a difference in median survival times or rates For the purposes of this study when I use the term.... ‘effect size hypothesised’ I am referring to the values used in the statistical hypothesis in the design of the trial. ‘effect size observed’ I am referring to actual results seen after the trial is completed. In advanced cancer trials, the differences between treatment arms are often illustrated using Kaplan Meier Survival Curves. The ‘relative effect size’ is often compared using hazard ratio’s. The ‘absolute effect size’ is often measure in median survival times.

Methods We sought: Randomised controlled trial Trial testing a targeted therapy, including immunotherapy and hormone therapy Included direct measure of benefit (PFS or OS) Metastatic solid organ malignancies ≥ 100 participants in each treatment arm Adult population Superiority hypothesis Published 2005 to 2015 We performed a Medline search for randomised phase 3 trials. Included trials were of targeted therapies including immunotherapy and hormone therapies. Did not include vaccines or radioisotopes No restriction on control arm (eg. could be targeted or compared to chemotherapy), excluded those evaluating dose or schedule

Data Extraction For each trial we determined: the primary endpoint the statistical hypothesis: the effect size hypothesized in relative terms and in absolute terms, power, significance, sample size the effect size observed trial characteristics including tumour type, sponsor, experimental agent For each trial we extracted data from the primary publication for each trial and the study protocol if this was publically available. We extracted the primary endpoint, the statistical hypothesis the effect size observed and characteristics of the trial

ASCO recommendations included: The effect sizes hypothesised were compared to the American Society of Clinical Oncology (ASCO) recommendations for clinical trial design1 ASCO recommendations included: All trials with OS as primary endpoint should be designed HR ≤ 0.8 Relative and absolute effect size recommendations for tumour types in specific clinical scenarios The effect sizes hypothesised were compared to the American Society of Clinical Oncology (ASCO) recommendations for clinical trial design1 And were based on expert consensus General recommendations were given for all trials with Overall Survival as a primary endpoint. This was that trials should be designed with a HR ≤ 0.8 More specific recommendations were given for particular tumour types in specific clinical scenarios, for Ellis, L. M., et al. (2014). "American Society of Clinical Oncology perspective: Raising the bar for clinical trials by defining clinically meaningful outcomes." Journal of Clinical Oncology 32(12): 1277-1280.

Characteristics of 216 included trials Results Characteristics of 216 included trials N (%) Experimental Agent Tyrosine Kinase Inhibitor 85 (39) Monoclonal Antibody 63 (29) Immunotherapy 20 (9) Hormonal Therapy 14 (6) mTOR inhibitors 12 Other 22 (10) Sponsor Industry 160 (74) Academic 52 (24) Unspecified 4 (2) The experimental agents evaluated included TKI 39%, monoclonal antibodies 30%, immunotherapies and 9% hormonal agents 6% The majority of trials were commercially sponsored, and first authors were predominantly from North America and Europe.

Primary endpoint by tumour type OS PFS Other Total Tumour Types N (%) Lung 34 21 55 (25) Breast 3 1 38 (18) Colorectal 14 2 30 (14) Melanoma 15 5 (10) Renal 7 10 17 (8) Prostate 12 (6) Gastric / GOJ 8 9 (4) HCC Pancreas 6 (3) Head and Neck 4 (2) Ovarian / Peritoneal 115 (53%) 97 (45%) (2%) 216 (100) The most common primary tumour types were lung, breast, colorectal and melanoma 4 trials either did not specify the endpoint or used endpoints other than PFS or OS Endpoint used was Overall Survival in 53% of trials, and Progression Free Survival in 44% Tumour types with longer expected survival times such as breast and ovarian cancer were more likely to use PFS as the primary endpoint. Trials with a shorter expected survival time such as gastric and pancreatic cancer were more likely to use OS as the primary endpoint.

Insufficient information was available to determine both the relative and absolute effect sizes hypothesised in 21% of trials Sufficient information to determine: Phase 3 trials N (%) Both relative and absolute effect size 170 (79) Relative effect size only 39 (18) Absolute effect size only (0) Neither 7 (3) Total 216 (100) Information from both publications and publically available protocols The number of trials with sufficient information to determine all critical elements of the design (relative and absolute effect sizes hypothesised, the estimated survival in the control arm, power and significance) was xx%

More extreme relative effect sizes hypothesised for PFS than OS ● PFS median HR 0.71 (IQR 0.67 to 0.75) ● OS median HR 0.75 (IQR 0.73 to 0.78) difference p < 0.0001 This graph shows the distribution of the relative effect sizes hypothesised in the phase 3 trials More extreme relative effect sizes hypothesised for trials with PFS as the primary endpoint than OS. The median relative effect size hypothesised was a HR of 0.71 for PFS and 0.75 for OS

Smaller absolute effect sizes hypothesised for PFS and OS ● PFS median 2.3 months (IQR 1.5 to 3.1) ● OS median 2.9 months (IQR 2.2 to 3.8) difference p = 0.03 This graph shows the distribution of the absolute effect sizes hypothesised in the phase 3 trials shown as the difference in median survival times. What was striking was how many of the trials hypothesised an absolute improvement of between 2 to 3 months for both OS and PFS

Hypothesised versus observed relative effect size for OS The relationship was low correlation. With sensitivity analysis, excluding outlier, correlation was 0.13, p=0.20 Explain graph: X axis Y axis HR 0.8 was the ASCO recommendation for minimum clinically meaningful benefit. Black triangles the trials that were statistically significant Grey triangles those that were not statistically significant I will come to the coloured data points shortly Three main points: 1. Just over half of the trials (56%) that were statistically significant found an observed effect size that was less extreme (closer to 1) than what was hypothesised, and were still statistically significant For example this point which had a hypothesised HR of 0.67, an observed HR of 0.75 and was still statistically significant 2. The red dots show the trials that were statistically significant, but in which the observed HR was less extreme (closer to 1) than 0.8, and so may be of uncertain clinical importance. This occurred in a quarter of trials that were statistically significant. 3. The green dots show the trials that were not statistically significant, but in which the observed HR was more extreme than 0.8 (closer to 0), and so may be of clinical importance despite not being of clinical significance.

Hypothesised versus observed relative effect size for PFS This graph shows the same relationship but for progression free survival as the primary endpoint. There was modest correlation. There are no guidelines to recommendation the minimum clinically meaningful benefit for PFS. We would usually expect to see a bigger difference in PFS than OS, so it would be reasonable to assume that it would be a HR that is more extreme than 0.8. 57% (55 of 100 ) of trials with PFS as the primary outcome had a statistically significant observed effect size Just over a third (20 (36%) ) of the statistically significant trials had an observed effect size that was less extreme (closer to 1) than what was hypothesised. For example hypothesised HR 0.8, and was statistically significant for HR 0.9 There were 4 trials that had an observed effect size where the HR was less extreme (closer to 1) than 0.8, and so may be of uncertain clinical importance.

ASCO recommended that a HR ≤ 0 ASCO recommended that a HR ≤ 0.8 should be used for phase 3 trials with a primary outcome of overall survival This was achieved by most trials: 109/111 (98%) 5 trials did not specify a target HR (5/116) 2 trials > 0.8 (one was 0.82, one was 0.85) 22 trials exactly 0.8

Absolute Effect Size (months) ASCO recommendations met for relative but not absolute effect sizes for specific disease types   Lung Cancer Pancreatic Cancer ASCO Proposal Relative Effect Size HR ≤ 0.8 Absolute Effect Size (months) ≥ 2.5 for squamous, ≥ 3.25 for non-squamous HR ≤ 0.75 ≥ 3 Met 22/23 (96%) 4/23 (17%) 5/6 (83%)  (0%) Not met 0  10/23 (43%) 1/6 4/6 (67%) Not able to assess 1/23 (4%) 9/23 (39%)  0 2/6 (33%) ASCO provided recommendations for the relative effect size and absolute effect size in 4 clinical scenarios, all of which were in the first line metastatic setting with a primary outcome of overall survival. The were not enough trials for the scenarios of breast and colon cancer, and so I have focussed on lung and pancreatic cancer. In lung cancer ASCO recommended trials should be designed for a HR of 0.8 or less and an absolute improvement in median survival of around 2.5 months for SCC, and 3.25 months for non-squamous. The majority of trials met the recommendations for relative effect size, but far fewer met the recommendations for absolute effect size. The same can be seen for trials assessing pancreatic cancer.

Conclusions Insufficient information were available to determine both the relative and absolute effect sizes hypothesised in 21% of trials Median effect sizes hypothesised were: OS: HR 0.75, 2.9 months PFS: HR 0.71, 2.3 months Many trials were designed with observed survival benefits that could be both statistically significant and clinically unimportant

Implications for phase 3 trials Trial reports and protocols should provide all critical elements of trial design Relative and absolute benefits should both be considered and specified Relative effects are best for depicting the activity of the treatments Absolute benefits are best for assessing the benefits and harms of treatment for patients Randomised phase 3 trials should be designed to provide statistically significant results for observed effects that are clinically important Relative effects are best for depicting activity of treatment Absolute benefits are best for depicting benefits and harms of treatment Randomised phase 3 trials should be designed to provide statistically significant results for observed effects that are clinically important. If phase 3 trials are designed to look for bigger differences, this will require a smaller sample size.