The DELTA2 Study: Summary of Methodology and Results Quantifying Effect Sizes William Sones Tuesday 16th May 2017 PSI 2017 Conference The Grange Tower Bridge Hotel, London

Introduction

Target difference estimation Target difference (effect size) estimation has been established as a key area of uncertainty in RCTs Although research has been performed examining techniques applied, little guidance exist for practical implementation DELTA - Guidance for a standard superiority two-arm parallel group trial (Cook at al. 2015) Research has been performed to determine techniques applied (but this may not cover best approaches)

Target difference estimation The MRC and NIHR kindly offered funding to: Determine scope and design required Write and publish guidance The DELTA2 study aims to establish guidance required and publish guidance on target difference estimation

The DELTA2 Study

The DELTA2 Study Review existing guidance UK and International funding programmes and regulatory bodies Identify key methodological developments and changes Systematic review of recent methodological developments Consensus process Delphi study Consensus meetings Draft guideline feedback Construction and dissemination of final guidelines

Review of existing guidance

Review of existing guidance Funding bodies, regulatory authorities, and advisory groups reviewed include: Arthritis Research UK, BHF, HRA, MRC, RDS, Cancer Research UK, Wellcome Trust, NIHR, FDA, Health Canada, European Commission (H2020), NIHR Statistics Group, CIHR, PCORI, NIH, NHMRC, and AHRQ Majority of sites contained little, if any guidance Guidance observed generally provided information on the role of the target difference, and little on implementation

Key methodological developments and changes

Key methodological developments and changes Systematic review was performed upon 22 Journals between the dates 2011/01/01 and 2016/03/31 1395 publications were returned using search terms: sample size, target difference, effect size, important difference, detectable difference, power calculation, value of information, value of perfect information, value of partial perfect information, value of sampling information, expected net gain Review of title and abstract reduced the number to 85 Full text investigation showed 41 publications containing information of relevance

Consensus process

Consensus process Delphi study Consensus meetings Stakeholder engagement Multiple online surveys (core stakeholders with interest and expertise in the design of RCTs) Outcome of each survey is fedback to stakeholders, influencing opinion Establish consensus (guidance scope and level of contents) 2-day consensus meeting of experts representing different aspects of RCTs Session at SCT with potential scope and content of guidance Establish scope Session at PSI presenting draft guidance

Delphi survey

Delphi survey Aimed to determine a wide opinion upon the required scope of guidance Invitation to participate was sent to: Funding panel Chairs/Directors, CTU directors, support groups (e.g. Hubs, NIHR stats group, RDS), statisticians 162 invitations were sent. 78 (48%) participated The online survey investigated topics including: Type of study, methods for specifying target difference, approaches to consider, complex designs, value of existing guidance

Delphi results - Participant details Your role in RCTs (select all that apply):

Delphi results - Participant details Your role in RCTs (select all that apply):

Delphi results - Participant details Primary RCT related affiliation:

Delphi results - Participant details Primary RCT related affiliation:

Delphi results - Participant details Where do you work? If you work across Europe or Internationally please choose the category in which the majority of your work is performed.

Delphi results - Participant details Where do you work? If you work across Europe or Internationally please choose the category in which the majority of your work is performed.

Delphi Results - Types of studies Guidance for specifying the target difference for a phase III/IV (often called definitive or confirmatory) trial needs to be dealt with separately from early phase, pilot or feasibility trials.

Delphi results - Methods for specifying target difference Should the following approaches be considered a formal method and covered within the guidance? 78% believe standardised effect size should be covered whilst 17% believe it should not 62% believe Value of Information should be covered whilst 13% believe it should not. ~A quarter adopted a neutral/no opinion for VoI, whilst only 5% did so for SES

Delphi results - Methods for specifying target difference Should the following approaches be considered a formal method and covered within the guidance? 78% believe standardised effect size should be covered whilst 17% believe it should not 62% believe Value of Information should be covered whilst 13% believe it should not. ~A quarter adopted a neutral/no opinion for VoI, whilst only 5% did so for SES

Delphi results - Special topics Degree of coverage required for the following special topics

Delphi results - Special topics Degree of coverage required for the following special topics

Delphi results - Special topics Additional comments under special topics include: Additional topics of interest E.g. absolute vs relative risks Different perspectives E.g. provider and regulatory agency perspectives

Delphi results - Complex designs Degree of coverage for trials with more complex designs

Delphi results - Complex designs Degree of coverage for trials with more complex designs

Delphi results - Existing guidance The existing paper (DELTA; Cook at al. 2015) is useful

Delphi results - Existing guidance Criticism of DELTA guidance: Guidance style (e.g. more examples/example to provide greater coverage of trials/different approach-less descriptive and more practical guidance) Increase coverage for different forms of trials Improvement of guidance within 2-arm superiority

Delphi results - Further suggestions Views covered to greater extent/importance include: More complicated designs beyond those addressed exist Wider range of guidance/Support for a wider audience Will DELTA2 provide greater support for what is wanted rather than trials performed?

Delphi survey Summary

Summary Majority of respondents had statistical experience, worked in academia, and were UK based Suggestions: Phase III/IV should be reviewed separately from early phase, pilot or feasibility Standardised effect size should be covered in guidance VoI should be covered (perhaps to a lesser extent) Alternative research questions and multiple primary outcomes deserved greater attention than other topics Support for a wider audience

Summary (cont.) Suggestions (cont.): Adaptive designs and cluster randomised trials deserved greater attention than within-subject paired and cross-over designs Existing DELTA guidance is useful

Acknowledgments The project was jointly funded by the Medical Research Council (MRC) and National Institute for Health Research (NIHR) Survey was performed using BOS, University of Bristol Email: William.sones@ndorms.ox.ac.uk