1. The DELTA2 Study: Summary of Methodology and Results
Quantifying Effect Sizes
William Sones
Tuesday 16th May 2017
PSI 2017 Conference
The Grange Tower Bridge Hotel, London

2. Introduction

3. Target difference estimation
Target difference (effect size) estimation has been established as a key area of uncertainty in RCTs
Although research has been performed examining techniques applied, little guidance exist for practical implementation
DELTA - Guidance for a standard superiority two-arm parallel group trial (Cook at al. 2015)
Research has been performed to determine techniques applied (but this may not cover best approaches)

4. Target difference estimation
The MRC and NIHR kindly offered funding to:
Determine scope and design required
Write and publish guidance
The DELTA2 study aims to establish guidance required and publish guidance on target difference estimation

5. The DELTA2 Study

6. The DELTA2 Study Review existing guidance
UK and International funding programmes and regulatory bodies
Identify key methodological developments and changes
Systematic review of recent methodological developments
Consensus process
Delphi study
Consensus meetings
Draft guideline feedback
Construction and dissemination of final guidelines

7. Review of existing guidance

8. Review of existing guidance
Funding bodies, regulatory authorities, and advisory groups reviewed include:
Arthritis Research UK, BHF, HRA, MRC, RDS, Cancer Research UK, Wellcome Trust, NIHR, FDA, Health Canada, European Commission (H2020), NIHR Statistics Group, CIHR, PCORI, NIH, NHMRC, and AHRQ
Majority of sites contained little, if any guidance
Guidance observed generally provided information on the role of the target difference, and little on implementation

9. Key methodological developments and changes

10. Key methodological developments and changes
Systematic review was performed upon 22 Journals between the dates 2011/01/01 and 2016/03/31
1395 publications were returned using search terms:
sample size, target difference, effect size, important difference, detectable difference, power calculation, value of information, value of perfect information, value of partial perfect information, value of sampling information, expected net gain
Review of title and abstract reduced the number to 85
Full text investigation showed 41 publications containing information of relevance

11. Consensus process

12. Consensus process Delphi study Consensus meetings
Stakeholder engagement
Multiple online surveys (core stakeholders with interest and expertise in the design of RCTs)
Outcome of each survey is fedback to stakeholders, influencing opinion
Establish consensus (guidance scope and level of contents)
2-day consensus meeting of experts representing different aspects of RCTs
Session at SCT with potential scope and content of guidance
Establish scope
Session at PSI presenting draft guidance

13. Delphi survey

14. Delphi survey
Aimed to determine a wide opinion upon the required scope of guidance
Invitation to participate was sent to:
Funding panel Chairs/Directors, CTU directors, support groups (e.g. Hubs, NIHR stats group, RDS), statisticians
162 invitations were sent. 78 (48%) participated
The online survey investigated topics including:
Type of study, methods for specifying target difference, approaches to consider, complex designs, value of existing guidance

15. Delphi results - Participant details
Your role in RCTs (select all that apply):

16. Delphi results - Participant details
Your role in RCTs (select all that apply):

17. Delphi results - Participant details
Primary RCT related affiliation:

18. Delphi results - Participant details
Primary RCT related affiliation:

19. Delphi results - Participant details
Where do you work? If you work across Europe or Internationally please choose the category in which the majority of your work is performed.

20. Delphi results - Participant details
Where do you work? If you work across Europe or Internationally please choose the category in which the majority of your work is performed.

21. Delphi Results - Types of studies
Guidance for specifying the target difference for a phase III/IV (often called definitive or confirmatory) trial needs to be dealt with separately from early phase, pilot or feasibility trials.

22. Delphi results - Methods for specifying target difference
Should the following approaches be considered a formal method and covered within the guidance?
78% believe standardised effect size should be covered whilst 17% believe it should not
62% believe Value of Information should be covered whilst 13% believe it should not.
~A quarter adopted a neutral/no opinion for VoI, whilst only 5% did so for SES

23. Delphi results - Methods for specifying target difference
Should the following approaches be considered a formal method and covered within the guidance?
78% believe standardised effect size should be covered whilst 17% believe it should not
62% believe Value of Information should be covered whilst 13% believe it should not.
~A quarter adopted a neutral/no opinion for VoI, whilst only 5% did so for SES

24. Delphi results - Special topics
Degree of coverage required for the following special topics

25. Delphi results - Special topics
Degree of coverage required for the following special topics

26. Delphi results - Special topics
Additional comments under special topics include:
Additional topics of interest
E.g. absolute vs relative risks
Different perspectives
E.g. provider and regulatory agency perspectives

27. Delphi results - Complex designs
Degree of coverage for trials with more complex designs

28. Delphi results - Complex designs
Degree of coverage for trials with more complex designs

29. Delphi results - Existing guidance
The existing paper (DELTA; Cook at al. 2015) is useful

30. Delphi results - Existing guidance
Criticism of DELTA guidance:
Guidance style (e.g. more examples/example to provide greater coverage of trials/different approach-less descriptive and more practical guidance)
Increase coverage for different forms of trials
Improvement of guidance within 2-arm superiority

31. Delphi results - Further suggestions
Views covered to greater extent/importance include:
More complicated designs beyond those addressed exist
Wider range of guidance/Support for a wider audience
Will DELTA2 provide greater support for what is wanted rather than trials performed?

32. Delphi survey
Summary

33. Summary
Majority of respondents had statistical experience, worked in academia, and were UK based
Suggestions:
Phase III/IV should be reviewed separately from early phase, pilot or feasibility
Standardised effect size should be covered in guidance
VoI should be covered (perhaps to a lesser extent)
Alternative research questions and multiple primary outcomes deserved greater attention than other topics
Support for a wider audience

34. Summary (cont.) Suggestions (cont.):
Adaptive designs and cluster randomised trials deserved greater attention than within-subject paired and cross-over designs
Existing DELTA guidance is useful

35. Acknowledgments
The project was jointly funded by the Medical Research Council (MRC) and National Institute for Health Research (NIHR)
Survey was performed using BOS, University of Bristol