A Review of Methods used to Quantify Effect Sizes in Clinical Trials Joanne Rothwell University of Sheffield PSI Conference, London, May 2017

PSI Conference, London, May 2017 Outline Aims of Research Methods Results Limitations Further Work PSI Conference, London, May 2017

PSI Conference, London, May 2017 Aims of Research To investigate: which methods are commonly used and reported for eliciting the target effect size how “accurate” the target effect size is compared to the observed effect size whether particular clinical areas are consistently over- or under-achieving the target effect size examples of well-justified sample size calculations and target effect size elicitation. To be completed. PSI Conference, London, May 2017

PSI Conference, London, May 2017 Aims of Research Standard sample size formula for superiority trials Fixed Most sensitive variable As Steven mentioned Estimated PSI Conference, London, May 2017

PSI Conference, London, May 2017 Aims of Research Standard sample size formula for superiority trials Fixed If d is halved, the sample size quadruples Estimated PSI Conference, London, May 2017

PSI Conference, London, May 2017 Methods Why is this research useful? It would be interesting to know which methods are most commonly used to elicit the target difference and which are most reported. This would be useful for the DELTA2 guidance. It would be useful to know how many studies are not observing a difference close to the target difference which the study was originally powered on. PSI Conference, London, May 2017

PSI Conference, London, May 2017 Methods How was this research completed? Review the Health Technology Assessment (HTA) reports for randomised controlled trials Extract data from the reports over the period 2006-2016 HTA used because they are the main public funder in the UK PSI Conference, London, May 2017

Inclusion/Exclusion Criteria for Reports Extracted variables included: Target and observed effect size, target and achieved sample size, elicitation method (if reported), clinical area, outcome measure, target power. Exclusion based on initial title and abstract reading: systematic reviews (most common X%), literature reviews, feasibility and pilot studies, observational studies, cost-effectiveness and decision analysis. Once extraction commenced, other exclusions were required: factorial trials (no clear target difference or elicitation appeared), non-inferiority studies (no target difference, based on NI margin), equivalence studies (again, based on equivalence margin), vaccination trials (Hard to extract all relevant information from these). PSI Conference, London, May 2017

Report Selection PSI Conference, London, May 2017 994 reports (107 RCT reports since 5 reports document two RCTs in one report.)   684 reports 310 reports excluded: Published before 2006 75 reports excluded: 9 Factorial trials 5 Non-inferiority trials 10 Equivalence trials 20 Cluster Trials 19 reports not RCTs 1 Cross-over trial 11 Too difficult to extract or not enough information provided 177 reports 507 reports excluded: Not RCTs PSI Conference, London, May 2017

Total Number of Reports General Results Volume Year Total Number of Reports Included RCTs 20 2016 95 19 2015 102 18 2014 71 12 17 2013 61 11 16 2012 49 8 15 2011 45 6 14 2010 60 13 2009 62 10 2008 36 2 2007 53 3 2006 50 Total 684 107 PSI Conference, London, May 2017

Elicitation Categories 7 categories based on DELTA publication (Cook et al. 2014) [1] Anchor method Distribution method Health Economic method Opinion-seeking method Pilot-study method Review of evidence base method Standardised effect size method Jonathan will define these more formally later. I'm sure many of you have used one or a combination of these methods to elicit a target effect size. PSI Conference, London, May 2017

Study Characteristics Most common Clinical Area Mental Health (16.7%) Setting Hospital (51%) Elicitation Methods Review of Evidence Base (46%) PSI Conference, London, May 2017

PSI Conference, London, May 2017 General Results PSI Conference, London, May 2017

PSI Conference, London, May 2017 General Results PSI Conference, London, May 2017

PSI Conference, London, May 2017 General Results PSI Conference, London, May 2017

PSI Conference, London, May 2017 General Results Shows that most common method across all the clinical areas is using previous research These elicitation categories are ones which contain slightly more detail than the condensed DELTA categories. This shows that across all clinical areas previous research is most commonly used to eliciti the target difference. PSI Conference, London, May 2017

PSI Conference, London, May 2017 General Results Using DELTA categories, most common method is consistently Review of Evidence Using the DELTA categories, it is clear to see that the Review of Evidence method (using previous research) comes out on top as the most common method, particularly in Mental health. PSI Conference, London, May 2017

Estimated/Target Standardised Effect Size Smallest Target Effect sizes Largest Target Effect sizes PSI Conference, London, May 2017

Estimated/Target Standardised Effect Size Smallest Target Effect sizes Seems very large Largest Target Effect sizes PSI Conference, London, May 2017

Estimated/Target Standardised Effect Size Smallest Target Effect sizes Large variation Whilst this could be due to different interventions, it is still interesting to see. Largest Target Effect sizes PSI Conference, London, May 2017

Estimated vs Observed Standardised Effect Size General relationship - Estimated SES are higher than Observed It would be expected for the protocol to have a higher expected/target effect size than observed, as half of all publicly fundedtrials are not statistically significant. PSI Conference, London, May 2017

Estimated vs Observed Standardised Effect Size General relationship - Estimated SES are higher than Observed. As predicted PSI Conference, London, May 2017

Estimated vs Observed Standardised Effect Size Very high observed effect size - Renal/Urology study. p<0.001 PSI Conference, London, May 2017

PSI Conference, London, May 2017 Limitations Only one reviewer, 177 reports to read through and 102 to extract information from. No QA available Some reports difficult to extract from due to unclear sample size justifications In order to compare effect sizes for clinical categories, categories may need to be combined. This needs consultation with a clinical expert. Only one reviewer. PSI Conference, London, May 2017

PSI Conference, London, May 2017 Summary Mental Health is most commonly published clinical area for RCTs. Most common method of elicitation is using previous research or a combination of various methods including a review of the evidence. (49%) Target effect sizes are typically larger than observed effect sizes (this is as expected) Only one reviewer. PSI Conference, London, May 2017

PSI Conference, London, May 2017 References JA Cook, J Hislop, TE Adewuyi, K Harrild, DG Altman, CR Ramsay, C Fraser, B Buckley, P Fayers, I Harvey, AH Briggs, JDNorrie, D Fergusson, I Ford, LD Vale. Assessing methods to specify the target difference for a randomised controlled trial: DELTA (Difference ELicitation in TriAls). Health Technology Assessment 18(28), 2014 Only one reviewer. PSI Conference, London, May 2017

Thank you for your attention. ? ? ? Any Questions? ? Only one reviewer. PSI Conference, London, May 2017