Indirect and mixed treatment comparisons Hannah Buckley Co-authors: Hannah Ainsworth, Clare Heaps, Catherine Hewitt, Laura Jefferson, Natasha Mitchell, Carole Torgerson, David Torgerson

Overview Exemplar trials Direct comparisons Indirect comparisons –Methodological approaches Mixed treatment comparisons Assumptions

Exemplar trials 2 RCTs of literacy interventions EEF ‘writing bundle’ Grammar for writing (GfW) 1 Improving writing quality (IWQ) 2 Improving writing quality of struggling year 6 pupils

Grammar for writing 15 guided writing sessions over 4 weeks 53 schools Progress in English 11 Long Form Split plot design rt_-_Grammar_for_Writing_-_February_2014.pdf

Grammar for writing

Improving writing quality Self-regulated strategy development combined with memorable experiences 23 primary schools, 3 secondary school Progress in English 11 Long Form Cluster trial _Improving_Writing_Quality_-_May_2014.pdf

Direct comparison Treatment effect estimates usually from direct comparisons between two treatments in an RCT

MD = % CI: (0.00, 1.56) MD = % CI: (0.90, 4.16) MD = mean difference Direct comparisons

Indirect comparisons Used to provide estimates when evidence from direct comparisons not available Adjusted indirect comparison (IC) - common comparator required

IC methods - overview Frequentist/Bayesian approach Naïve IC (no advantages of RCT) Adjusted IC Meta-regression Generalised linear mixed models (IPD data) Confidence profile method Bayesian Markov chain Monte Carlo (MCMC)

IC methods – simple adjusted Frequentist approach Uses aggregate trial data Adjusted based on common comparison Estimates from trials extracted If more than one trial for each comparison then use a weighted combination as in meta-analysis

IC methods –adjusted

IC methods – meta-regression Frequentist approach Uses aggregate data Fixed/random effects Ө BC modelled as a function of one or more study characteristics as predictor variable(s) Co-efficient of indicator for comparison gives effect estimate

Indirect comparison Comparing GfW and IWQ

IC - example Effect estimate for B vs C: Ө BC = Ө AB – Ө AC = 2.53 – 0.78 = 1.75 TrialEffect estimateVariance of effect estimate IWQ Ө AB = 2.53var(Ө AB ) = 0.69 GfW Ө AC = 0.78var(Ө AC ) = 0.16

IC - example Variance of effect estimate for B vs C: var(Ө BC ) = var(Ө AB ) + var(Ө AC ) = = 0.85 TrialEffect estimateVariance of effect estimate IWQ Ө AB = 2.53var(Ө AB ) = 0.69 GfW Ө AC = 0.78var(Ө AC ) = 0.16

IC - example TrialEffect estimateVariance of effect estimate IWQ Ө AB = 2.53var(Ө AB ) = 0.69 GfW Ө AC = 0.78var(Ө AC ) = 0.16

IC - example No evidence of a difference in means between pupils receiving each intervention with a non-significant increase of 1.75 marks (95% CI: -0.06, 3.56) in writing score for those receiving the IWQ intervention compared with those receiving the GfW intervention

Assumptions – IC Homogeneity assumption: –  2 test –I 2 Similarity assumption in terms of effect moderators –populations should be similar in both sets of trials –participants in trial AB could have been randomised in trial AC –Same estimate would be obtained in trial ABC

Combining direct and indirect evidence Indirect evidence supplements direct evidence 1 RCT of direct evidence is as precise as indirect evidence based on 4 RCTs 3 Mixed treatment comparison

Assumptions - MTC Consistency –indirect estimate would be the same as estimate from direct evidence

Conclusions Indirect comparisons can provide of relative effectiveness MTC may provide gains in precision Methods may be particularly applicable in an education setting where BAU frequently used as a comparator Caution must be taken with interpretation

References 1.Torgerson, D., Torgerson, C., Mitchell, N., Buckley, H., Ainsworth, H., et al. (2014). Grammar for Writing Evaluation Report and Executive Summary. Published by the Education Endowment Foundation on educationendowmentfoundation.org.uk. Last accessed 09 Sep Torgerson, D., Torgerson, C., Ainsworth, H., Buckley, H., Heaps, C., et al. (2014). Improving Writing Quality Evaluation Report and Executive Summary. Published by the Education Endowment Foundation on educationendowmentfoundation.org.uk. Last accessed 09 Sep Glenny, A, D Altman, et al. (2005) Indirect Comparisons of Competing Interventions. Health Technology Assessment vol. 9, no. 26.

Resources Bucher, Heiner C. et al. (1997) The Results of Direct and Indirect Treatment Comparisons in Meta-Analysis of Randomized Controlled Trials. Journal of Clinical Epidemiology, vol. 50, no. 6: 683–91. Miladinovic, B., et al (2014). Indirect Treatment Comparison. Stata Journal vol 14, no. 1: 76–86. Jansen JP, et al (2011). Interpreting indirect treatment comparisons and network meta-analysis for health-care decision making: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 1. Value in Health vol 14, no. 4: