1. Indirect and mixed treatment comparisons Hannah Buckley Co-authors: Hannah Ainsworth, Clare Heaps, Catherine Hewitt, Laura Jefferson, Natasha Mitchell, Carole Torgerson, David Torgerson

2. Overview Exemplar trials Direct comparisons Indirect comparisons –Methodological approaches Mixed treatment comparisons Assumptions

3. Exemplar trials 2 RCTs of literacy interventions EEF ‘writing bundle’ Grammar for writing (GfW) 1 Improving writing quality (IWQ) 2 Improving writing quality of struggling year 6 pupils

4. Grammar for writing 15 guided writing sessions over 4 weeks 53 schools Progress in English 11 Long Form Split plot design https://educationendowmentfoundation.org.uk/uploads/pdf/FINAL_EEF_Evaluation_Repo rt_-_Grammar_for_Writing_-_February_2014.pdf

5. Grammar for writing

7. Improving writing quality Self-regulated strategy development combined with memorable experiences 23 primary schools, 3 secondary school Progress in English 11 Long Form Cluster trial https://educationendowmentfoundation.org.uk/uploads/pdf/EEF_Evaluation_Report_- _Improving_Writing_Quality_-_May_2014.pdf

8. Direct comparison Treatment effect estimates usually from direct comparisons between two treatments in an RCT

9. MD = 0.78 95% CI: (0.00, 1.56) MD = 2.53 95% CI: (0.90, 4.16) MD = mean difference Direct comparisons

10. Indirect comparisons Used to provide estimates when evidence from direct comparisons not available Adjusted indirect comparison (IC) - common comparator required

11. IC methods - overview Frequentist/Bayesian approach Naïve IC (no advantages of RCT) Adjusted IC Meta-regression Generalised linear mixed models (IPD data) Confidence profile method Bayesian Markov chain Monte Carlo (MCMC)

12. IC methods – simple adjusted Frequentist approach Uses aggregate trial data Adjusted based on common comparison Estimates from trials extracted If more than one trial for each comparison then use a weighted combination as in meta-analysis

13. IC methods –adjusted

14. IC methods – meta-regression Frequentist approach Uses aggregate data Fixed/random effects Ө BC modelled as a function of one or more study characteristics as predictor variable(s) Co-efficient of indicator for comparison gives effect estimate

15. Indirect comparison Comparing GfW and IWQ

16. IC - example Effect estimate for B vs C: Ө BC = Ө AB – Ө AC = 2.53 – 0.78 = 1.75 TrialEffect estimateVariance of effect estimate IWQ Ө AB = 2.53var(Ө AB ) = 0.69 GfW Ө AC = 0.78var(Ө AC ) = 0.16

17. IC - example Variance of effect estimate for B vs C: var(Ө BC ) = var(Ө AB ) + var(Ө AC ) = 0.69 + 0.16 = 0.85 TrialEffect estimateVariance of effect estimate IWQ Ө AB = 2.53var(Ө AB ) = 0.69 GfW Ө AC = 0.78var(Ө AC ) = 0.16

18. IC - example TrialEffect estimateVariance of effect estimate IWQ Ө AB = 2.53var(Ө AB ) = 0.69 GfW Ө AC = 0.78var(Ө AC ) = 0.16

19. IC - example No evidence of a difference in means between pupils receiving each intervention with a non-significant increase of 1.75 marks (95% CI: -0.06, 3.56) in writing score for those receiving the IWQ intervention compared with those receiving the GfW intervention

20. Assumptions – IC Homogeneity assumption: –  2 test –I 2 Similarity assumption in terms of effect moderators –populations should be similar in both sets of trials –participants in trial AB could have been randomised in trial AC –Same estimate would be obtained in trial ABC

21. Combining direct and indirect evidence Indirect evidence supplements direct evidence 1 RCT of direct evidence is as precise as indirect evidence based on 4 RCTs 3 Mixed treatment comparison

22. Assumptions - MTC Consistency –indirect estimate would be the same as estimate from direct evidence

23. Conclusions Indirect comparisons can provide of relative effectiveness MTC may provide gains in precision Methods may be particularly applicable in an education setting where BAU frequently used as a comparator Caution must be taken with interpretation

24. References 1.Torgerson, D., Torgerson, C., Mitchell, N., Buckley, H., Ainsworth, H., et al. (2014). Grammar for Writing Evaluation Report and Executive Summary. Published by the Education Endowment Foundation on educationendowmentfoundation.org.uk. Last accessed 09 Sep 2015. 2.Torgerson, D., Torgerson, C., Ainsworth, H., Buckley, H., Heaps, C., et al. (2014). Improving Writing Quality Evaluation Report and Executive Summary. Published by the Education Endowment Foundation on educationendowmentfoundation.org.uk. Last accessed 09 Sep 2015. 3.Glenny, A, D Altman, et al. (2005) Indirect Comparisons of Competing Interventions. Health Technology Assessment vol. 9, no. 26.

25. Resources Bucher, Heiner C. et al. (1997) The Results of Direct and Indirect Treatment Comparisons in Meta-Analysis of Randomized Controlled Trials. Journal of Clinical Epidemiology, vol. 50, no. 6: 683–91. Miladinovic, B., et al (2014). Indirect Treatment Comparison. Stata Journal vol 14, no. 1: 76–86. Jansen JP, et al (2011). Interpreting indirect treatment comparisons and network meta-analysis for health-care decision making: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 1. Value in Health vol 14, no. 4: 417-28.