The potential role of mixed treatment comparisons Deborah Caldwell Tony Ades MRC HSRC University of Bristol

Outline of presentation Indirect comparisons and mixed treatment comparisons (MTC). Potential concerns regarding use of indirect comparisons/ MTC. Hypothetical ‘simulation’ example of MTC evidence structure. MTC of NICE appraisal for thrombolysis Addressing potential concerns Should MTC be routine and future areas of research.

Background For any given condition there are an array of possible interventions/ treatments. Treatment recommendations & decisions should be evidence based. Principle sources are systematic reviews of randomised controlled trials. Systematic reviews focus on pairwise, direct comparisons of treatments.

Indirect comparisons In absence of trials comparing treatments A versus B, an indirect estimate of odds ratio d AB is obtained from RCTs comparing A vs C and B vs C: d AB = d AC – d BC

Indirect comparisons In absence of trials comparing treatments A versus B, an indirect estimate of odds ratio d AB is obtained from RCTs comparing A vs C and B vs C: d AB = d AC – d BC

Indirect comparisons In absence of trials comparing treatments A versus B, an indirect estimate of odds ratio d AB is obtained from RCTs comparing A vs C and B vs C: d AB = d AC – d BC

Mixed treatment comparisons Where there are 3 or more treatments, compared using direct and indirect evidence from several RCTs = mixed (multiple) treatment comparisons (MTC). MTC evidence structures are pervasive in Health Technology Assessments (HTA) – decisions between >5 treatments are commonplace. A unified, coherent analysis of multiple treatments can only be achieved by including the entire evidence structure of relevant RCTs.

Potential concerns about MTC Indirect comparisons produce relatively imprecise estimates of treatment effect They are not randomised comparisons Suffer the biases of observational studies (level 3 of EBM evidence hierarchies?) Direct and indirect evidence should be considered separately. Direct evidence should take precedence.

Contradictions in the ‘received wisdom’ Why is lower level evidence used when level one is unavailable but it is irrelevant when it isn’t? What do we do when direct evidence is inconclusive but in combination with indirect is conclusive? If 5 treatments are all compared with each other does it make sense to separate the 10 direct pairwise comparisons from the 70 indirect?

Hypothetical evidence structure

Objectives ‘Simulation’ exercise to explore benefit of increasing levels of complexity in MTC evidence structures. –To examine additional benefit of including evidence routinely excluded from systematic reviews. –To what extent different MTC evidence structures give increasing levels of precision. –Address some of the concerns outlined.

Method Contrast estimates of posterior precision of log odds ratios (LOR) from i.A standard pairwise meta-analysis ii.Use of mixed treatment comparison analysis Compare estimates of posterior precision i.LOR of treatment A vs. treatment B ii.‘Average’ precision - across all 15 possible treatment comparisons. Assumptions: Equal amounts of information on each treatment comparison.

Simulation results: precision of d AB Precision of pairwise d AB = 1 Precision of MTC d AB = 1.01

Hypothetical evidence structure

Simulation results: precision of d AB Additional data on a single indirect comparison increases precision by 0.51

Simulation results: precision of d AB Each additional indirect treatment comparison increases precision in d AB by 0.5

Simulation results: precision of d AB Is there value in ‘linking’ indirect comparisons? ‘Linking’ comparison is treatment C vs D

Simulation results: precision of d AB Adding ‘linking’ comparisons doesn’t increase precision of d AB estimate. Property of this particular evidence structure

Summary of simulation results: precision of d AB If all you believe is ‘direct’ data –precision d AB stays = 1 Mixed Treatment Comparison analysis – Adding data on a single indirect comparison increases precision by 0.51 –Adding multiple indirect comparisons further increases precision –Equivalent to 2 extra trials on d AB comparison

Simulation results: ‘average’ precision

5 pieces of data/ 15 possible treatment comparisons.

Simulation results: ‘average’ precision Maximum ‘average’ pairwise precision = pieces of data/ 15 possible treatment comparisons

Simulation results: ‘average’ precision MTC allows us to say something about all 15 pairwise comparisons.

Simulation results: ‘average’ precision Equivalent number of trials 0.67*15 = 10 trials worth of data

Early thrombolysis for acute myocardial infarction (AMI).

Technology appraisal for National Institute for Clinical Excellence (Boland et al, 2003) Affects 274,000 people each year 50% die within 30 days of AMI. National Service Framework for heart disease states thrombolysis should be given within 60 minutes. Thrombolysis = pharmaceutical agents to dissolve blood clots.

Thrombolytic treatments Four treatments assessed: –Streptokinase (SK), –Tissue-plasminogen activator (t-PA), –Tenecteplase (TNK) –Reteplase (r-PA). Distinction made between accelerated t-PA and standard t-PA. SK + t-PA used in two trials

Thrombolysis conclusions (Boland et al, 2003) “Definitive (sic) conclusions on efficacy are that streptokinase is as effective as non-accelerated alteplase, that tenecteplase is as effective as accelerated alteplase, and that reteplase is at least as effective as streptokinase. “Some conclusions require interpretation of data, i.e. whether streptokinase is as effective as, or inferior to accelerated alteplase; and whether reteplase is as effective as accelerated alteplase or not. “Depending on these, two further questions on indirect comparisons arise, whether tenecteplase is superior to as streptokinase or not and whether reteplase is as effective as tenecteplase or not.”

What is needed? A single statistical analysis providing estimates for all the 15 pairwise comparisons, between 6 treatments. –Using classical or Bayesian statistical methods. An assessment of which of these treatments is most likely to be best. Method –Bayesian Markov chain Monte Carlo method

Were all relevant treatments included? Primary percutaneous transluminal coronary angioplasty (PCTA). Keeley et al meta-analysis of PCTA vs thrombolysis (22 RCTs) –PCTA is better than thrombolysis (OR 0.70 [0.58 – 0.85]) –But surely the relevant comparison is the ‘best’ thrombolytic NOT the ‘average’ one? 7 treatments 21 possible pairwise comparisons

Extended evidence structure

Consistency of odds ratios and CIs: fixed effect analysis

Lumping vs splitting: Fixed effect analysis of At-PA versus PCTA

Breaking randomisation? NO! –There are statistically invalid methods. –Our MTC analyses are based only on randomised comparisons. –Lack of assumptions about baseline risks across studies. –A weighted combination of valid estimates of treatment effect.

Generalisability Key assumption in MTC is that relative treatment effect of one treatment vs another is same across entire set of trials. Irrespective of which treatments were actually evaluated in each trial –True odds ratio of A vs B trials is exactly the same as the A vs B odds ratio in the A vs C, B vs C trials. (fixed effect) –Common distribution of treatment effects is the same across all sets of trials (random effects).

Generalisability Helpful to consider which target population we are making treatment recommendation for. –The type of patients in the previous A vs B trials? OR – The kind of patients in ALL the MTC trials? Clinical and epidemiological judgement necessary Poor judgements may introduce heterogeneity

Should MTC be routine? A more appropriate question is can MTC analyses be avoided? –No real alternative in multi-treatment decision making. Transparency –No need to lump treatments –No ‘under the table’ indirect comparisons MTC same assumptions as meta-analysis

Future areas of research What is the extra literature searching burden of MTC – how far should searches go? –Should we include discontinued treatments in the evidence base? –Placebo controlled trials? Greater awareness of MTC by commissioners of research when ‘scoping’ HTAs –NICE Obesity appraisals –Thrombolysis & PCTA