1. Network Meta-analysis
What is a network meta-analysis?
How does NMA generate effect estimates
Determining credibility of NMA

2. Comparing Multiple Treatments: Introduction to Network Meta-Analyses
Many disease areas where many alternatives exist
Clinicians/patients need to know about relative merits
Impractical to test each comparator directly
Simultaneous comparison of multiple treatments
“Network meta-analysis”, “mixed treatment comparisons”, “adjusted indirect comparisons”
In medicine today, clinicians face many instances when they can use one of three or more interventions and in which there are few direct comparisons of those interventions. It would be attractive under these circumstances if we could make simultaneous comparisons to see which of these treatments are best. There are a variety of names given to such simultaneous comparisons, but the term network meta-analysis will be the one I’ll use here.

3. Conventional meta-analysis Pooled Estimate assumption
Single best estimate of treatment effect
Patients
Interventions
Outcomes
Methodology
Assumes effect similar across
A meta-analysis generates a single best estimate for each outcome.
The assumption of the meta-analysis is that the effect of treatment is similar across the patients enrolled in the eligible trials, the interventions administered, they way the outcomes were measured, and the study design and methods of each study.
One might refer to this assumption as the homogeneity assumptions
“Homogeneity assumption”

4. Relative Risk with 95% CI for Vitamin D Non-vertebral Fractures
Favors Vitamin D Favors Control '
0.1 1 10
Chapuy et al, (1994) 0.79 (0.69, 0.92)
Lips et al, (1996) 1.10 (0.87, 1.39)
Dawson-Hughes et al, (1997) 0.46 (0.24, 0.88)
Pfeifer et al, (2000) 0.48 (0.13, 1.78)
Meyer et al, (2002) 0.92 (0.68, 1.24)
Chapuy et al, (2002) 0.85 (0.64, 1.13)
Seems to be a problem with the visual here
This slide depicts the results of a systematic review and meta-analysis addressing the impact of Vitamin D on non-vertebral fractures. One might ask the question: to what extent does this study meet the homogeneity assumption? To answer this question we might first consider the extent to which the point estimates are similar. We can see that all but two of the estimates are on the benefit side, but there is a substantial difference between the largest estimate favoring treatment and the single estimate favoring control. Next, we might look at the extent to which the confidence intervals overlap. They do overlap to a considerable extent, but the confidence intervals of the first and second studies are almost completely non-overlapping. Third, one could look at the statistical test for heterogeneity which asks the question: if the underlying effect were the same across studies, how often – were the trials repeated over and over – would we see variability in results as great or greater, simply by chance, than we do here. In this case, the answer is: 5% of the time. Finally, we can use a statistic called I squared which varies between 0% and 100%. An I squared of 0 means that the differences between studies are easily explained by chance. As the variability between studies gets larger, the I squared gets larger. I square values under 25% suggest results are reasonably consistent across studies. Between 25 and 50% we might begin to question the homogeneity assumption. Over 50% we start to have serious concerns about the homogeneity assumption, and over 75% we definitely have serious concerns about the homogeneity assumptions. Here, the I square is 53%.
Trivedi et al, (2003) 0.67 (0.46, 0.99)
Pooled Random Effect Model
0.82 (0.69 to 0.98)
p= 0.05 for heterogeneity, I2=53%
Relative Risk 95% CI

5. Indirect Comparisons Less confidence than direct? Why?
Interested in A versus B
available data A vs C, B vs C
Alendronate (A)
Risedronate (B)
Placebo (C)
Up to now, we have focused on direct comparisons. That is, we are interested in A versus B, and A versus B have been directly compared. We may, however, be interested in A versus B but only have randomized trials of A versus C and B versus C. For instance, we may wish to choose between two bisphosphanates given for osteoporosis, Alendronate and Risedronate. Ideally we would have direct head-to-head comparisons between the two. This may, however, not be available. But if both Alendronate and Risedronate have been compared to placebo, we could make an indirect comparison that would provide some information on their relative benefit. For example, as is in fact the case, if Alendronate shows a 50% relative risk reduction in fracture and risedronate 35%, one might be tempted to conclude that alendronate is superior. Such indirect comparisons, are however, weaker – some might argue very much weaker – than direct or head-to-head comparisons.
Less confidence than direct?
Why?

6. Vulnerability of Indirect comparison
Effect modifiers
Patients
Optimal interventions
Comparator
Cointerventions
Outcome measures
Risk of bias

7. Combine direct and indirect comparisons
- additional assumption mediators same in direct and indirect
- “consistency” or “coherence” assumption
Fundamental idea of network meta-analysis is to combine direct and indirect comparison
additional assumptions to “heterogeneity” and “similarity” assumptions is the “consistency assumption” that direct and indirect comparisons are measuring same effect

8. This is an example of a comparisons between two treatments for smoking addiction
buproprion’s nine trials I squared 54% means homogeneity assumption in question
direct comparison suggests benefit from buproprion
indirect comparisons suggests little or no effect
combined CI wide because of disagreement between direct and indirect (violation of consistency assumption)

9. Network Meta-analysis Case Study: Which Approach to Nicotine Addiction Works Best
I will employ a case study to illustrate how network meta-analysis is conducted and its limitations.
-This slide shows evidence available for treatments for nicotine addiction, nicotine replacement therapy, antidepressants, varenicline, antidepressants, and combination
We have direct evidence about combined NRT vs NRT, NRT and antidepressants against NRT, antidepressants against NRT, NRT and antidepressants against antidepressants, and Varenicline against antidepressants
But it would be very nice if we could know about all possible comparisons and which is the best treatment

10. Network Meta-analysis Case Study
Combines effect estimates from direct and indirect comparisons
Placebo
Nicotine replacement treatment (NRT)
Varenicline
We can get at all comparisons by use of indirect comparisons
black represents direct comparisons, red indirect
so we can, for instance, get indirect information about vareneline versus NRT by looking at the two drugs relative effect in comparison to placebo, and the two drugs relative effect against the combination of NRT and antidepressants
Antidepressants + NRT
Antidepressants 10

11. Comparison
with
Treatments
NRT
1.01
(0.88 -
1.15) .5 1 2
.75
1.33
Odds ratio 4
Antidepressants
NRT + NRT
1.35
(1.04
1.75)
1.34
(1.00
1.78)
Varenicline
1.17
(0.98
1.39)
1.16
(0.97
1.38)
NRT +
antidepressant
1.30
1.73)
1.29
(0.96
1.74)
1.12
(0.81
1.55)
(0.85
1.58)
1.04
(0.71
1.52)
- This slide shows that with the use of the indirect comparisons we now have estimates of the effect of all our treatments and combinations against one another

12. Direct Comparison 1.85, I2=13% 67 comparisons NRT control 1.88, I2=19%
NRT+NRT
buspirone
1.12
1 comparison
1.54, I2=46%
5 comparisons
rimonabant
0.73, I2=0%
2 comparisons
1.85, I2=13%
67 comparisons
1.28, I2=0%
3 comparisons
NRT
1.36, I2=0%
2 comparisons
control
1.63, I2=0%
4 comparisons
1.14, I2=63%
6 comparisons
4.85
1 comparison
antidepressants
+NRT
clonidine
This slide focuses on the comparison of NRT versus antidepressants
3 trials have directly compared the two and suggest antidepressants may be superior (OR 1.34), though CI very wide
but if you look at NRT versus control and antidepressants against control the effects are almost identical
thus, the indirect evidence suggests the treatments are almost identical in their effects (OR 1.01)
1.88, I2=19%
29 comparisons
2.68, I2=82%
5 comparisons
varenicline
Direct evidence (3 trials)
1.28
1 comparison
Antide-
pressants
1.34 (0.71, 2.56)
I-squared=43.7%
1.70, I2=0%
3 comparisons .5 1 1 1 . 5 2
2.5
NRT superior
Antidepressants
superior

13. Indirect Comparison 1 1.85, I2=13% 67 comparisons NRT control
NRT+NRT
buspirone
1.12
1 comparison
1.54, I2=46%
5 comparisons
rimonabant
0.73, I2=0%
2 comparisons
1.85, I2=13%
67 comparisons
1.28, I2=0%
3 comparisons
NRT
1.36, I2=0%
2 comparisons
control
1.63, I2=0%
4 comparisons
1.14, I2=63%
6 comparisons
4.85
1 comparison
antidepressants
+NRT
clonidine
Indirect evidence
This slide focuses on the comparison of NRT versus antidepressants
3 trials have directly compared the two and suggest antidepressants may be superior (OR 1.34), though CI very wide
but if you look at NRT versus control and antidepressants against control the effects are almost identical
thus, the indirect evidence suggests the treatments are almost identical in their effects (OR 1.01)
1.88, I2=19%
29 comparisons
2.68, I2=82%
5 comparisons
1.01 (0.81,1.27)
varenicline
Direct evidence (3 trials)
1.28
1 comparison
Antide-
pressants
1.34 (0.71, 2.56)
I-squared=43.7%
1.70, I2=0%
3 comparisons .5 1 1 1 . 5 2
2.5
NRTsuperior
Antidepressants
superior

14. Indirect Comparison 2 1.85, I2=13% 67 comparisons NRT control
NRT+NRT
buspirone
1.12
1 comparison
1.54, I2=46%
5 comparisons
rimonabant
0.73, I2=0%
2 comparisons
1.85, I2=13%
67 comparisons
1.28, I2=0%
3 comparisons
NRT
1.36, I2=0%
2 comparisons
control
1.63, I2=0%
4 comparisons
1.14, I2=63%
6 comparisons
1.14, I2=63%
6 comparisons
4.85
1 comparison
antidepressants
+NRT
clonidine
2.68, I2=82%
5 comparisons
Another way to get at the relative effect of NRT vs antidepressants is an even more indirect “loop”
We see that NRT is superior to control (OR 1.85), and that Varenicline is even better against control (OR 2.68) suggesting modest superiority of Varenicline over NRT
but the direct comparison of varenicline versus antidepressants suggests varenicline is a lot better
thus the indirect evidence here suggestions that antidepressants are inferior to NRT (OR 0.88)
antidepressants
+NRT
Indirect evidence
1.88, I2=19%
29 comparisons
0.85 (0.38, 1.92)
Direct evidence (3 trials)
1.28
1 comparison
Antide-
pressants
1.34 (0.71, 2.56)
I-squared=43.7%
varenicline .5 1 1 1 . 5 2
2.5
1.70, I2=0%
3 comparisons
NRTsuperior
Antidepressants
superior

15. 5 Paths to Indirectly Compare Antidepressants vs NRT1
NRT+NRT
buspirone 2 3
1 comparison 4
5 comparisons
rimonabant
2 comparisons 5
NRT
3 comparisons
67 comparisons
placebo and
nonplacebo control
2 comparisons
3 comparisons
4 comparisons
6 comparisons
clonidine
1 comparison
There are actually 5 ways of getting indirect comparisons of antidepressants versus NRT
the most direct is the one I showed you first, which looks at both drugs against control (in yellow)
another is the one I’ve just shown you, which also involves varenicline (in red)
but another (light blue) also involves the combination of NRT and NRT
any time you can link up treatments to a common comparator you can create an indirect comparison
antidepressants
+NRT
5 comparisons
29 comparisons
varenicline
antidepressants
1 comparison
3 comparisons

16. 5 Paths to Indirectly Compare Antidepressants vs NRT1
1.01 (0.81, 1.27)
NRT+NRT
buspirone 2
0.85 (0.38, 1.92) 3
0.89 (0.29, 2.77)
1 comparison 4
1.56 (0.54, 4.49)
5 comparisons
rimonabant
2 comparisons 5
1.31 (0.25, 6.76)
NRT
3 comparisons
67 comparisons
placebo and
nonplacebo control
2 comparisons
3 comparisons
4 comparisons
6 comparisons
clonidine
1 comparison
The indirect comparisons all generate their own estimate of the relative effect of antidepressants against NRT
the most direct of the indirect suggests the treatments have similar effectiveness (yellow)
the red and green loops suggest that NRT is superior, while the dark and light blue lops suggest that antidepressants are superior
antidepressants
+NRT
5 comparisons
29 comparisons
varenicline
antidepressants
1 comparison
3 comparisons

17. Antidepressants superior Antidepressants superior
Comparative effectiveness of NRT vs. Antidepressants on prolonged abstinence (≥6 months)
Indirect evidence
NRT+NRT
buspirone
Path
1.12
1 comparison 1
1.01 (0.81, 1.27)
1.54, I2=46%
5 comparisons 2
0.73, I2=0%
2 comparisons
0.85 (0.38, 1.92)
1.85, I2=13%
67 comparisons
1.28, I2=0%
3 comparisons 3
0.89 (0.29, 2.77)
rimonabant 4
1.56 (0.54, 4.49)
NRT
1.36, I2=0%
2 comparisons 5
control
1.31 (0.25, 6.76)
Direct evidence
1.34, I2=44%
3 comparisons .5 1 1
1.5 2
2.5
1.63, I2=0%
4 comparisons
1.14, I2=63%
6 comparisons
NRT superior
Antidepressants superior
4.85
1 comparison
2.68, I2=82%
5 comparisons
This slide shows the direct evidence suggesting antidepressants are superior but with a very wide confidence interval. Because of the wide CI, the direct comparison will get only modest weight in the network meta-analysis
the most direct of the indirect comparisons has a very narrow confidence interval because there are lots of trials. This will receive most weight in the meta-analysis
the more indirect comparisons have wider confidence intervals, in large part because they are more indirect. They will therefore receive less weight in the meta-analysis
1.88, I2=19%
29 comparisons
clonidine
3 trials pooled
1.28
1 comparison
1.34 (0.71, 2.56)
Antide-
pressants
varenicline
I-squared = 43.7%
1.70, I2=0%
3 comparisons .5 1 1
1.5 2
2.5
NRT superior
Antidepressants superior

18. Antidepressants superior Antidepressants superior
Comparative effectiveness of NRT vs. Antidepressants on prolonged abstinence (≥6 months)
Indirect evidence
NRT+NRT
buspirone
Path
1.12
1 comparison 1
1.01 (0.81, 1.27)
1.54, I2=46%
5 comparisons 2
0.73, I2=0%
2 comparisons
0.85 (0.38, 1.92)
1.85, I2=13%
67 comparisons
1.28, I2=0%
3 comparisons 3
0.89 (0.29, 2.77)
rimonabant 4
1.56 (0.54, 4.49)
NRT
1.36, I2=0%
2 comparisons 5
control
1.31 (0.25, 6.76)
Direct evidence
1.34, I2=44%
3 comparisons .5 1 1
1.5 2
2.5
1.63, I2=0%
4 comparisons
1.14, I2=63%
6 comparisons
NRT superior
Antidepressants superior
4.85
1 comparison
2.68, I2=82%
5 comparisons
- The combined estimate using evidence from all the indirect comparisons suggests that antidpressants and NRT are virtually identical in their effect (OR 0.98), and is associated with a very narrow CI
1.88, I2=19%
29 comparisons
clonidine
3 trials pooled
0.98
(95% )
1.28
1 comparison
1.34 (0.71, 2.56)
Antide-
pressants
varenicline
I-squared = 43.7%
1.70, I2=0%
3 comparisons .5 1 1
1.5 2
2.5
NRT superior
Antidepressants superior

19. Antidepressants superior Antidepressants superior
Comparative effectiveness of NRT vs. Antidepressants on prolonged abstinence (≥6 months)
Indirect evidence
pooled estimate
1.01
(95% )
NRT+NRT
buspirone
Path
1.12
1 comparison 1
1.01 (0.81, 1.27)
1.54, I2=46%
5 comparisons 2
0.73, I2=0%
2 comparisons
0.85 (0.38, 1.92)
1.85, I2=13%
67 comparisons
1.28, I2=0%
3 comparisons 3
0.89 (0.29, 2.77)
rimonabant 4
1.56 (0.54, 4.49)
NRT
1.36, I2=0%
2 comparisons 5
control
1.31 (0.25, 6.76)
Direct evidence
1.34, I2=44%
3 comparisons .5 1 1
1.5 2
2.5
1.63, I2=0%
4 comparisons
1.14, I2=63%
6 comparisons
NRT superior
Antidepressants superior
4.85
1 comparison
2.68, I2=82%
5 comparisons
The pooled estimate combining the direct and indirect estimates also suggests the treatments are virtually identical in their effects (OR 1.01) and is also associated with a narrow CI
1.88, I2=19%
29 comparisons
clonidine
3 trials pooled
0.98
(95% )
1.28
1 comparison
1.34 (0.71, 2.56)
Antide-
pressants
varenicline
I-squared = 43.7%
1.70, I2=0%
3 comparisons .5 1 1
1.5 2
2.5
NRT superior
Antidepressants superior

20. Ranking of intereventions
this slide depicts the likely rankings for both efficacy and acceptability for the drugs
note that escitalopram appears the best
it is most likely to be first in terms of acceptability, and very unlikely to be worse than fourth
it is most likely to be second in terms of efficacy and unlikely to be worse than fourth

21. Clinical Scenario
You are seeing a 45-year-old patient for whom, 6 weeks previously, you have prescribed paroxetine, a selective serotonin reuptake inhibitor (SSRI), for treatment of generalized anxiety disorder (GAD). The patient reports reduced anxiety, but also reduced interest in sex, and insomnia.
Could you offer other options to the patient?

22. Relevant Network MA
Baldwin D, Woods R, Lawson R, Taylor D. Efficacy of drug treatments for generalised anxiety disorder: systematic review and meta-analysis. BMJ. 2011;342:d1199.

23. WHAT WAS THE AMOUNT OF EVIDENCE IN THE NETWORK?

24. Were the review methods credible?
Did the review explicitly address a sensible clinical question?
population variability
intervention variability (dose)
Intervention comprehensiveness
possible gain with obsolete or uninteresting interventions
outcome variability (duration of follow-up)

25. Were the review methods credible?
Was the search for relevant studies exhaustive?
Was the selection and assessment of studies reproducible?

26. What about our scenario?
Did the authors define a sensible clinical questions?
P, GAD patients; I and C, available drugs; O, efficacy and tolerability.
Was the search exhaustive?
Search was exhaustive, but did not include RCTs of reboxetine, buspirone, or alprazolam
Eligibility assessment done by single individual with 10% of articles another reviewer

28. Confidence in estimates
what should you look for?
risk of bias
inconsistency
indirectness
imprecision
publication bias

29. Risk of bias what should you look for? concealment of randomization
blinding
loss to follow-up
no risk of bias assessment

30. Precision

31. Consistency two sources – what are they? homogeneity assumption
heterogeneity: what should you look for?
no forest plots, no reports of tests for pair-wise heterogeneity
coherence: no statistically significant differences between direct and indirect
satisfactory?

33. INCOHERENCE

34. Precision

35. Indirectness sources of indirectness? no apparent problems population
intervention
outcomes

36. Publication bias how can we judge? not enough studies for funnel plot
no search through regulatory authority information

38. Confidence in estimates
risk of bias?
precision?
consistency?
heterogeneity
incoherence
directness?
publication bias?

39. Presentation of Results
Often presented with rankings
Potentially very misleading
Small difference between ranks
Everything low or very low confidence
First ranked lower confidence than others

41. OR (95% confidence interval) Direct evidence confidence in estimates
Comparison
Direct evidence
OR (95% confidence interval)
Direct evidence confidence in estimates
Indirect evidence OR (95% credible interval)
Indirect evidence confidence in estimates
Network
OR (95% credible interval)
Network confidence in estimates
Teriparatide vs. Placebo
---
0.42 ( )
very low 3, 6
very low
Zoledronate vs. Placebo
---
0.50 ( )
high
0.50 ( )
Risedronate vs. Placebo
0.17 (0.05 to 0.59)
moderate 1
0.54 ( )
low 6
0.48 ( )
moderate

42. Conclusions: buyer beware
Network meta-analysis
here to stay, can be very helpful
problems common
neglect of risk of bias
paucity of direct comparisons, limited evidence
If includes confidence in estimates for each comparison interpretable; if confidence moderate or high for some very helpful