Learning Programs to Accelerate the BioPharma Transition Network Meta-analysis What is a network meta-analysis? GRADE approach to confidence in estimates Determining credibility of NMA

Learning Programs to Accelerate the BioPharma Transition Comparing Multiple Treatments: Introduction to Network Meta-Analyses Many disease areas where many alternatives exist Clinicians/patients need to know about relative merits Impractical to test each comparator directly Simultaneous comparison of multiple treatments “Network meta-analysis”, “mixed treatment comparisons”, “adjusted indirect comparisons”

Learning Programs to Accelerate the BioPharma Transition Conventional meta-analysis Pooled Estimate assumption PatientsInterventionsOutcomesMethodology Assumes effect similar across “Homogeneity assumption” Single best estimate of treatment effect

Learning Programs to Accelerate the BioPharma Transition Relative Risk with 95% CI for Vitamin D Non-vertebral Fractures Chapuy et al, (2002) 0.85 (0.64, 1.13) Pooled Random Effect Model 0.82 (0.69 to 0.98) p= 0.05 for heterogeneity, I 2 =53% Chapuy et al, (1994) 0.79 (0.69, 0.92) Lips et al, (1996) 1.10 (0.87, 1.39) Dawson-Hughes et al, (1997) 0.46 (0.24, 0.88) Pfeifer et al, (2000) 0.48 (0.13, 1.78) Meyer et al, (2002) 0.92 (0.68, 1.24) Trivedi et al, (2003) 0.67 (0.46, 0.99) Favors Vitamin D Favors Control Relative Risk 95% CI ' ' ' ' ' ' ' '

Learning Programs to Accelerate the BioPharma Transition Alendronate (A) Risedronate (B) Placebo (C) Interested in A versus B available data A vs C, B vs C  Less confidence than direct?  Why? Indirect Comparisons

Learning Programs to Accelerate the BioPharma Transition Vulnerability of Indirect comparison Effect modifiers –Patients –Optimal interventions –Comparator –Cointerventions –Outcome measures –Risk of bias

Learning Programs to Accelerate the BioPharma Transition Combine direct and indirect comparisons - additional assumption mediators same in direct and indirect - “consistency” or “coherence” assumption

Learning Programs to Accelerate the BioPharma Transition

Network Meta-analysis Case Study: Which Approach to Nicotine Addiction Works Best

Learning Programs to Accelerate the BioPharma Transition Network Meta-analysis Case Study Combines effect estimates from direct and indirect comparisons Placebo Varenicline Antidepressants + NRT Nicotine replacement treatment (NRT) Antidepressants

Learning Programs to Accelerate the BioPharma Transition

Antide- pressants NRT Direct Comparison control buspirone rimonabant varenicline antidepressants +NRT clonidine NRT+NRT 1.63, I 2 =0% 4 comparisons 1.36, I 2 =0% 2 comparisons 0.73, I 2 =0% 2 comparisons 2.68, I 2 =82% 5 comparisons 1.28, I 2 =0% 3 comparisons comparison 1.54, I 2 =46% 5 comparisons 1.14, I 2 =63% 6 comparisons 1.70, I 2 =0% 3 comparisons comparison comparison 1.88, I2=19% 29 comparisons 1.85, I2=13% 67 comparisons NRT superiorAntidepressants superior Direct evidence (3 trials) 1.34 (0.71, 2.56) I-squared=43.7%

Learning Programs to Accelerate the BioPharma Transition Antide- pressants NRT Indirect Comparison 1 control buspirone rimonabant varenicline antidepressants +NRT clonidine NRT+NRT 1.63, I 2 =0% 4 comparisons 1.36, I 2 =0% 2 comparisons 0.73, I 2 =0% 2 comparisons 2.68, I 2 =82% 5 comparisons 1.28, I 2 =0% 3 comparisons comparison 1.54, I 2 =46% 5 comparisons 1.14, I 2 =63% 6 comparisons 1.70, I 2 =0% 3 comparisons comparison comparison 1.88, I 2 =19% 29 comparisons 1.85, I 2 =13% 67 comparisons NRTsuperiorAntidepressants superior Direct evidence (3 trials) Indirect evidence 1.34 (0.71, 2.56) I-squared=43.7% 1.01 (0.81,1.27)

Learning Programs to Accelerate the BioPharma Transition Antide- pressants NRT Indirect Comparison 2 control buspirone rimonabant varenicline antidepressants +NRT clonidine NRT+NRT 1.63, I 2 =0% 4 comparisons 1.36, I 2 =0% 2 comparisons 0.73, I 2 =0% 2 comparisons 2.68, I 2 =82% 5 comparisons 1.28, I 2 =0% 3 comparisons comparison 1.54, I 2 =46% 5 comparisons 1.14, I 2 =63% 6 comparisons 1.70, I 2 =0% 3 comparisons comparison comparison 1.88, I 2 =19% 29 comparisons 1.85, I 2 =13% 67 comparisons antidepressants +NRT 1.14, I 2 =63% 6 comparisons NRTsuperiorAntidepressants superior Direct evidence (3 trials) Indirect evidence 1.34 (0.71, 2.56) I-squared=43.7% 0.85 (0.38, 1.92)

Learning Programs to Accelerate the BioPharma Transition 5 Paths to Indirectly Compare Antidepressants vs NRT placebo and nonplacebo control buspirone rimonabant varenicline antidepressants NRT antidepressants +NRT clonidine NRT+NRT 4 comparisons 2 comparisons 5 comparisons 29 comparisons 67 comparisons 3 comparisons 1 comparison 5 comparisons 6 comparisons 3 comparisons 1 comparison

Learning Programs to Accelerate the BioPharma Transition 5 Paths to Indirectly Compare Antidepressants vs NRT placebo and nonplacebo control buspirone rimonabant varenicline antidepressants NRT antidepressants +NRT clonidine NRT+NRT 4 comparisons 2 comparisons 5 comparisons 29 comparisons 67 comparisons 3 comparisons 1 comparison 5 comparisons 6 comparisons 3 comparisons 1 comparison (0.81, 1.27) 0.85 (0.38, 1.92) 0.89 (0.29, 2.77) 1.56 (0.54, 4.49) 1.31 (0.25, 6.76)

Learning Programs to Accelerate the BioPharma Transition Antide- pressants NRT Comparative effectiveness of NRT vs. Antidepressants on prolonged abstinence (≥6 months) control buspirone rimonabant varenicline clonidine NRT+NRT 1.63, I 2 =0% 4 comparisons 1.36, I 2 =0% 2 comparisons 0.73, I 2 =0% 2 comparisons 2.68, I 2 =82% 5 comparisons 1.28, I 2 =0% 3 comparisons comparison 1.54, I 2 =46% 5 comparisons 1.14, I 2 =63% 6 comparisons 1.70, I 2 =0% 3 comparisons comparison comparison 1.88, I 2 =19% 29 comparisons 1.34, I 2 =44% 3 comparisons 1.85, I 2 =13% 67 comparisons Indirect evidence Direct evidence I-squared = 43.7% 1.34 (0.71, 2.56) trials pooled NRT superiorAntidepressants superior 0.89 (0.29, 2.77) 1.56 (0.54, 4.49) 0.85 (0.38, 1.92) 1.31 (0.25, 6.76) 1.01 (0.81, 1.27) Antidepressants superior NRT superior Path

Learning Programs to Accelerate the BioPharma Transition Antide- pressants NRT Comparative effectiveness of NRT vs. Antidepressants on prolonged abstinence (≥6 months) control buspirone rimonabant varenicline clonidine NRT+NRT 1.63, I 2 =0% 4 comparisons 1.36, I 2 =0% 2 comparisons 0.73, I 2 =0% 2 comparisons 2.68, I 2 =82% 5 comparisons 1.28, I 2 =0% 3 comparisons comparison 1.54, I 2 =46% 5 comparisons 1.14, I 2 =63% 6 comparisons 1.70, I 2 =0% 3 comparisons comparison comparison 1.88, I 2 =19% 29 comparisons 1.34, I 2 =44% 3 comparisons 1.85, I 2 =13% 67 comparisons Indirect evidence Direct evidence I-squared = 43.7% 1.34 (0.71, 2.56) trials pooled NRT superiorAntidepressants superior 0.89 (0.29, 2.77) 1.56 (0.54, 4.49) 0.85 (0.38, 1.92) 1.31 (0.25, 6.76) 1.01 (0.81, 1.27) Antidepressants superior NRT superior Path 0.98 (95% )

Learning Programs to Accelerate the BioPharma Transition Antide- pressants NRT Comparative effectiveness of NRT vs. Antidepressants on prolonged abstinence (≥6 months) control buspirone rimonabant varenicline clonidine NRT+NRT 1.63, I 2 =0% 4 comparisons 1.36, I 2 =0% 2 comparisons 0.73, I 2 =0% 2 comparisons 2.68, I 2 =82% 5 comparisons 1.28, I 2 =0% 3 comparisons comparison 1.54, I 2 =46% 5 comparisons 1.14, I 2 =63% 6 comparisons 1.70, I 2 =0% 3 comparisons comparison comparison 1.88, I 2 =19% 29 comparisons 1.34, I 2 =44% 3 comparisons 1.85, I 2 =13% 67 comparisons Indirect evidence Direct evidence I-squared = 43.7% 1.34 (0.71, 2.56) trials pooled Antidepressants superior 0.89 (0.29, 2.77) 1.56 (0.54, 4.49) 0.85 (0.38, 1.92) 1.31 (0.25, 6.76) 1.01 (0.81, 1.27) Antidepressants superior NRT superior Path 0.98 (95% ) pooled estimate 1.01 (95% ) NRT superior

Confidence in Estimates

Step 1: Present direct and indirect estimate for each comparison of the evidence network Step 2: Rate the confidence in the direct and indirect estimate Comparison Direct evidence OR (95% confidence interval) Direct evidence confidence in estimates Indirect evidence OR (95% credible interval) Indirect evidence confidence in estimates Network OR (95% credible interval) Network confidence in estimates Alendronate vs. Raloxifene 0.49 (0.04 to 5.45) 0.53 (0.30 to 0.90) ExampleApproach Comparison Direct evidence OR (95% confidence interval) Direct evidence confidence in estimates Indirect evidence OR (95% credible interval) Indirect evidence confidence in estimates Network OR (95% credible interval) Network confidence in estimates Alendronate vs. Raloxifene 0.49 (0.04 to 5.45)  LOW 0.53 (0.30 to 0.90)  MODERATE Step 3: Present the NMA estimate Step 4: Rate the confidence in the NMA estimate Comparison Direct evidence OR (95% confidence interval) Direct evidence confidence in estimates Indirect evidence OR (95% credible interval) Indirect evidence confidence in estimates Network OR (95% credible interval) Network confidence in estimates Alendronate vs. Raloxifene 0.49 (0.04 to 5.45)  LOW 0.53 (0.30 to 0.90)  MODERATE 0.51 ( ) Comparison Direct evidence OR (95% confidence interval) Direct evidence confidence in estimates Indirect evidence OR (95% credible interval) Indirect evidence confidence in estimates Network OR (95% credible interval) Network confidence in estimates Alendronate vs. Raloxifene 0.49 (0.04 to 5.45)  LOW 0.53 (0.30 to 0.90)  MODERATE 0.51 ( )  MODERATE

Step 1: Present direct and indirect estimate for each comparison of the evidence network Step 2: Rate the confidence in the direct and indirect estimate Comparison Direct evidence OR (95% confidence interval) Direct evidence confidence in estimates Indirect evidence OR (95% credible interval) Indirect evidence confidence in estimates Network OR (95% credible interval) Network confidence in estimates Alendronate vs. Raloxifene 0.49 (0.04 to 5.45) 0.53 (0.30 to 0.90) ExampleApproach Comparison Direct evidence OR (95% confidence interval) Direct evidence confidence in estimates Indirect evidence OR (95% credible interval) Indirect evidence confidence in estimates Network OR (95% credible interval) Network confidence in estimates Alendronate vs. Raloxifene 0.49 (0.04 to 5.45)  LOW 0.53 (0.30 to 0.90)  MODERATE Step 3: Present the NMA estimate Step 4: Rate the confidence in the NMA estimate Comparison Direct evidence OR (95% confidence interval) Direct evidence confidence in estimates Indirect evidence OR (95% credible interval) Indirect evidence confidence in estimates Network OR (95% credible interval) Network confidence in estimates Alendronate vs. Raloxifene 0.49 (0.04 to 5.45)  LOW 0.53 (0.30 to 0.90)  MODERATE 0.51 ( ) Comparison Direct evidence OR (95% confidence interval) Direct evidence confidence in estimates Indirect evidence OR (95% credible interval) Indirect evidence confidence in estimates Network OR (95% credible interval) Network confidence in estimates Alendronate vs. Raloxifene 0.49 (0.04 to 5.45)  LOW 0.53 (0.30 to 0.90)  MODERATE 0.51 ( )  MODERATE

Starting level of indirect Intutitively lower confidence –Previous GRADE guidance start at moderate NMA enthusiasts argue no different –New guidance: Start at high –Only rate down for lack of similarity (intransivity)

Alendronate n=5,084 Raloxifene n=10,975 Zoledronate n=4,954 Denosumab n=3,933 Vitamin D n=12,469 Calcium n=3,896 Ibandronate n=1,912 Placebo n=41,548 Teriparatide (PTH) n=1,093 Risedronate n=6,850 Hip Fractures # of trials =40 # of participants =139,647 # of hip fracture =2,567 Vitamin D and Calcium n=46,933

Alendronate n=5,084 Raloxifene n=10,975 Zoledronate n=4,954 Denosumab n=3,933 Vitamin D n=12,469 Calcium n=3,896 Ibandronate n=1,912 Placebo n=41,548 Teriparatide (PTH) n=1,093 Risedronate n=6,850 Hip Fractures # of trials =40 # of participants =139,647 # of hip fracture =2,567 Vitamin D and Calcium n=46,933 Dominant link

Confidence in indirect estimate Lowest of direct estimates A versus B comparison of interest Through C dominant link A versus C high; B versus C moderate or low Confidence based on B versus C

Step 1: Present direct and indirect estimate for each comparison of the evidence network Step 2: Rate the confidence in the direct and indirect estimate Comparison Direct evidence OR (95% confidence interval) Direct evidence confidence in estimates Indirect evidence OR (95% credible interval) Indirect evidence confidence in estimates Network OR (95% credible interval) Network confidence in estimates Alendronate vs. Raloxifene 0.49 (0.04 to 5.45) 0.53 (0.30 to 0.90) ExampleApproach Comparison Direct evidence OR (95% confidence interval) Direct evidence confidence in estimates Indirect evidence OR (95% credible interval) Indirect evidence confidence in estimates Network OR (95% credible interval) Network confidence in estimates Alendronate vs. Raloxifene 0.49 (0.04 to 5.45)  LOW 0.53 (0.30 to 0.90)  MODERATE Step 3: Present the NMA estimate Step 4: Rate the confidence in the NMA estimate Comparison Direct evidence OR (95% confidence interval) Direct evidence confidence in estimates Indirect evidence OR (95% credible interval) Indirect evidence confidence in estimates Network OR (95% credible interval) Network confidence in estimates Alendronate vs. Raloxifene 0.49 (0.04 to 5.45)  LOW 0.53 (0.30 to 0.90)  MODERATE 0.51 ( ) Comparison Direct evidence OR (95% confidence interval) Direct evidence confidence in estimates Indirect evidence OR (95% credible interval) Indirect evidence confidence in estimates Network OR (95% credible interval) Network confidence in estimates Alendronate vs. Raloxifene 0.49 (0.04 to 5.45)  LOW 0.53 (0.30 to 0.90)  MODERATE 0.51 ( )  MODERATE

Step 1: Present direct and indirect estimate for each comparison of the evidence network Step 2: Rate the confidence in the direct and indirect estimate Comparison Direct evidence OR (95% confidence interval) Direct evidence confidence in estimates Indirect evidence OR (95% credible interval) Indirect evidence confidence in estimates Network OR (95% credible interval) Network confidence in estimates Alendronate vs. Raloxifene 0.49 (0.04 to 5.45) 0.53 (0.30 to 0.90) ExampleApproach Comparison Direct evidence OR (95% confidence interval) Direct evidence confidence in estimates Indirect evidence OR (95% credible interval) Indirect evidence confidence in estimates Network OR (95% credible interval) Network confidence in estimates Alendronate vs. Raloxifene 0.49 (0.04 to 5.45)  LOW 0.53 (0.30 to 0.90)  MODERATE Step 3: Present the NMA estimate Step 4: Rate the confidence in the NMA estimate Comparison Direct evidence OR (95% confidence interval) Direct evidence confidence in estimates Indirect evidence OR (95% credible interval) Indirect evidence confidence in estimates Network OR (95% credible interval) Network confidence in estimates Alendronate vs. Raloxifene 0.49 (0.04 to 5.45)  LOW 0.53 (0.30 to 0.90)  MODERATE 0.51 ( ) Comparison Direct evidence OR (95% confidence interval) Direct evidence confidence in estimates Indirect evidence OR (95% credible interval) Indirect evidence confidence in estimates Network OR (95% credible interval) Network confidence in estimates Alendronate vs. Raloxifene 0.49 (0.04 to 5.45)  LOW 0.53 (0.30 to 0.90)  MODERATE 0.51 ( )  MODERATE

Confidence in NMA estimate If only direct or indirect use that If both use higher

Confidence in NMA estimate Direct and indirect very different Terminology differs Ours (for now) –Heterogeneity for direct –Transitivity for indirect –Incoherence for direct versus indirect

Judging incoherence Difference in point estimates Extent of overlap in confidence intervals P-value for test of incoherence

Confidence in NMA estimate Options with incoherence Focus on highest confidence –Use that confidence rating Focus on NMA rating –Rated down for incoherence

P-value for test of incoherence 0.02

Credibility of the Process of NMA Usual criteria for systematic review –Explicit sensible questions –Search exhaustive –Selection and assessment reproducible –Present results ready for clinical application Address confidence in effect estimates

Presentation of Results Often presented with rankings Potentially very misleading –Small difference between ranks –Everything low or very low confidence –First ranked lower than others

ComparisonDirect evidence OR (95% confidence interval) Direct evidence confidence in estimates Indirect evidence OR (95% credible interval) Indirect evidence confidence in estimates Network OR (95% credible interval) Network confidence in estimates Teriparatide vs. Placebo ( )very low 3, ( )very low Zoledronate vs. Placebo ( )high0.50 ( )high Risedronate vs. Placebo0.17 (0.05 to 0.59)moderate ( )low ( )moderate

Conclusion NMA will play important role in EBCP Needs usual criteria for SR/MA credibility If includes confidence in estimates interpretable If no confidence very difficult to interpret –Probably shouldn’t bother