Quantitative Synthesis II Prepared for: The Agency for Healthcare Research and Quality (AHRQ) Training Modules for Systematic Reviews Methods Guide

Systematic Review Process Overview

 To understand how to explore between-study heterogeneity in a meta-analysis  To understand the pros and cons of subgroup analyses  To understand what meta-regression is and why it is useful  To understand the usefulness of control-rate meta-regression Learning Objectives

Statistical Homogeneity Barrington KJ. Cochrane Database Syst Rev 2000;(2):CD

Statistical Heterogeneity Reprinted from Pakos E, et al. J Bone Joint Surg Am 2005;87: , with permission from Rockwater, Inc.. Patellar Resurfacing in Total Knee Arthroplasty for Pain RE = random effects model

 In meta-analysis, heterogeneity refers to between-study diversity.  This term is often used to refer to:  differences in study characteristics, and  variability of study results. What Is Heterogeneity?

 Methodological diversity pertains to specifics of study design and analysis:  Type of study  Length of followup  Proportion and handling of dropouts  Clinical diversity pertains to differences in:  Populations  Interventions and cointerventions  Outcomes Methodological and Clinical Diversity

 Statistical heterogeneity exists when the results of individual studies are not “consistent” among themselves. Statistical Heterogeneity Clinical diversity Methodological diversity Biases Chance Statistical heterogeneity

 Clinical and methodological diversity is abundant: Our aim in a meta-analysis is to explore that diversity and use our observations to formulate interesting hypotheses.  Often clinical and methodological heterogeneity results in a statistically significant test.  Chance, technical issues, or biases may result in statistically significant results in heterogeneity tests. Clinical and Methodological Diversity Versus Statistical Heterogeneity

 One of the most important roles of meta-analytic methodologies is to quantify statistical heterogeneity and to explore whether and to what extent it is explained by clinical and methodological diversity. Exploration of Heterogeneity Is Central to Evidence Synthesis

Dealing With Heterogeneity (I)

OVERALL ESTIMATE Combining Summary Data Combining Summary Data META-REGRESSION Modeling Summary Data Modeling Summary Data RESPONSE SURFACE Modeling Individual Patient Data Modeling Individual Patient Data SUBGROUP ANALYSES Differentiating Effects in Subgroups Differentiating Effects in Subgroups Dealing With Heterogeneity (II)

 Subgroup analyses can help:  identify modifiers of the treatment effect,  recognize biologically interesting phenomena, or  formulate hypotheses.  We discuss two illustrative examples:  Time-to-thrombolysis affects the treatment effect of thrombolytic drugs in patients with acute myocardial infarction.  Effects of vitamin E supplementation on mortality may differ by vitamin E dose. Promises of Subgroup Analyses

Risk Ratio (95% Confidence Interval) Effects of Thrombolytic Therapy on Mortality in Patients With Acute Myocardial Infarction (Mean Time-to- Treatment) Subgroup Analysis: A Meta-analysis of Thrombolytic Therapy for Acute Myocardial Infarction (I)

Largest Effect in the Subgroup of Trials With Mean Time-to-Treatment of 0 to 3 Hours Subgroup Analysis: A Meta-analysis of Thrombolytic Therapy for Acute Myocardial Infarction (II)

Treatment Effect Diminishes When Mean Time-to- Treatment Is Between 3.1 and 5 Hours Subgroup Analysis: A Meta-analysis of Thrombolytic Therapy for Acute Myocardial Infarction (III)

Subgroup Analysis: A Meta-analysis of Thrombolytic Therapy for Acute Myocardial Infarction (IV) Treatment Effect Diminishes Further When Mean Time-to-Treatment Is Between 5.1 and 10 Hours

No Evidence of a Treatment Effect When Mean Time to Treatment Longer than 10 Hours Subgroup Analysis: A Meta-analysis of Thrombolytic Therapy for Acute Myocardial Infarction (V)

Subgroup Analysis: A Meta-analysis of Vitamin E Doses and Mortality Miller ER 3rd, et al. Ann Intern Med. 2005;142: Reprinted with permission from the American College of Physicians.

 Subgroup analyses are a form of multiple testing.  When uncontrolled, multiple testing can yield spurious findings.  Most meta-analyses do not perform statistical adjustments for multiple testing. Hazards of Subgroup Analyses: Multiple Testing (I)

 In the example that follows, we discuss subgroup analyses from the Second International Study of Infarct Survival (ISIS-2), a 2x2 factorial study of streptokinase versus placebo and aspirin versus placebo in more than 17,000 patients with myocardial infarction.  For streptokinase versus placebo, the treatment effect does not differ across subgroups by history of prior myocardial infarction.  For aspirin versus placebo, the treatment effect does differ. Hazards of Subgroup Analyses: Multiple Testing (II)

Reprinted from ISIS-2 Collaborative Group. Lancet 1988;2:349-60, with permission from Elsevier. Subgroup Analysis: Second International Study of Infarct Survival (ISIS-2) (I)

Subgroup Analysis: Second International Study of Infarct Survival (ISIS-2) (II) Reprinted from ISIS-2 Collaborative Group. Lancet 1988;2:349-60, with permission from Elsevier.

 When analyzing data, it is important to distinguish subgroup analyses that are specified a priori (without knowing what the data are) versus those that are specified post hoc (after the researcher has been exposed to the data).  This distinction is very clear when analyzing a prospective study.  In most meta-analyses, the distinction is not as clear. Can You Avoid the Hazards of Subgroup Analyses? (I)

 Most meta-analyses use data that are published (and potentially known).  When researchers adequately prepare before embarking on a meta-analysis, they inevitably become acquainted with the data they will analyze.  This makes it difficult for researchers to claim that they specified subgroups without knowing anything about their data. Can You Avoid the Hazards of Subgroup Analyses? (II)

 Meta-analysts should do their best to define subgroups that make methodological and biological sense.  Treat the results of subgroup analyses with a healthy dose of skepticism, especially when adjustments for multiple testing are not performed. Can You Avoid the Hazards of Subgroup Analyses? (III)

 Meta-regression can help examine how the treatment effect changes across the levels of a variable.  All subgroup analyses can be formulated in a meta-regression framework, but meta-regression goes well beyond subgroup analyses. Beyond Subgroup Analyses: Meta-Regression

Subgroup Analysis: A Meta-analysis of Vitamin E Doses and Mortality Miller ER 3rd, et al. Ann Intern Med. 2005;142: Reprinted with permission from the American College of Physicians.

Corresponding Univariate Meta-Regression: A Meta-analysis of Vitamin E Doses and Mortality Miller ER 3rd, et al. Ann Intern Med. 2005;142: Reprinted with permission from the American College of Physicians.

~τ 2 ~τ’ 2 Reprinted from Ioannidis JPA. In: Publication bias in meta-analysis: prevention, assessment and adjustments, with permission from Wiley-Blackwell, Copyright © Meta-Regression: A Meta-analysis of Zidovudine Monotherapy Versus Placebo

 Study level  Examples: presence/absence of blinding, intervention dose (in experimental studies)  Participant level  Examples: mean age, proportion of diabetic patients, mean intake of vitamin E (in observational studies) Two Types of Covariates in Meta-Regressions

 Aggregate-data meta-regressions on participant- level covariates can mislead, because they are susceptible to ecological fallacy.  The observed relationship between the study- level treatment effect and the mean of a patient- level factor does not necessarily reflect the corresponding true relationship at the individual patient level. Spurious Associations in Meta-Regressions or Subgroup Analyses (I)

 Therefore, associations of treatment effect and participant-level covariates should be interpreted with caution.  Such associations can be biologically plausible and informative, or can mislead.  Unfortunately, there is no universal way to distinguish true from fallacious findings. Spurious Associations in Meta-Regressions or Subgroup Analyses (II)

 In control-rate meta-regression, we examine whether the treatment effect changes across studies with different event rates in the controls.  The control rate is a surrogate for the baseline risk for the event of interest (severity of disease).  This approach examines whether the underlying risk of an event explains differences in the treatment effect across studies. Control-Rate Meta-Regression

Intravenous Streptokinase Therapy in Patients With Acute Myocardial Infarction (I)

Intravenous Streptokinase Therapy in Patients With Acute Myocardial Infarction (II)

Control Rate Meta-Regression in the Preceding Streptokinase Example Schmid CH, et al. Stat Med 1998;17:

 Subgroup analyses, meta-regressions, and control-rate meta-regressions are tools to explore between-study heterogeneity.  Use them to understand data.  They are mostly hypothesis-forming tools.  Especially for meta-regressions on patient-level covariates, ecological fallacy may mislead.  Beware when interpreting their results. Key Messages

 Barrington KJ. Umbilical artery catheters in the newborn: effects of position of the catheter tip. Cochrane Database Syst Rev 2000;(2):CD  DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials 1986;7:  ISIS-2 (Second International Study of Infarct Survival) Collaborative Group. Randomized trial of intravenous streptokinase, oral aspirin, both, or neither among 17,817 cases of suspected acute myocardial infarction: ISIS-2. Lancet 1988;2:  Miller ER 3rd, Pastor-Barriuso R, Dalal D, et al. Meta- analysis: high-dosage vitamin E supplementation may increase all-cause mortality. Ann Intern Med 2005;142: References (I)

 Pakos E, Ntzani EE, Trikalinos TA. Patellar resurfacing in total knee arthroplasty. A meta- analysis. J Bone Joint Surg Am 2005;87:  Ioannidis JPA. Differentiating biases from genuine heterogeneity: distinguishing artifactual from substantive effects. In: Rothstein HR, Sutton AJ and Borenstein M, eds. Publication bias in meta-analysis: prevention, assessment and adjustments. Chichester, England: Wiley; p  Schmid CH, Lau J, McIntosh MW, et al. An empirical study of the effect of the control rate as a predictor of treatment efficacy in meta-analysis of clinical trials. Stat Med 1998;17: References (II)

 This presentation was prepared by Joseph Lau, M.D., and Thomas Trikalinos, M.D., Ph.D., members of the Tufts Medical Center Evidence- based Practice Center.  The information in this module is based on Chapter 9 in Version 1.0 of the Methods Guide for Comparative Effectiveness Reviews (available at: es/2007_10DraftMethodsGuide.pdf). Authors