Systematic Review & Meta-Analysis 系统综述和meta 分析 Xu Xiong, MD, DrPH School of Public Health and Tropical Medicine Tulane University

Narrative Reviews, Systematic Reviews, and Meta-Analysis Narrative Review: traditional expert review Subjective, no formal rules in selecting studies, no standard statistical methods for combining studies Systematic Review: review in which there is a comprehensive search for relevant studies on a specific topic, and those identified are then appraised and synthesized according to a predetermined and explicit method. Meta-Analysis: systematic review that employs statistical methods (a quantitative summary) to combine and summarize the results of several studies.

Definition: Meta-Analysis Coined by Glass in 1976 from the Greek prefix “meta” meaning “after,” “more comprehensive,” or “transcending” and the root, “analysis” The statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings (Glass, 1976) A statistical synthesis of the data from separate but similar (i.e., comparable) studies, leading to a quantitative summary of the pooled results (Last, 2001) Synonyms: Research synthesis, Pooled analysis, Quantitative review, Overview Gene Glass Pooled analysis is sometimes used as a synonym. However, this term incorrectly implies that source data are merged, when in fact summary estimates are traditionally combined.

Number of Papers Referencing Meta-Analysis, 1985-2011 Results from MEDLINE search using MeSH or text word “meta-analysis”

Historical Note 1904 – Karl Pearson derived formulas to combine correlations from different samples 1932 – R.A. Fisher developed a method to combine p-values from different studies 1976 – Gene Glass coined the term meta-analysis 1977 – Smith & Glass published the first meta-analysis paper cited in MEDLINE Meta-analysis of psychotherapy outcome studies (Am Psychol 1977;32:752-760) 1989 – Meta-analysis was adopted by MEDLINE as a subject heading 1993 – Meta-analysis was adopted by MEDLINE as a publication type

Strength of Evidence Concerning Efficacy of Treatment

Meta-Analysis Meta-analysis differs from: Primary analysis: the original analysis of data from a research study Secondary analysis: the re-analysis of data to answer new research questions Meta-analysis methods focus on: Contrasting and comparing results from different studies in anticipation of identifying consistent patterns and sources of disagreements among the results

Why Use Meta-analysis To provide a more objective appraisal of the evidence To reduce the probability of false negative results To test treatment effects in subgroups of patients To explore and explain heterogeneity between study results To generate research questions to be addressed in future studies

Meta-Analysis Objectives Synthetic goal (estimation of summary) Analytic goal (estimation of differences) Meta-analysis is the part of the review process that concerns itself with the analysis of data extracted from the primary research included, and which uses quantitative methods to explore the heterogeneity of study results, estimate the overall measures of association or effect and assess the sensitivity of the results to possible threats from validity such as publication bias and study quality.

When to Use Meta-Analysis When individual trials or studies are too small to give reliable answers When large trials or studies are impractical or impossible When there have been many trials or studies showing small effects are important When trial or study results are inconclusive or conflicting

Potential Limitations of Meta-Analysis Problems associated with design or reporting original studies Publication bias Limitations of using published data Retrospective research Variation of standard treatments over time Heterogeneity of studies Statistical methods

Steps of Meta-Analysis Formulate research question Develop a proposal Comprehensive literature search Selection of study Critical appraisal of study Extraction of data Synthesis of data Sensitivity and subgroup analyses if appropriate and possible Preparing a structured report

Formulating a Research Question What are the study objectives? To validate results in a large population To guide new studies What are the operational definitions? Disease or condition of interest Population and setting Treatment, intervention or exposures (e.g. risk factor, medication, diagnostic test) Outcomes of interest (both beneficial and harmful) What types of study designs? Randomized controlled trials: e.g., Cochrane Review Observational studies

Literature Scoping Useful for formulating the research question Preliminary assessment of potentially relevant literature Search for existing reviews and primary studies relevant to the topic Usually only undertaken in a small range of databases relevant to the topic Cochrane Controlled Trials Register (CCTR) MEDLINE and EMBASE for medical topics PsycLIT for reviews of psychological and psychiatric topics

Study Design Population Interventions Comparisons Outcomes Inclusion Criteria Study Design Population Interventions Comparisons Outcomes

Practical Considerations in Defining Eligibility for a Meta-Analysis Study designs to be included Years of publication or study conduct Languages Choice among multiple publications Restrictions due to sample size or follow-up duration Similarity of treatment and/or exposure Completeness of information

Comprehensive Data Search Need a well formulated and coordinated effort Seek guidance from a librarian Specify language constraints Requirements for comprehensiveness of the search depends on the field and question to be addressed

Tulane Medical Library

Literature Search Computerized bibliographic database (MEDLINE) Bibliography searches Current contents Dissertations Textbooks Databases of unpublished work Citation searches Expert survey Meeting proceedings and abstracts Granting agencies Trial registries Industry Journal hand-searching

Literature Search Challenges Database Bias - “No single database is likely to contain all published studies on a given subject.” Publication Bias - selective publication of articles that show positive treatment of effects and statistical significance. It is important to search for unpublished studies through a manual search of conference proceedings, correspondence with experts, and a search of clinical trials registries. English-language bias - occurs when reviewers exclude papers published in languages other than English Citation bias - occurs when studies with significant or positive results are referenced in other publications, compared with studies with inconclusive or negative findings

Unbiased Selection and Extraction Process Study Selection Two independent reviewers select studies Based on a priori specification of the population, intervention, outcomes and study design Level of agreement: kappa Differences are resolved by consensus Specify reasons for rejecting studies

Data Extraction Two independent reviewers extract data using pre-established forms Should be explicit, unbiased, and reproducible Include all relevant measures of benefit and harm of the intervention Contact investigators of the studies for clarification in published methods, data Extract individual patient data when published data do not answer questions about: intention to treat analyses, time-to-event analyses, subgroups, dose-response relationships Methodological quality Level of agreement: kappa Differences in data extraction are resolved by consensus

Data Extraction: Study Characteristics Types of publication (journal article, abstract or unpublished data) Publication year and country of origin Study participants (sample size, age, gender, race, health status) Design details (case-control, cohort, parallel or cross-over, randomization, blinding) Nature of treatment and control Study duration Measurement of compliance Definition and measurement of outcome Other confounders

Data Extraction: Study Outcome Continuous variables Mean difference between treatment and control groups Binary variables Odds ratios Relative risks Hazard ratios Absolute risk reduction or number of patients needed to be treated to prevent one event

Critical Appraisal of Data Description of Studies Size of study Characteristics of study participants Details of specific interventions used Details of outcomes assessed

Study Quality Assessment Choose a method of assessment of quality of original studies, for example, Chalmers RCT Quality Score (Controlled Clinical Trials 1981;2:31-49) Assess quality of each study in uniform, systematic and complete manner Identify acceptable studies and give score to their quality Keep a list of unacceptable studies Consider weighting each study result by quality score, or stratifying by quality

Synthesis of Data Graphic displays Flow diagram Forest plot Pooling data Fixed-effects model Random-effects model Test for heterogeneity Subgroup analysis Meta-regression Statistical techniques for meta-analysis can broadly be classified into two models, differing in how the variability of the results between studies is treated. The fixed effects models assumes variability is exclusively due to random variation or chance and that if all studies were infinitely large their results would be the same. The random effects model assumes a different underlying effect for each study and takes this into consideration as an additional source of variation.

Active Management of 3rd Stage: Cochrane Review/Meta-Analysis A Forest Plot Active Management of 3rd Stage: Cochrane Review/Meta-Analysis Cochrane Review http://www.cochrane.org/

Data Analysis Include all relevant and clinically useful measures of treatment effect Perform a narrative, qualitative summary when data are too sparse, or too low quality or too heterogeneous to proceed with a meta-analysis Specify if fixed or random effects model is used Describe proportion of participants used in final analysis Use confidence intervals Include a power analysis Consider cumulative meta-analysis (by order of publication date, baseline risk, study quality) to assess the contribution of successive studies

Software for Meta-analysis: e.g., WinPEPI

Subgroup Analyses Pre-specify hypothesis-testing subgroup analyses and keep few in number Label all posteriori subgroup analyses When subgroup differences are detected, interpret in light of whether they were: Established a priori Few in number Supported by plausible mechanisms Important (qualitative vs. quantitative) Consistent across studies Statistically significant (adjusted for multiple testing)

Sensitivity Analysis Test robustness of results relative to key features of the studies and key assumptions and decisions Include tests of bias due to retrospective nature (e.g., with/without studies of lower methodological quality)

Publication bias: A Funnel Plot “A funnel plot is used as a way to assess publication bias in meta-analysis.” Kevin C. Chung, MD, Patricia B. Burns, MPH, H. Myra Kim, ScD. “Clinical Perspective: A Practical Guide to Meta-Analysis.” The Journal of Hand Surgery. Vol.31A No.10 December 2006. p. 1676

Prepare a Structured Report Include a structured abstract Include a table of the key elements of each study Include a flow diagram detailing the study selection process Include summary data from which the measures are computed Employ formative graphic displays representing confidence intervals, group event rates, sample sizes

Interpretation of Findings Interpret results in context of current health care State methodological limitations of the individual studies included and in the meta-analysis Consider size of effect in studies and meta-analysis, consistency of effect sizes and any dose-response relationship Consider interpreting results in context of temporal cumulative meta-analysis Interpret results in light of other valuable evidence Make recommendations clear and practical Propose future research agenda (clinical and methodological requirements)

Summary A well conducted meta-analysis allows for a more objective appraisal of the evidence than traditional narrative reviews Meta-analysis may resolve uncertainties and disagreements in original research Meta-analysis may enhance the precision of estimates of treatment effects Exploratory analyses (i.e., subgroups who are likely to respond well to a treatment) may guide cost effective treatment decisions Meta-analyses may demonstrate areas where the evidence is inadequate and thus identity areas where further research is needed