Systematic Synthesis of the Literature: Introduction to Meta-analysis Linda N. Meurer, MD, MPH Department of Family and Community Medicine

.. It is necessary, while formulating the problems of which in our advance we are to find solutions, to call into council the views of those of our predecessors who have declared an opinion on the subject, in order that we may profit by whatever is sound in their suggestions and avoid their errors. Aristotle, De Anima

Clinical Overview  Purpose:  Provide general information on a topic  Good to review diagnosis and management  Disseminate experience and opinions of an expert  Methods:  Usually don’t include a methods secsion  References chosen to illustrate points  Conclusions may or may not be “evidence-based”

Critical/ Systematic Review  Purpose  More focused topic; answers specific question(s)  Represent a summary of systematically gathered and analyzed primary research  May lead to new conclusions/ knowledge  Saves the busy clinician the work of interpreting multiple studies on the same subject  Methods  Should always include methods section with at least:  Study inclusion criteria, search strategy, analysis method  References chosen through clear criteria to minimize author bias

Key characteristics of a systematic review  Clearly stated title and objectives  Comprehensive strategy to search for relevant studies (unpublished and published)  Explicit and justified criteria for the inclusion or exclusion of any study  Clear presentation of characteristics of each study included and an analysis of methodological quality  Comprehensive list of all studies excluded and justification for exclusion

Key characteristics of a systematic review (cont.)  Clear analysis of the results of the eligible studies  statistical synthesis of data (meta-analysis) if appropriate and possible;  or qualitative synthesis  Structured report of the review clearly stating the aims, describing the methods and materials and reporting the results

Meta-analysis – Systematic Review with statistical synthesis  Purpose  Usually answers one specific question  Can generate summary estimates of effect from multiple studies  Considered primary research with included studies treated as data  Methods  Identical to other types of Systematic Reviews  Explicit, systematic collection of studies  Uses statistical procedures to combine data or results from different but similar studies

Meta-analysis - advantages  Increase statistical power  Resolve uncertainty when reports disagree  Improve precision of estimates of effect size  Answer questions not posed at the beginning of original studies through examination of study differences, sensitivity analyses

X X X X X X X X X X Relative Risk Example: Forrest Plot  Meta-analysis results often displayed graphically  Each X = results of a single study  Horizontal lines = 95% CI  -X- represents weighted summary estimate after combining all studies.  Note better precision  Most studies not significant by themselves contribute to highly significant summary

Threats to validity  When considering whether the results of any study reflect ‘truth’, there are generally 4 threats:  Selection bias  Study sample doesn’t represent the population of interest  Information bias  Measurement errors, misclassification etc.  Confounding  Association between variables due to or affected by their shared association with another variable  Chance  The probability that data reveals an association that is not real

Meta-analysis - limitations  Threat #1: Selection bias  In the case of meta-analysis, reflects bias in the selection or availability of studies included:  Retrieval bias: Investigator conducting review selects studies that support hypothesis (or are otherwise biased)  Reporting bias: Investigators of original studies only report data that supports view (e.g. drug sponsored?)  Publication bias: Only studies with statistically significant results make it to the journals

Minimizing selection bias  Retrieval bias:  Systematic protocol a priori (before study starts) that includes  Clear selection criteria  Explicit exhaustive search for relevant articles  Multiple reviewers  Reporting bias:  Examine the source of support for work  Conduct sub-analyses to see if source influences results  Publication bias:  Seek unpublished sources of data  Demonstrate through use of a funnel plot

Funnel Plot  A scatterplot of  individual study results (effect size) on the x- axis;  A measure of study size on the y-axis  As sample size goes up  variance decreases  a funnel shape forms

If a publication bias exists:  You might see a skewed plot  Hole in the funnel plot around the null suggests a bias  Results in an overestimate of pooled effect

Meta-analysis: potential threats to validity  Threat #2: Information bias  Quality of a meta-analysis is dependent on quality of original articles, including:  Selection  Measurement  Confounding  The author should conduct a very careful validity assessment of each article included in the study

Meta-analysis: potential threats to validity  Threat #3: Confounding  As with information bias, confounding in individual studies will be transmitted into the meta-analysis.  Differences in populations studied, settings, specific intervention details (dose, duration), measurements used, etc. may result in differences in study results  This can increase generalizability if studies agree  If studies do not agree, may need to explore confounders that might account for disagreement (heterogeneity)

Meta-analysis: potential threats to validity  Threat #4: Chance  By combining the results of smaller studies, the increased power achieved produces a more precise estimate with greater statistical significance  Assuming the included studies are valid, inter-study variability will still occur  Statistical testing for homogeneity can determine whether this variability is greater than one would expect due to chance alone

Tests for homogeneity  Test the probability that observed differences among the results of individual studies were due to chance alone.  Reported as a Cochrane Chi Square (Q-statistic):  statistical significance shows results are not homogenous  Due to outliers?  What do you do?

Heterogeneity  A clue that differences in the studies exist that may lead to new discoveries:  Design  Population: risk factors, setting  Intervention: dose, duration, preparation  Measurements  Finding heterogeneity should prompt an author to explore these factors more fully  Finding heterogeneity also influences the choice of statistical model used to combine the data

You’ve found heterogeneity. So what does one do?  Try to explain  Eliminate obvious outliers and retest  Subgroup analyses  Regression analysis on study characteristics  Incorporate between-study differences  Use a random effects model

Method/model 2 statistical models used  Fixed effect model  estimates treatment effect as if all studies are estimating one single true value  ignores between study variability;  Used when study results are homogeneous  Random effects model  estimates treatment effect as if each study is estimating a distinct value from a distribution of possible results  accounts for between- study variability  Should be used when heterogeneity exists

Summary  You have been introduce to the basic concepts and terminology you need to critically use a meta- analysis, including:  Purpose  Advantages  Potential threats to validity  Analysis methods  Please return to the ANGEL course page (should still be open in another window) and click proceed to move on.