Conducting and Interpreting Systematic Reviews and Meta- Analyses July 12, 2007

Initial Notes about Reviews Reviews are considered the “highest” form of evidence in the EBM evidence pyramid Reviews are considered the “highest” form of evidence in the EBM evidence pyramid Techniques/issues discussed here are different from the type of review (e.g., Psychological Bulletin) you are used to Techniques/issues discussed here are different from the type of review (e.g., Psychological Bulletin) you are used to Reviews are “research of research” (this is the derivation of the ‘meta’ component in ‘meta- analysis’) Reviews are “research of research” (this is the derivation of the ‘meta’ component in ‘meta- analysis’) As such, reviews must follow, and must make explicit, their methodology just like primary studies do As such, reviews must follow, and must make explicit, their methodology just like primary studies do

Evidence Pyramid for Treatment Effectiveness Questions Goal: Use best available evidence

Types of Reviews Systematic Review: literature review focused on a single question which tries to identify, appraise, select and synthesize all high quality research evidence relevant to that question Systematic Review: literature review focused on a single question which tries to identify, appraise, select and synthesize all high quality research evidence relevant to that question Meta-analysis: a statistical technique often used in systematic reviews that enables the results from a number of studies to be combined into a common metric to determine the average effect of a given technique. Comparisons can then be made about the relative effectiveness of various techniques Meta-analysis: a statistical technique often used in systematic reviews that enables the results from a number of studies to be combined into a common metric to determine the average effect of a given technique. Comparisons can then be made about the relative effectiveness of various techniques Integrative Review: literature review that attempts to combine experimental and nonexperimental studies, or from experimental and theoretical work. Can be used to review theories, evidence, or to analyze methodological issues Integrative Review: literature review that attempts to combine experimental and nonexperimental studies, or from experimental and theoretical work. Can be used to review theories, evidence, or to analyze methodological issues Qualitative Review: methods for combining qualitative research studies. Techniques include meta-synthesis, formal grounded theory, and meta-ethnography Qualitative Review: methods for combining qualitative research studies. Techniques include meta-synthesis, formal grounded theory, and meta-ethnography

Review Stages Problem formulation Problem formulation Literature search Literature search Data evaluation/extraction Data evaluation/extraction Data analysis Data analysis Presentation Presentation

Systematic Reviews Definition: summary of the medical literature that uses explicit methods to perform a thorough literature search and critical appraisal of individual studies and that uses appropriate statistical techniques to combine these valid studies Definition: summary of the medical literature that uses explicit methods to perform a thorough literature search and critical appraisal of individual studies and that uses appropriate statistical techniques to combine these valid studies Most evaluate the effectiveness of therapy, but systematic reviews of diagnostic test are also important Most evaluate the effectiveness of therapy, but systematic reviews of diagnostic test are also important Systematic reviews are: Systematic reviews are: Systematic in identification of literature Systematic in identification of literature Explicit in statement of objectives, materials, and methods Explicit in statement of objectives, materials, and methods Reproducible in its methodology and conclusions Reproducible in its methodology and conclusions Best individual sources: Cochrane, PubMed Clinical Queries Best individual sources: Cochrane, PubMed Clinical Queries

Systematic Reviews: Search Strategies Search Strategies Search Strategies Electronic Databases (Medline, EMBASE, Cochrane; Clinical Trials Registry Database) Electronic Databases (Medline, EMBASE, Cochrane; Clinical Trials Registry Database) Hand searching for published studies not in electronic databases Hand searching for published studies not in electronic databases Checking reference lists Checking reference lists Contacting authors Contacting authors Balance between comprehensiveness and precision Balance between comprehensiveness and precision Document search strategy in final review Document search strategy in final review Additional resources: Additional resources: Cochrane handbook: Cochrane handbook: Searchable database of research papers (dissertations and theses) Searchable database of research papers (dissertations and theses) Current Contents Connect Current Contents Connect

Systematic Reviews: Critical Appraisal Critical appraisal = quality assessment Critical appraisal = quality assessment Is the evidence valid and relevant for your clinical question or situation? Is the evidence valid and relevant for your clinical question or situation? Ways validity assessments are used Ways validity assessments are used As a threshold for inclusion As a threshold for inclusion As an explanation for differences in results between studies As an explanation for differences in results between studies As a sensitivity measure As a sensitivity measure As a means for weighting studies in statistical analyses As a means for weighting studies in statistical analyses Dimensions of critical appraisal Dimensions of critical appraisal Internal validity Internal validity Applicability Applicability Precision (size of CI) Precision (size of CI) Useful resources Useful resources

Systematic Reviews: Conclusions Methodological Issues Methodological Issues Number of studies and participants Number of studies and participants Size of treatment effect Size of treatment effect Precision of treatment effect Precision of treatment effect Consistency of outcomes and results Consistency of outcomes and results Apparent dose-response relationships, if any Apparent dose-response relationships, if any Generalizability and applicability of findings to other populations and settings Generalizability and applicability of findings to other populations and settings

Meta-Analysis Meta analysis is a set of statistical techniques for combining information from different studies to derive an overall estimate of a treatment's effect. Meta analysis is a set of statistical techniques for combining information from different studies to derive an overall estimate of a treatment's effect. Potential issues (a “Catch 22”): Potential issues (a “Catch 22”): Publication bias Publication bias Varying quality of studies Varying quality of studies Suggestions for high quality meta-analyses: Suggestions for high quality meta-analyses: A formal protocol should be written specifying the exact question under investigation and describing the studies that will be included in the analysis. A formal protocol should be written specifying the exact question under investigation and describing the studies that will be included in the analysis. All research, not just published research, should be included. The issue of “fugitive literature” is important. All research, not just published research, should be included. The issue of “fugitive literature” is important. Registries should be established so that studies can be tracked from their inception and not just on publication. This idea has been given a push so that drug companies would not be able to publish trials showing benefit from their products while suppressing those that do not. Registries should be established so that studies can be tracked from their inception and not just on publication. This idea has been given a push so that drug companies would not be able to publish trials showing benefit from their products while suppressing those that do not. Measures of study quality should be included Measures of study quality should be included Clear eligibility criteria should be established Clear eligibility criteria should be established Many meta analytic techniques should be used and all results should be reported. A result would be considered reliable only if all of the techniques give the same result. Many meta analytic techniques should be used and all results should be reported. A result would be considered reliable only if all of the techniques give the same result.

Steps in Statistical Analysis Select measure of effect (rate difference, odds ratio, standardized mean differenceetc.) Select measure of effect (rate difference, odds ratio, standardized mean differenceetc.) RCT: rate difference (EER v. CER) or rate ratio RCT: rate difference (EER v. CER) or rate ratio Case Control: odds or rate ratios Case Control: odds or rate ratios Standardized mean difference between E and C Standardized mean difference between E and C Specify effects model: Specify effects model: Fixed: results generalizable to reviewed studies – assumes that there is an underlying effect running through all studies Fixed: results generalizable to reviewed studies – assumes that there is an underlying effect running through all studies Random: results generalizable to universe of such studies (computationally more intense) – assumes that the true effect estimate varies across studies Random: results generalizable to universe of such studies (computationally more intense) – assumes that the true effect estimate varies across studies

Meta-Analysis: Data Analytic Techniques Dichotomous outcomes (e.g., cured v. not) Dichotomous outcomes (e.g., cured v. not) Mantel-Haenszel: pooled odds ratio across a strata of 2x2 tables Mantel-Haenszel: pooled odds ratio across a strata of 2x2 tables Analysis example: sts/mh.htm Analysis example: sts/mh.htm sts/mh.htm sts/mh.htm

Meta-Analysis: Data Analytic Techniques (cont’d) Continuous Data: Standardized difference as an example (e.g., Cohen’s d) Continuous Data: Standardized difference as an example (e.g., Cohen’s d) d = M1-M2/SD d = M1-M2/SD Pooled SD of experimental and control groups is usually used; square root of the average of the two squared SD’s Pooled SD of experimental and control groups is usually used; square root of the average of the two squared SD’s Can be computed from t-test (d = 2t/  df Can be computed from t-test (d = 2t/  df Other measures: Hedges’s g Other measures: Hedges’s g

Therapeutic trials of streptokinase for MI

Van Emmerik, A.A.P., Kamphuis, J.A., Hulsbosch, A.M., & Emmelkamp, P.M.G. (2002). Single-session debriefing after psychological trauma: A meta-analysis. Lancet, 360,

Integrative Reviews Provide review of available literature according to clear methodological approach Provide review of available literature according to clear methodological approach Potential sources of bias Potential sources of bias Incomplete literature search stage Incomplete literature search stage Errors in data extraction Errors in data extraction Inappropriate data analysis, particularly in combining experimental and nonexperimental methods Inappropriate data analysis, particularly in combining experimental and nonexperimental methods

Integrative Reviews Defining the review question clearly Defining the review question clearly Literature search issues Literature search issues Obtaining needed literature often challenging Obtaining needed literature often challenging Publication bias Publication bias Identify maximum number of primary sources through multiple strategies; make strategies explicit (i.e., specify search terms) in review Identify maximum number of primary sources through multiple strategies; make strategies explicit (i.e., specify search terms) in review Data evaluation issues Data evaluation issues Explicit assessment of study quality – complex when included studies have vastly different designs Explicit assessment of study quality – complex when included studies have vastly different designs How to define quality of nonempirical studies? How to define quality of nonempirical studies?

Integrative Reviews (cont’d) Data analysis issues Data analysis issues Developing a classification system for managing data from diverse methodologies Developing a classification system for managing data from diverse methodologies Classifying studies based on some logical, predetermined system (e.g., type of evidence, demographic characteristics, predetermined conceptual classification) and analyzing each subgroup sequentially Classifying studies based on some logical, predetermined system (e.g., type of evidence, demographic characteristics, predetermined conceptual classification) and analyzing each subgroup sequentially

Qualitative Reviews Qualitative research is typically nonexperimental, but it may be useful to include in systematic reviews because it evaluates important aspects of clinical research, often from the patient perspective (e.g., “views” research) Qualitative research is typically nonexperimental, but it may be useful to include in systematic reviews because it evaluates important aspects of clinical research, often from the patient perspective (e.g., “views” research) Relies on inductive rather than deductive reasoning Relies on inductive rather than deductive reasoning Qualitative research involves different measures of report quality than do RCT’s Qualitative research involves different measures of report quality than do RCT’s Integrating quantitative and qualitative data sources is a key challenge to systematic reviewers Integrating quantitative and qualitative data sources is a key challenge to systematic reviewers

Types of Qualitative Data

Major Types of Qualitative Research Participant Observation: used to assess the impact of healthcare events from the patient perspective Participant Observation: used to assess the impact of healthcare events from the patient perspective Interviews: used to assess range of ideas and concepts among individuals involved in healthcare delivery and receipt of healthcare services. Oral histories are one example Interviews: used to assess range of ideas and concepts among individuals involved in healthcare delivery and receipt of healthcare services. Oral histories are one example Interview example: Interview example: Record review: used to assess phenomenology that leads to health care decisions (e.g., what factors lead to discharge or prescription of particular medications or behavioral intervention strategies) Record review: used to assess phenomenology that leads to health care decisions (e.g., what factors lead to discharge or prescription of particular medications or behavioral intervention strategies) Focus groups Focus groups Case studies Case studies

Qualitative Research: Analytic Strategy Examples Constant Comparison Method: All data sources are compared to all others to search for commonalities or differences Constant Comparison Method: All data sources are compared to all others to search for commonalities or differences “Grounded Theory” (Glaser, Strauss) – systematic method of generating theory from data “Grounded Theory” (Glaser, Strauss) – systematic method of generating theory from data Ethnographic methods: How individuals or cultural groups conceptualize or enact health/illness or health-seeking Ethnographic methods: How individuals or cultural groups conceptualize or enact health/illness or health-seeking

Thomas, et al., (2004). Integrating qualitative research with trials in systematic reviews. BMJ, 328, Integrating Qualitative and Quantitative Date in Systematic Reviews