EPI-214: Lecture 1 Designing a Systematic Review (Meta-analysis) Dejana Braithwaite Assistant Professor UCSF Department of Epidemiology and Biostatistics

8 steps of a systematic review 1 Formulate research question Lecture 1 2 Develop review protocol 3 Initiate search strategy 4 Apply inclusion /exclusion criteria 5 Quality appraisal 6 Data abstraction 7 Analysis Lectures 1, 2 & 3 8 Interpret findings Bent et al. 2004

What’s a Systematic Review? “A review of the evidence on a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant primary research, and to extract and analyze data from the studies that are included in the review.” Cochrane Collaboration

…and meta-analysis? Statistical combination of >= 2 studies to produce single estimate of effect of exposure NOTE: meta-analysis is simply the statistical combination of findings; it should not be considered as a type of review

Rapid growth Pubmed search up to 3/29/2010: Systematic review >1.5 million hits Meta-analysis >40,000 hits up to 1/1/1990: Systematic review 300,994 hits Meta-analysis 845

The Cochrane Collaboration International systematic review initiative Archie Cochrane’s vision led to the opening of the first Cochrane centre (in Oxford, UK) in 1992 and the founding of the Cochrane Collaboration in 1993 The Cochrane Collaboration is a group of over 27,000 volunteers in more than 90 countries who review the effects of health care interventions tested in biomedical randomized controlled trials.[3] A few more recent reviews have also studied the results of non-randomized, observational studies. The results of these systematic reviews are published as "Cochrane Reviews" in the Cochrane Library. Source: http://www.cochrane.org/cochrane/archieco.htm

Meta-analysis news coverage Sattar N, et al. Statins and risk of incident diabetes: a collaborative meta-analysis of randomised statin trials. Lancet. 2010 Feb 27;375(9716):735-42. Epub 2010 Feb 16.

Systematic Review in context Traditional review Systematic Review Meta-analysis Author An individual A team Search strategy Individual Based on a protocol Summary Conclusion Author’s judgement Can be qualitative Summary statistical techniques

Narrative review Uses informal, unsystematic and subjective methods Searching quality and synthesis not described Disadvantage may have preconceived biases and may overestimate value of some studies

Systematic review Driven by evidence-based medicine movement and Cochrane collaboration Advantages: Reduces bias Replicable Resolves controversy between conflicting findings Provides reliable basis for decision making

Use of meta-analysis as a prelude to clinical trials Define pre-trial expected effect sizes sample size estimation Determine effect estimates in key subgroups (e.g. based on gender, race/ethnicity or age) Identify sources of heterogeneity in prior studies Addressing these sources in design phase of new trial Determine effect estimates in key subgroups (e.g. based on gender, race/ethnicity or age) Identify (through meta-regression) the sources of heterogeneity in prior studies

Use of meta-analysis in study designs that are not clinical trials Observational studies (e.g. case control, cohorts, cross-sectional prevalence studies, etc.) Studies evaluating diagnostic tests (sensitivity, specificity, predictive value) “IPD” = individual patient data studies Qualitative studies (meta-ethnography) Although meta-analysis is most widely known in biomedical settings for its application to clinical trials, the technique can also be used to synthesize/analyze other study designs:

Meta-analyses IPD* Systematic reviews *IPD= individual participant data

Resources required for systematic reviewing Can be time consuming Team science (to reduce bias) Bibliographic software (e.g. Endnote) Statistical software (if appropriate)

Formulate research question Step 1 Formulate research question

FINER criteria for research question Feasible Interesting Novel Ethical Relevant Usually the best if systematic reviews and meta-analyses are conducted by teams that include investigators with both methodologic and content expertise. Hulley S, et al. 2001 Designing Clinical Research

Components of a research question (PICOT) Patient: Disease or condition Stage, severity Demographic characteristics (age, gender, etc.) Intervention (or “Exposure”): Type of intervention or exposure Dose, duration, timing, route, etc. Comparison: Absence of risk or treatment Placebo or alternative therapy Outcome: Risk or protective Dichotomous or continuous Type: mortality, morbidity, quality of life, etc.

PICOT Type of Study: RCTs Cohort Case-control Cross-sectional All

Formulation of an etiology question Exposure Outcome Is smoking a risk factor for lung cancer? Patient Exposure Are people who smoke regularly at a greater risk of developing lung cancer as compared to those who do not smoke? + cohort & case-control studies Outcome Comparison

Formulation of a diagnosis question Test (intervention) Outcome Is MRI a good screening test for breast cancer? Test (intervention) Outcome Is MRI a more sensitive and specific test in diagnosing breast cancer as compared to mammography among high risk women? Comparison

Develop review protocol Step 2 Develop review protocol

Protocol Background Objectives Pre-determined selection criteria Planned search strategy Planned data abstraction Proposed method of synthesis of findings Establishment of an advisory group Background give rationale for systematic review as well as importance of research question

Initiate search strategy Step 3 Initiate search strategy

Where to locate studies Pubmed CINAHL Web of Science EMBASE PsychINFO

Additional sources to identify studies for systematic reviews Reference lists of retrieved articles Manual searching of relevant publications Experts in the field Corresponding or first authors of published studies identified for the systematic review

Publication bias Search bias Issues to consider Publication bias Search bias

Pubmed citation example Title: Interaction between 5-HTTLPR genotype, stressful life events and depression Search terms: Life stress Life event Depression Depress Serotonin transporter 5-HTTLPR Interaction Moderation Risch et al. JAMA 2010

Apply inclusion /exclusion criteria Step 4 Apply inclusion /exclusion criteria

Inclusion/exclusion criteria P - Population I - Intervention C - Comparison (if necessary) O - Outcome T - Type of study (if necessary) Subject headings OR Textwords To find studies using all of the PICO elements: P and I and C and O (and T) Reviewers must determine which studies are relevant and suitable for inclusion in the systematic review. As noted before, it is crucial that the criteria for study selection be determined ahead of time to the extent possible, so investigators cannot, for example, decide to exclude certain types of studies that do not show a positive result. The specific inclusion criteria will depend on the study question.

Exclusion criteria Keep log of excluded studies Note reasons for exclusion Have eligibility checked by more than one reviewer Develop strategy to resolve disagreements

Search strategy example Risch et al. JAMA 2010

Step 5 Quality appraisal

Principles of quality appraisal Quantitative studies Internal Validity allocation bias, confounding, attrition, statistical analysis, intervention integrity, withdrawals and dropouts External Validity (generalizability or applicability)

Are there quality appraisal tools? Consider assessment by more than 1 reviewer

QUORUM for trials Moher et al. Improving the quality of reports of meta-analyses of randomized controlled trials: The QUORUM statement. Lancet 1999;354:1896-1900

MOOSE for observational designs Stroup et al. Meta-analysis of observational studies in epidemiology. JAMA 2000;283:2008-12

Qualitative research Checklists available to assess the quality of qualitative research. E.g. CASP appraisal tool for qualitative research (http://www.phru.nhs.uk/casp/qualitat.htm)

Allocate to intervention and control groups Recruit participants SELECTION BIAS ALLOCATION BIAS Allocate to intervention and control groups Intervention group Control group CONFOUNDING INTEGRITY OF INTERVENTION Implement intervention Implement intervention INTENTION-TO-TREAT Follow-up participants Follow-up participants WITHDRAWALS/ DROP OUTS Always assess selection and allocation bias, blinding and handling of patient attrition Measure outcomes BLINDING OUTCOME ASSESSORS Measure outcomes DATA COLLECTION METHODS Analyze outcomes Analyze outcomes STATISTICAL ANALYSIS

Step 6 Data abstraction

Data abstraction Design and pilot data abstraction form Consider >1 reviewer Consider blinding of observers to authors, institutions and journals

Data abstraction elements publication details study design population details (n, characteristics) intervention details setting outcomes and findings

Summary of study characteristics - example Risch et al. JAMA 2009

Data abstraction examples http://www.cochrane.org

Step 7 Analysis

Synthesizing the Evidence NARRATIVE SYNTHESIS primary studies explained qualitatively and summarized META-ANALYSIS findings summarized and then combined statistically Before deciding which approach to take, it is important to tabulate the findings from the studies and examine heterogeneity (i.e. diversity of studies).

Is there heterogeneity? No Yes Meta-analysis Narrative synthesis Deal with heterogeneity? (e.g. subgroup analysis)

Guidelines for narrative synthesis Not available yet

Step 8 Interpret findings

Interpretation of results Strength of the evidence Explanations of effectiveness Applicability Trade-offs between benefits and harms Implications for practice These issues should be included in the discussion and recommendations section of a systematic review:

Strength of the evidence Describe the overall strength of the evidence The quality of the evidence The size and consistency of the results (particularly important in population based studies where a small effect at the community level may have much significance than the effect of comparable size at the individual level) Describe biases and limitations of the review process (e.g. difficulties in locating articles, inability to translate non-English articles)

8 steps of a systematic review Bent et al. 2004 1 Formulate research question Lecture 1 2 Develop review protocol 3 Initiate search strategy 4 Apply inclusion /exclusion criteria 5 Quality appraisal 6 Data abstraction 7 Analysis Lectures 1,2 & 3 8 Interpret findings

Section 1 Assignment Developing a Protocol for a Systematic Review Homework: Write a 1-2 page protocol to address the clinical problem described in the assignment.