EPI-214: Lecture 1 Designing a Systematic Review (Meta-analysis) Dejana Braithwaite Assistant Professor UCSF Department of Epidemiology and Biostatistics April 11, 2013

Agenda  Introduction and definitions  Steps of a systematic review  Issues/Controversies  Conclusions

What’s a Systematic Review? “A review of the evidence on a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant primary research, and to extract and analyze data from the studies that are included in the review.” Cochrane Collaboration

…and meta-analysis? Statistical combination of >= 2 studies to produce single estimate of effect of exposure

*IPD= individual participant data Systematic reviews Meta-analyses IPD*

The Cochrane Collaboration International systematic review initiative Archie Cochrane’s vision led to the opening of the first Cochrane centre (in Oxford, UK) in 1992 and the founding of the Cochrane Collaboration in 1993 Source:

Sattar NSattar N, et al. Statins and risk of incident diabetes: a collaborative meta-analysis of randomised statin trials. Lancet Feb 27;375(9716): etLancet. Meta-analysis news coverage

Systematic review  Driven by evidence-based medicine movement and Cochrane collaboration  Advantages:  Reduces bias  Replicable  Resolves controversy between conflicting findings  Provides reliable basis for decision making

Use of meta-analysis as a prelude to clinical trials  Define pre-trial expected effect sizes sample size estimation  Determine effect estimates in key subgroups (e.g. based on gender, race/ethnicity or age)  Identify sources of heterogeneity in prior studies  Address these sources in design phase of new trial

Use of meta-analysis in study designs that are not clinical trials  Observational studies  (e.g. case control, cohorts, cross-sectional prevalence studies, etc.)  Studies evaluating diagnostic tests (sensitivity, specificity, predictive value)  “IPD” = individual patient data studies  Qualitative studies (meta-ethnography)

Resources required for systematic reviewing  Can be time consuming  Team science (to reduce bias)  Bibliographic software (e.g. Endnote)  Statistical software (if appropriate)

*IE Allen & I Olkin: JAMA. 1999;282(7): doi: /pubs.JAMA-ISSN jbk0818 Citations Retrieved for a Meta-analysis and Total Hours Required to Complete the Meta-analysis* The mean total number of hours was 1139 (median, 1110), with a wide range from 216 to 2518 hours. (1)Pre-analysis search, retrieval, and database development: 588 (337) hours; (2)statistical analysis & validation: 144 (106) hours; (3)report and manuscript writing: 206 (125) hours; (4)other (administrative): 201 (193) hours. Total time= x − x 2, where x is the number of citations before exclusion criteria are applied. Total time= x − x 2, where x is the number of citations before exclusion criteria are applied.

1. Formulate research question Protocol 2. Identifying relevant work Apply inclusion /exclusion criteria Analysis 3. Assessing the quality of studies 4. Summarizing evidence 5 steps of a systematic review 5. Interpreting the findings

Free-form question: Is it safe to provide population-wide drinking water fluoridation to prevent caries? Structured question: The populations—Populations receiving drinking water sourced through a public water supply The interventions or exposures—Fluoridation of drinking water (natural or artificial) compared with non-fluoridated water The outcomes—Cancer is the main outcome of interest for the debate in your health authority The study designs—Comparative studies of any design examining the harmful outcomes in at least two population groups, one with fluoridated drinking water and the other without.

 Patient:  Disease or condition  Demographic characteristics  Intervention (or “Exposure”):  Type of intervention  Dose, duration, timing, etc.  Comparison:  Absence of risk or treatment  Placebo or alternative therapy Components of a research question (PICOT)  Outcome:  Risk or protective  Dichotomous or continuous  Type: mortality, quality of life, etc.  Type of Study:  RCTs  Cohort  Case-control  Cross-sectional  All

Formulation of an etiology question Is smoking a risk factor for breast cancer? Are people who smoke regularly at a greater risk of developing breast cancer as compared to those who do not smoke? Risk factor/ExposureOutcome ExposurePatient OutcomeComparison + cohort & case-control studies

Protocol  Background  Objectives  Pre-determined selection criteria  Planned search strategy  Planned data abstraction  Proposed method of synthesis of findings

1. Formulate research question Protocol 2. Identifying relevant work Apply inclusion /exclusion criteria Analysis 3. Assessing the quality of studies 4. Summarizing evidence 5 steps of a systematic review 5. Interpreting the findings

Where to locate studies At least two of these:  Pubmed  Web of Science  EMBASE  Cochrane Central Subject specific:  PsychINFO  CINAHL

Additional sources to identify studies for systematic reviews  Reference lists of retrieved articles  Manual searching of relevant publications  Experts in the field  Corresponding or first authors of published studies identified for the systematic review

Issues to consider  Publication bias  Search bias

Inclusion/exclusion criteria P - Population I - Intervention C - Comparison (if necessary) O - Outcome T - Type of study (if necessary) Subject headings OR Textwords To find studies using all of the PICO elements: P and I and C and O (and T)

Exclusion criteria  Keep log of excluded studies  Note reasons for exclusion  Have eligibility checked by more than one reviewer  Develop strategy to resolve disagreements

PRISMA-based search strategy flow-chart Braithwaite et al Breast Ca Res Treatm 2012

EXAMPLE: SEARCH STRATEGY You are interested in evaluating the benefits and harms associated with the use of screening mammography in women aged 70 and older. You need to determine what literature is available to conduct your review.

Quiz 1: You decide to conduct a systematic review (SR). What are two ways in which SRs differ from narrative reviews?  They are the same thing  Systematic reviews use comprehensive searches and explicit methods  Systematic reviews always employ quantitative synthesis (meta-analysis)

Quiz 1 answers: SRs versus narrative reviews  They are the Same – Incorrect since narrative reviews do not employ explicit methods  Comprehensive Search and Explicit Methods – Correct!  Meta-analysis – Incorrect since SRs may or may not employ quantitative synthesis

 You balanced precision and recall  You have employed every possible relevant citation regardless of the workload involved  That you limit the search so that only relevant citations are found Quiz 2: When conducting a comprehensive search, what do you need to assure?

Quiz 2 answers: When conducting a comprehensive search, what do you need to assure?  Balancing precision and recall - This is correct!  Precision - proportion of retrieved articles that are relevant.  Recall - proportion of potentially relevant articles retrieved by the search.  You’ve Gotten it All– incorrect since this is unnecessarily onerous  Only Have Relevant Citations in your Search – incorrect since overly restrictive strategy may miss citations

Quiz 4: You try to decide whether to do one search or two searches (one for benefits and another for harms). Which of the following would cause you to perform two searches?  Performing two searches is redundant and should be avoided  You decide a priori to only allow RCT data when evaluating benefits but allowing RCTs and observational trials for harms  If the available literature base on the topic is extensive

 Never Do Multiple Searches - This is incorrect. Extensive literature may necessitate multiple searches.  Separate Benefits and Harms Search - This is correct! Benefits searches usually limited to RCTs due to greater internal validity. Harms searches usually broader.  When the available literature is extensive, two searches improves precision without appreciably impacting recall. Quiz 4 answers: one versus multiple searches

Quiz 5: You decide to use hand searching of references from identified SRs, studies, and abstract booklets from prominent meetings within the field of interest. Why would you do this?  Hand searching can capture citations that are not indexed or are improperly indexed.  Limiting hand searching to these sources can target this time intensive activity to areas where the yield will be the greatest.

Quiz 5: Value of hand searching  Both are Correct. Hand searching can yield up to 13-25% of citations that do not come up in database searches.  Since hand searching is the most time intensive search strategy, it should be targeted to maximize the yield.

Quiz 6: You decide to search the FDA website and ClinicalTrials.gov to identify studies that might be appropriate for your systematic review. These website are an example of:  Grey literature  Informally published written material

Quiz 6 answers: Grey Literature  Grey Literature or informally published written material – both are correct!  This can be important for identifying studies that were conducted but are, as of now, unpublished or incompletely published.  Can minimize publication bias.

Search strategy - summary A comprehensive understanding of the topic, the use of multiple searches, multiple databases, use of hand searching and grey literature, and development of a transparent reporting structure will give readers confidence in your systematic review.

1. Formulate research question Protocol 2. Identifying relevant work Apply inclusion /exclusion criteria Analysis 3. Assessing the quality of studies 4. Summarizing evidence 5 steps of a systematic review 5. Interpreting the findings

Principles of quality assessment  Quantitative studies  Internal Validity  Allocation bias confounding  Attrition  Statistical analysis  Intervention integrity  Withdrawals and dropouts  External Validity  generalizability or applicability

Quality Scoring of Publications  Examines design and reporting  Jadad scoring technique (5 point Max)  Randomization (1 + 1 points)  Blinding (1 + 1 points)  Withdrawals (1 point) (1996, Controlled Clinical Trials) This followed the earlier 3 page quality scoring technique suggested by: Chalmers TC, Smith H, Blackburn B, Silverman B, Schroeder B, Reitman D, Ambroz A: A method for assessing the quality of a randomized control trial. Controlled Clinical Trials 2:31-49, 1981 Slide courtesy of I. Elaine Allen, PhD

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) statement  Evidence-based minimum set of items for reporting in SRs and meta-analyses  The aim of the PRISMA Statement is to help authors improve the reporting of SRs and meta-analyses  Focused on randomized trials but can also be used as a basis for reporting SRs of other types of research

Recruit participants Allocate to intervention and control groups Intervention groupControl group Implement intervention Follow-up participants Measure outcomes Analyze outcomes CONFOUNDING INTEGRITY OF INTERVENTION INTENTION-TO-TREAT WITHDRAWALS/ DROP OUTS BLINDING OUTCOME ASSESSORS DATA COLLECTION METHODS STATISTICAL ANALYSIS SELECTION BIAS ALLOCATION BIAS

1. Formulate research question Protocol 2. Identifying relevant work Apply inclusion /exclusion criteria Analysis 3. Assessing the quality of studies 4. Summarizing evidence 5 steps of a systematic review 5. Interpreting the findings

Data abstraction Design and pilot data abstraction form Consider >1 reviewer Consider blinding of observers to authors, institutions and journals

Data abstraction elements publication details study design population details (n, characteristics) intervention details setting outcomes and findings

Synthesizing the Evidence  NARRATIVE SYNTHESIS  findings summarized and explained qualitatively  META-ANALYSIS  f indings summarized and then combined statistically

Is there heterogeneity? NoYes Meta-analysisNarrative synthesis Deal with heterogeneity? (e.g. subgroup analysis)

1. Formulate research question Protocol 2. Identifying relevant work Apply inclusion /exclusion criteria Analysis 3. Assessing the quality of studies 4. Summarizing evidence 5 steps of a systematic review 5. Interpreting the findings

Interpretation of results  Strength of the evidence  Applicability (generalizability) of results  Trade offs between benefits, harms and costs (if applicable)  Implications for practice

Strength of the evidence  Rate how strong the overall case for the use or cessation of the intervention is:  H ow good is the quality of included trials?  How large and significant are the observed effects?  How consistent are the effects across trials?  Is there a dose-response relationship?  External factors  Biological plausibility  Other evidence  Concordance with related reviews

“Our meta-analysis data suggest that Chinese herbal medicine in the treatment of chronic hepatitis B infection may have potential therapeutic value; however, because the studies we found were of generally poor quality, we are unable to make firm conclusions.” Strength of the evidence: example Review on Chinese herbal medicine for hepatitis B McCulloch M et al. Chinese herbal medicine and interferon in the treatment of chronic hepatitis B: a meta-analysis of randomized controlled trials. Am J Pub Health 2002;92:

 To whom can the review results be applied?  Are there any compelling reasons why the evidence should not be applied under certain circumstances?  Biological issues  Cultural issues  Variation in baseline risk  Technology, skill, cost, etc. Applicability (generalizability) of results

Trade offs between benefits, harms and costs  Discuss adverse effects (potential for harm) E.g. compute NNH (number needed to harm that indicates how many patients need to be exposed to a risk factor over a specific time period to cause harm in a patient that would not have otherwise been harmed.  If possible, discuss cost issues  No need for a formal economic analysis!

For patient care or public health:  Review found no evidence at all or weak evidence  Review found evidence that clearly supports intervention  Review found clear evidence of lack of benefit  Review found clear evidence of potential for harm  Review found evidence of important trade-offs between known benefits and known adverse effects Implications of the review

“The currently available reliable evidence does not show a survival benefit of mass screening for breast cancer (and the evidence is inconclusive for breast cancer mortality), whereas it has been shown that mass screening leads to increased use of aggressive treatment. Women, clinicians and policy makers should consider these findings carefully when they decide whether or not to attend or support screening programs.” Olsen O et al. Example: Cochrane Mammography Review

Issues & Controversies Apples and oranges or ‘fruit salad’  Apples and oranges or ‘fruit salad’  Publication Bias (the file drawer problem)  Multiple Outcomes  Data from non-RCTs  Data from uncontrolled studies  Data from observational studies  Accumulating data and ‘stopping’ rules  Individual patient data

Systematic Reviews: Strengths and Limitations  Findings of a review of 300 studies  Not all systematic reviews equally reliable  Reporting could be improved by a standard set of guidelines  Out of 100 systematic reviews monitored:  7% needed updating at the time of publication  Another 4% within a year  Another 11% within 2 years**  Findings of a 2003 study:  Extending searches beyond major databases, perhaps into grey literature, would increase the effectiveness of reviews***

The aim of any Meta-Analysis: Courtesy I. Elaine Allen PhD

Your own systematic review How could you use a systematic review to answer a specific clinical research question? What are the challenges and risks?