1. 1 DrPH Seminar Session 1 Use of Systematic Review in Public Health Policy & Getting Started Defining SR Questions Mei Chung, PhD, MPH Research Assistant Professor Nutrition/Infection Unit, Department of Public Health and Community Medicine, Tufts School of Medicine

2. 2 Going Over the Syllabus Class Website http://sites.tufts.edu/systematicreviews/home -page/ All pre-readings, text books (optional readings), homework assignments, and online discussion activities are available on the class Website

3. 3 Topics to discuss Very quick overview of steps/methods of a systematic review –Systematic review versus traditional (narrative) review How to formulate a systematic review research question –Analytic framework (conceptual framework) How to evaluate a systematic review Debates: scientific value of a systematic review (with or without meta-analysis)

4. 4 Evidence-based X Evidence-based public health is defined as the development, implementation, and evaluation of effective programs and policies in public health through application of principles of scientific reasoning, including systematic uses of data and information systems, and appropriate use of behavioral science theory and program planning models. Source: Brownson, Ross C., Elizabeth A. Baker, Terry L. Leet, and Kathleen N. Gillespie, Editors. Evidence-Based Public Health. New York: Oxford University Press, 2003.

5. 5 Evidence-based X Evidence-based dietetic practice is the use of systematically reviewed scientific evidence in making food and nutrition practice decisions by integrating best available evidence with professional expertise and client values to improve outcomes. Source: Academy Scope of Dietetics Practice Framework https://www.andeal.org/evidence-analysis-process-overview

6. 6 Organizations producing systematic reviews or using systematic review to inform evidence-based policy and practice guidelines http://sites.tufts.edu/systematicreviews/mainpage/

7. 7 Synthesizing Evidence Narrative Reviews Systematic Reviews Meta-Analysis Decision Analysis Cost-effectiveness analysis Clinical practice guidelines Algorithms

8. 8 What is a Systematic Review? (Sometimes called systematic evidence-based reviews or evidence review) Systematic review –a comprehensive summary and synthesis of all available evidence that meets predefined eligibility criteria to address a specific research question or range of questions Meta-analysis –commonly included in systematic reviews, a statistical method that quantitatively combines the results from different studies

9. 9 A systematic review ≠ a meta-analysis ≠ a (narrative) review with or without a systematic literature search All reviews Systematic reviews Meta- analysis

10. 10

11. 11 Basic Steps in a SR Prepare topic Search for studies Screen studiesExtract data Analyze and synthesize data Apply qualitative and/or quantitative methods Report findings

12. 12 Systematic review process flowchart

13. 13 Systematic review and meta-analysis is typically a retrospective exercise, suffering from all the limitations of being an observational design.

14. 14 For all research, and for systematic review in particular, a clear research question is needed An important clinical/public health question might not be a meaningful research question

15. 15 Ask Identify Acquire Appraise Synthesize

16. 16 Example - poorly formulated question: Should dietary supplements be recommended to patients with hypertension?

17. 17

18. 18 PICO approach to formulating answerable research question Counsell, 1997 Ppopulation Iintervention (or exposure) Ccomparator Ooutcomes D study design You’ll also see PICO, PICOS (study design), PICODD (+duration), PICOT (time), and others

19. 19 PI(E)COS approach to formulating answerable research question What is the relevant population? What is the intervention/exposure of interest? What is the appropriate comparison? What are the important outcomes of interest? In what setting would the results be applicable? You’ll also see PICOT (timing), PICOD (design/duration), and others

20. 20 The PICO method to formulate research question on interventions PopulationInterventions / Exposure ComparatorOutcomes Primary prevention Fish, fish oil, ALA PlaceboOverall mortality Secondary prevention DosageNo controlSudden death Background intake Active comparator Revascular- ization DurationStroke Blood pressure

21. 21 Populations Problems with defining condition Varying definitions –lack of an adequate reference standard (e.g., patients with anemia; patients with metabolic syndrome) Different levels of rigor –Loose vs. strict definitions (e.g. elderly vs. adults men and women who are 60 years old or grater) –Applicability/generalizability tradeoffs

22. 22 Example – population of interest Primary Prevention - patients without prior history of cardiovascular disease –Country –Background diet Secondary Prevention – patients with prior history of cardiovascular disease

23. 23 Example – Intervention / Exposure of Interest What is an omega-3 fatty acid? EPA, DHA, (fish oil, fish) –Levels differ by type of fish –Levels (and/or effect) may differ by preparation (broiled, fried fish sandwiches) ALA (plant source: walnut, canola oil, mustard seed, etc.)

24. 24 Example – Outcomes of interest Hard outcomes (clinical events) –Overall mortality –Stroke –Myocardial infarction –Sudden death –Revascularization Soft [surrogate, intermediate] outcomes (biomarkers, measurements) –Coronary flow –Blood pressure –Lipid levels Intermediate –Diagnosis of hypertension

25. 25 Analytic Framework Series of specific questions can be formulated into a model that analyzes all effects and interactions between intervention or exposure and outcomes Analytic framework can be used to clarify and generate questions (topics) Can highlight what aspects are known and unknown Can clarify what study designs may be best to address specific questions 25

26. 26 http://www.uspreventiveservicestaskforce.org/Page/Name/uspstf- analytic-framework-for-older-adults-health-topics USPSTF Analytic Framework for Older Adults Health Topics

27. 27 Omega-3 fatty acids and CVD 27

28. 28 Ask Identify Acquire Appraise Synthesize Will be covered in session 2

29. 29 PRISMA 2009 Flow Diagram 29 Records identified through database searching (n = ) Screening Included Eligibility Identification Additional records identified through other sources (n = ) Records after duplicates removed (n = ) Records screened (n = ) Records excluded (n = ) Full-text articles assessed for eligibility (n = ) Full-text articles excluded, with reasons (n = ) Studies included in qualitative synthesis (n = ) Studies included in quantitative synthesis (meta-analysis) (n = ) From: Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group (2009). Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med 6(6): e1000097. doi:10.1371/journal.pmed1000097

30. 30 Goals of Data Extraction & Quality Assessment Data extraction –To collect key study characteristics and results from published articles pertaining to the SR research question –Important to use a standardized form, customized for the SR research question Quality / risk of bias assessment –To avoid “Garbage in, garbage out” –To assess the confidence in the validity of study findings

31. 31 Rationale for Quality Assessment Quality assessment of all studies included in the SRs is important: –estimate extent to which study’s design and methods prevented systematic errors (biases) –variation in quality may explain differences in results of SRs –necessary even if there is little variability among studies (consistent trash is still trash)

32. 32 Tools for Quality/Risk of Bias Assessment Many tools, but few “validated” tools: –Cochrane risk of bias assessments (http://bmg.cochrane.org/assessing-risk-bias-included- studies): RCTshttp://bmg.cochrane.org/assessing-risk-bias-included- studies –The Newcastle-Ottawa Scale (http://www.ohri.ca/programs/clinical_epidemiology/oxford.as p): Observational studieshttp://www.ohri.ca/programs/clinical_epidemiology/oxford.as p No well-accepted nutrition/public health specific [content specific] quality assessment tools –Lichtenstein AH, Yetley EA, Lau J. Application of systematic review methodology to the field of nutrition. J Nutr 2008;138:2207-2306

33. 33 Linking Quality Assessment to Analysis as a threshold for inclusion and exclusion of studies in the review (generally not recommended) as a possible explanation for differences in results between studies as a variable in sensitivity analysis (test of robustness) as weights in statistical analysis of the results

34. 34 Ask Identify Acquire Appraise Synthesize

35. 35 Qualitative & Quantitative Syntheses Qualitative synthesis – required –Summary tables (many different forms) Key study characteristics Summary of study results –Graphical presentation of study results (a plus) –Assess strength of body of evidence Quantitative synthesis (meta-analysis) - optional –highly depending on types of results/data, and reporting of the data –may not be appropriate – to pool or not to pool can be a tricky decision

36. 36 An Example of an Assessment of Strength of Body of Evidence High High level of assurance with the validity of the results (based on quality, applicability, effect size, consistency) for the key question At least 2 high quality studies with long-term followup No important disagreement across studies Moderate Good to moderate level of assurance with the validity of the results Fewer than 2 high quality studies Little disagreement across studies in the results Low Low level of assurance with the validity of results Based on studies of moderate to poor quality or limited applicability Insufficient Little data or disagreement across or within studies 36

37. 37 Combine data from multiple studies to illustrate trends in the data May be focused on describing study characteristics, results, or both Can be designed to include characteristics of all included studies –Examples: funding sources, assessment method, country of study Can be designed for subsets of included studies –Examples: summary tables for randomized controlled trials, prevalence studies, harms/side effects, outcomes for specific treatments Summary Tables (I)

38. 38 Simplified entry (one row) for each study Table columns may include, for example: –PICOTS (may be listed in table title or headers) –Methodological quality –Applicability –Study size (weight) –Magnitude of effect A single study may be represented in multiple summary tables (e.g., different outcomes) Summary Tables (II) PICOTS = population, intervention, comparator, outcomes, timing, and setting

39. 39 Example: Summary Table of Study Characteristics A basic summary table is the “study characteristics” table. The overall summary provides an overview of the state of the available studies in the literature. Hartmann KE, et al. AHRQ Evidence Report/Technology Assessment No. 187. Available at: http://www.ahrq.gov/downloads/pub/evidence/pdf/ bladder/bladder.pdf.

40. 40 Example: Summary Table of Study Characteristics (More descriptive, most common)

41. 41 Example: Summary Table for Cohort Studies Wang C, et al. AHRQ Evidence Report/Technology Assessment No. 94. http://www.ahrq.gov/downloads/pub/evidence/pdf/o3cardio/o3cardio.pdf. Available at: http://www.ahrq.gov/downloads/pub/evidence/pdf/o3cardio/o3cardio.pdf.

42. 42 Example: Summary tables can be specialized for different types of outcomes

43. 43 Summary Matrix Wang C, et al. AHRQ Evidence Report/Technology Assessment No. 94. Available at: http://www.ahrq.gov/downloads/pub/evidence/pdf/o3cardio/o3cardio.pdf.

44. 44 Example: Graphical presentation of the study results

45. 45 Summary tables provide key information on study characteristics and study findings. Through table and graphical formats, respectively Properly constructed summary tables: –Effectively convey results –Provide an overview of the literature in a given field –Enable the reader to grasp results for subsets of the literature Key Messages

46. 46 Meta-Analysis Commonly included in systematic reviews, a statistical method that quantitatively combines the results from different studies Other names for meta- analysis –Quantitative overview/Synthesis –Pooling Less precise Suggests that data from multiple sources are simply lumped together –Combining Preferred by some Suggests applying statistical procedures to data

47. 47 Reading a Generic Forest Plot Reference: Szajewska H. The role of meta-analysis in the evaluation of the effects of early nutrition on mental and motor development in children. Am J Clin Nutr. 2011 Dec;94(6 Suppl):1889S-1895S. Epub 2011 Apr 27. Review

48. 48 Reasons for Meta-analysis Get an overall estimate of effect size –But, getting one answer may not be appropriate Appreciate the degree of uncertainty Appreciate heterogeneity* Forces you to think rigorously about the data 48

49. 49 Heterogeneity of Data ~Diversity~ Clinical Are studies of similar treatments, populations, settings, design etc. such that an average effect would be clinically meaningful? Methodological Are studies of similar design and conduct such that an average effect would be clinically meaningful? Statistical Is the observed variability of effects greater than that expected by chance alone? Are the characteristics and effects of studies sufficiently similar to estimate an average effect?

50. 50 Summary: A systematic review vs. narrative review

51. 51 RCT Observational Systematic review of RCTs

52. 52 PRISMA Checklist

53. 53 PRISMA Checklist

54. 54 PRISMA Checklist

55. 55

56. 56 PRISMA Checklist

57. 57 PRISMA Checklist

58. 58 Scientific Value of Systematic Reviews: Survey of Editors of Core Clinical Journals Reference: Meerpohl JJ et al. 2012 http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0035732

59. 59

60. 60 References IOM (Institute of Medicine). 2011. Finding What Works in Health Care: Standards for Systematic Reviews, Washington, DC: Natl. Acad. Press IOM (Institute of Medicine). 2011. Clinical Practice Guidelines We Can Trust. Washington, DC: The National Academies Press. Systematic reviews: synthesis of best evidence for clinical decisions. Ann Intern Med. 1997 Mar 1;126(5):376-80.