The Campbell Collaborationwww.campbellcollaboration.org C2 Training: May 9 – 10, 2011 Introduction to Systematic Reviews.

Slides:



Advertisements
Similar presentations
Critical Reading Strategies: Overview of Research Process
Advertisements

Critical Reading Strategies: Overview of Research Process
What is a review? An article which looks at a question or subject and seeks to summarise and bring together evidence on a health topic.
Study Objectives and Questions for Observational Comparative Effectiveness Research Prepared for: Agency for Healthcare Research and Quality (AHRQ)
Protocol Development.
Student Learning Development, TCD1 Systematic Approaches to Literature Reviewing Dr Tamara O’Connor Student Learning Development Trinity College Dublin.
8. Evidence-based management Step 3: Critical appraisal of studies
Introduction to Research Methodology
Reading the Dental Literature
Reviewing and Critiquing Research
The Campbell Collaborationwww.campbellcollaboration.org Education Panel Session Comments and Points for Discussion Sandra Jo Wilson Vanderbilt University.
Introduction to Meta-Analysis Joseph Stevens, Ph.D., University of Oregon (541) , © Stevens 2006.
Evidenced Based Practice; Systematic Reviews; Critiquing Research
Introduction to Research
Business research methods: data sources
The Campbell Collaborationwww.campbellcollaboration.org C2 Training: May 9 – 10, 2011 Introduction to Systematic Reviews.
Campbell Collaboration Colloquium 2012 Copenhagen, Denmark The effectiveness of volunteer tutoring programmes Dr Sarah Miller Centre.
Critical Appraisal of an Article by Dr. I. Selvaraj B. SC. ,M. B. B. S
Chapter 2 Understanding the Research Process
September 26, 2012 DATA EVALUATION AND ANALYSIS IN SYSTEMATIC REVIEW.
Proposal Writing.
Developing Research Proposal Systematic Review Mohammed TA, Omar Ph.D. PT Rehabilitation Health Science.
1 The Literature Review March 2007 (3). 2 The Literature Review The review of the literature is defined as a broad, comprehensive, in- depth, systematic,
The Campbell Collaborationwww.campbellcollaboration.org C2 Training: May 9 – 10, 2011 Data Analysis and Interpretation: Computing effect sizes.
Systematic Approaches to Literature Reviewing. The Literature Review ? “Literature reviews …… introduce a topic, summarise the main issues and provide.
Advanced Statistics for Researchers Meta-analysis and Systematic Review Avoiding bias in literature review and calculating effect sizes Dr. Chris Rakes.
Program Evaluation. Program evaluation Methodological techniques of the social sciences social policy public welfare administration.
Systematic Reviews.
The Campbell Collaborationwww.campbellcollaboration.org C2 Training: May 9 – 10, 2011 Data Evaluation: Initial screening and Coding Adapted from David.
Evaluating a Research Report
Evidence Based Medicine Meta-analysis and systematic reviews Ross Lawrenson.
Major Types of Quantitative Studies Descriptive research –Correlational research –Evaluative –Meta Analysis Causal-comparative research Experimental Research.
Identifying the evidence Laura Macdonald Health Protection Scotland
Session I: Unit 2 Types of Reviews September 26, 2007 NCDDR training course for NIDRR grantees: Developing Evidence-Based Products Using the Systematic.
Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved.
Appraising Randomized Clinical Trials and Systematic Reviews October 12, 2012 Mary H. Palmer, PhD, RN, C, FAAN, AGSF University of North Carolina at Chapel.
Systematic reviews to support public policy: An overview Jeff Valentine University of Louisville AfrEA – NONIE – 3ie Cairo.
Quantitative and Qualitative Approaches
1 Copyright © 2011 by Saunders, an imprint of Elsevier Inc. Chapter 8 Clarifying Quantitative Research Designs.
Focusing the question Janet Harris Cochrane Qualitative Research Methods Group ESQUIRE Qualitative Systematic Review Workshop University of Sheffield 6.
LITERATURE REVIEW  A GENERAL GUIDE  MAIN SOURCE  HART, C. (1998), DOING A LITERATURE REVIEW: RELEASING THE SOCIAL SCIENCE RESEARCH IMAGINATION.
Developing a Review Protocol. 1. Title Registration 2. Protocol 3. Complete Review Components of the C2 Review Process.
META-ANALYSIS, RESEARCH SYNTHESES AND SYSTEMATIC REVIEWS © LOUIS COHEN, LAWRENCE MANION & KEITH MORRISON.
Copyright © Allyn & Bacon 2008 Intelligent Consumer Chapter 14 This multimedia product and its contents are protected under copyright law. The following.
Guidelines for Critically Reading the Medical Literature John L. Clayton, MPH.
Educational Research Chapter 3. Research Problem Systematic Research begins with a research problem - begin with a general topic and then narrow it down.
به نام او که انسان را به زیور « اندیشه » و « تفکر » آراست.
Sifting through the evidence Sarah Fradsham. Types of Evidence Primary Literature Observational studies Case Report Case Series Case Control Study Cohort.
Focusing the question Janet Harris
Introduction to Research. Purpose of Research Evidence-based practice Validate clinical practice through scientific inquiry Scientific rational must exist.
Research article structure: Where can reporting guidelines help? Iveta Simera The EQUATOR Network workshop 10 October 2012, Freiburg, Germany.
Systematic Review Krit Pongpirul, MD, MPH. Johns Hopkins University.
Family-based Prevention of Offending: A Meta-Analysis David P. Farrington & Brandon C. Welsh 2003 Jenna Ayers Radford University.
Conducting Research in the Social Sciences (From: Individuals and Families: A Diverse Perspective (2010))
Evidence Based Practice (EBP) Riphah College of Rehabilitation Sciences(RCRS) Riphah International University Islamabad.
Copyright © 2012 Wolters Kluwer Health | Lippincott Williams & Wilkins Chapter 27 Systematic Reviews of Research Evidence: Meta-Analysis, Metasynthesis,
Cedric D. Murry APT Instructor of Applied Technology in research and development.
Dr. Aidah Abu Elsoud Alkaissi An-Najah National University Employ evidence-based practice: key elements.
Systematic Reviews of Evidence Introduction & Applications AEA 2014 Claire Morgan Senior Research Associate, WestEd.
Week Seven.  The systematic and rigorous integration and synthesis of evidence is a cornerstone of EBP  Impossible to develop “best practice” guidelines,
Best Practice Systematic Review
NURS3030H NURSING RESEARCH IN PRACTICE MODULE 7 ‘Systematic Reviews’’
The Research Design Continuum
Supplementary Table 1. PRISMA checklist
Systematic Review (Advanced_Course_Module_6_Appendix)
Introduction to Systematic Reviews and Meta-analysis
Meta-analysis, systematic reviews and research syntheses
Research design and techniques Workshop ICBEDC 2010
Systematic Review (Advanced Course: Module 6 Appendix)
META-ANALYSIS PROCEDURES
Presentation transcript:

The Campbell Collaborationwww.campbellcollaboration.org C2 Training: May 9 – 10, 2011 Introduction to Systematic Reviews

C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Systematic Review Methods Systematic reviews are a form of research – secondary observations – in which studies are the unit of analysis Follow basic steps in the research process Aim to minimize bias and error – But SRs are not immune to bias and error (not a panacea)

C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org 3 Stages of a research synthesis (Cooper, 1982) The seminal article outlining five stages of a research review is by Harris Cooper: Cooper, H.M. (1982). Scientific guidelines for conducting integrative research reviews. Review of Educational Research, 52, Cooper, H. M. (2009). Research synthesis and meta- analysis: A step-by-step approach. Thousand Oaks, CA: Sage.

C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org 4 Stages of a research synthesis (Cooper, 1982) Problem formulation – Clarifying your questions and writing a protocol – Set explicit inclusion/exclusion criteria Data collection – Literature search – Information-gathering from studies Data evaluation – Criteria for including and excluding studies – Assessing study quality

C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org 5 Stages of a research synthesis (Cooper, 1982) Data analysis and interpretation – Integrating the effects from collected studies – Interpreting analysis results Report preparation – Narrative, statistics, graphs, tables

C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org What’s required? A team with – Substantive expertise – Methodological expertise – Statistical expertise – Information retrieval expertise Time and money – SRs are labor intensive – $50-$150K depending on scope, complexity, and number of studies

C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Protocols for SRs A detailed protocol (plan) for the SR should be developed and made available to readers (Higgins & Green, 2008; Moher et al., 2009) – Protocols increase transparency, limit ad hoc decisions The review process is iterative and plans may change during the process – The final report should document and explain changes made (deviations from the protocol)

C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org 8 Stages of a research synthesis Most of the work involved in conducting a review is not spent in statistical analysis. The scientific contribution of the final product is dependent on all stages of the review and not just the statistical analysis stage.

C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org 9 An example of the stages of a review We will illustrate each of the stages by considering the C2 systematic review on Multisystemic Therapy (MST): Littell, J.H., Popa, M., & Forsythe, B. (2005). Multisystemic Therapy for social, emotional, and behavioral problems in youth aged 10 – 17. (Campbell version)

C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org 10 Stage 1 Problem Formulation

C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Problem formulations Conceptual issues arise in attempts to combine results of studies that vary (to some degree) in their methods, treatments, samples, outcome measures – Apples, oranges, and other fruits – Parallel problems in studies of individuals (no 2 people are identical and there is no “average” person) Given variation in primary research – What should be included in a synthesis? – How should we synthesize results?

C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org SRs vary in scope Specific, narrow questions – Useful for testing effects of specific treatments Broad, global questions – Useful for generating new knowledge Identify common elements of effective programs (Lipsey, 2008) build better intervention theories to guide program development and evaluation design (Lipsey, 1997)

C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Scope of Systematic Reviews Not limited to questions about effects – Can address trends, epidemiology, accuracy of diagnostic and prognostic tests Not limited to randomized controlled trials or quantitative data – Qualitative synthesis (e.g., meta-ethnography, narrative analysis of qualitative research reports) – Mixed/multiple methods synthesis (e.g., Thomas, Harden, et al. on progams to combat childhood obesity)

C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org What kinds of research questions can be asked in a systematic review? Questions about intervention effects: – What are the effects of x intervention on y outcomes for z populations/problem? – Variations on this theme (e.g., differences in effects of interventions x 1 vs x 2 )

C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Kinds of research questions (continued) Questions about associations – How does x 1 relate to x 2 for population z? (direction and strength of correlation) – Variations on this theme (e.g., differences in relation of x 1 and x 2 between populations z 1 and z 2 ) Diagnostic/Prognostic questions – Which test (A vs. B) is a better predictor of y? – Which test (A vs. B) is a better predictor of y for z 1 vs. z 2 populations?

C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org 16 Problem formulation To assess the impacts of MST on out-of-home living arrangements, crime and delinquency, and other behavioral and psychosocial outcomes for youth and families.

C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org 17 Problem formulation What research evidence will be relevant to the problem or hypothesis of interest in the synthesis? – Studies where youth (age 10-17) with social, emotional, and/or behavioral problems were randomised to licensed MST programs or other conditions (usual services or alternative treatments).

C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org 18 Problem formulation Define the (a) variables and (b) relationships of interest so that relevant and irrelevant studies can be distinguished. Multisystemic Therapy (MST) is a multi-faceted, short-term, home- and community-based intervention for families of youth with severe psychosocial and behavioral problems. Based on social ecological and family systems theories, and on research on the causes and correlates ofserious antisocial behavior in youth (Henggeler 1998, Henggeler 2002a), MST is designed to address complex psychosocial problems and provide alternatives to out-of-home placement of children and youth.

C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Problem formulation (continued) Measures of behavioral, psychosocial, and family outcomes were examined. – Behavioral outcomes included antisocial behavior (as measured by arrest or conviction of a criminal offense), drug use (self-reports and drug tests), and school attendance. – Psychosocial outcomes included measures of psychiatric symptoms (on standardized scales),school performance (teacher reports), peer relations (self-reports and parent or teacher reports),and self esteem. – To assess the impacts of MST on out-of-home living arrangements, crime and delinquency, and other behavioral and psychosocial outcomes for youth and families.

C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Step two: Establish study inclusion and exclusion criteria The PICOS framework: – P opulation/Participants (problems/conditions) – I nterventions (if applicable) – C omparison group (e.g., absolute vs. relative effects, counterfactual conditions) – O utcomes (primary and secondary outcomes, acceptable outcome measures) – S tudy Design (should be fit for purpose) Geographic area, time, language, other criteria

C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Participants: Who is included in the sample of subjects? Why only? Exclude special populations? Exclude studies with only males or females? Include families of these children? What is the rationale for the choice?

C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Interventions: How do we define MST? Comparisons: What is the comparison? What kinds of programs are considered MST? What types of treatments are the “right” comparisons to MST?

C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org What are the Outcomes? Research question states: – Placement in out-of-home living arrangements – Crime and delinquency – Behavioral outcomes – Psychosocial outcomes

C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Study Designs: What types of study designs are relevant for our review? If the research question is the assessment of the impacts of MST on out-of-home living arrangements, crime and delinquency, and other behavioral and social outcomes for youth and families, what designs are most appropriate?

C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org What studies are relevant? Will we attempt to generalize to all studies done? Will we look only at “high quality” studies? Will we look at both published and unpublished? Will we set a time period for the studies? Will we look at only randomized controlled trials or will we include quasi-experimental studies? Why?

C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Study eligibility criteria Usually focus on inclusion criteria, with few exclusion criteria Studies should not be excluded based on: – Sample size (or statistical power considerations) – Publication status

C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org What kinds of studies should be included? Garbage in, garbage out Synthesis of invalid studies produces invalid conclusions What constitutes credible evidence? – Depends on the question Need to set clear standards – Based on methodological principles – Relevant for topic and context

C2 Training Materials – Oslo – May 2011www.campbellcollaboration.org Small group exercise: Problem formulation Form a working group of 2 – 5 people and develop a preliminary research question for a comprehensive review Elect a recorder for the group who will report back to the whole Each group will share the research question, and any challenges that they will face in studying this question See the first exercise on problem formulation