RESEARCH SYNTHESIS: The Core Methodology of Evidence-Based Reviews Susan N. Labin, Ph.D.

Slides:



Advertisements
Similar presentations
Session 1 Introduction to course. Session 1 structure 1.Why are mental health promotion and mental disorder prevention important? 2. Contents of this.
Advertisements

Synthesizing the evidence on the relationship between education, health and social capital Dan Sherman, PhD American Institutes for Research 25 February,
A MERICAN I NSTITUTES FOR R ESEARCH An Analysis of the Research and Impact of ICT in Education in Developing Country Contexts Nitika Tolani-Brown, PhD.
Action Research Not traditional educational research often research tests theory not practical Teacher research in classrooms and/or schools/districts.
Potential of Public Health Systematic Reviews to Impact on Primary Research Professor Laurence Moore September 2007.
Developing a framework for evaluating qualitative research Liz Spencer, Jane Ritchie, Jane Lewis, Lucy Dillon NatCen Team 24 June 2004.
Practicing Community-engaged Research Mary Anne McDonald, MA, Dr PH Duke Center for Community Research Duke Translational Medicine Institute Division of.
Reading the Dental Literature
Utilizing Evidence Based Practice in the Acute Care Clinical Setting Brenda P. Johnson, PhD, RN Department of Nursing Southeast Missouri State University.
Childhood Obesity Scenario: Quasi- Experiments and Natural Experiments Versus RCTs Steven Gortmaker, Ph.D. Harvard School of Public Health /Harvard Prevention.
1 Teaching Cultural Competency: A Review of the Literature Sunita Mutha MD 1,2, Carol Allen MA 1, Cynthia Salinas MD 3, Arnab Mukherjea MPH 4 1 The Network.
Evidenced Based Practice; Systematic Reviews; Critiquing Research
Summarizing Community-Based Participatory Research: Background and Context for the Review Lucille Webb, MEd Eugenia Eng, DrPH Alice Ammerman, DrPH Meera.
Chapter 7. Getting Closer: Grading the Literature and Evaluating the Strength of the Evidence.
By Dr. Ahmed Mostafa Assist. Prof. of anesthesia & I.C.U. Evidence-based medicine.
Introduction to evidence based medicine
Violence and Injury Prevention Short Course Presentation 1. Public Health Approach to Child Maltreatment Prevention.
What is a Systematic review?. Systematic review  Combination of the best research projects in a specific area Selecting Identifying Synthesizing  Health.
Their contribution to knowledge Morag Heirs. Research Fellow Centre for Reviews and Dissemination University of York PhD student (NIHR funded) Health.
STUDY PLANNING & DESIGN TO ENHANCE TRANSLATION OF HEALTH BEHAVIOR RESEARCH Lisa Klesges, Russell Glasgow, Paul Estabrooks, David Dzewaltowski, Sheana Bull.
Research Designs for Complex Community Interventions for Childhood Obesity Prevention Robert W. Jeffery, Ph.D. Division of Epidemiology University of Minnesota.
Single-Case Research: Standards for Design and Analysis Thomas R. Kratochwill University of Wisconsin-Madison.
Allison Metz, Ph.D., Karen Blase, Ph.D., Dean L. Fixsen, Ph.D., Rob Horner, Ph.D., George Sugai, Ph.D. Frank Porter Graham Child Development Institute.
Program Evaluation. Program evaluation Methodological techniques of the social sciences social policy public welfare administration.
Research Designs Murray W. Enns Professor of Psychiatry.
Accountability in Health Promotion: Sharing Lessons Learned Management and Program Services Directorate Population and Public Health Branch Health Canada.
Evidence-Based Public Health Nancy Allee, MLS, MPH University of Michigan November 6, 2004.
Session I: Unit 2 Types of Reviews September 26, 2007 NCDDR training course for NIDRR grantees: Developing Evidence-Based Products Using the Systematic.
STANDARDS OF EVIDENCE FOR INFORMING DECISIONS ON CHOOSING AMONG ALTERNATIVE APPROACHES TO PROVIDING RH/FP SERVICES Ian Askew, Population Council July 30,
1 Sandy Keenan TA Partnership for Child and Family Mental Health(SOC) National Center for Mental Health Promotion and Youth Violence Prevention(SSHS/PL)
Secondary Translation: Completing the process to Improving Health Daniel E. Ford, MD, MPH Vice Dean Johns Hopkins School of Medicine Introduction to Clinical.
Systematic reviews to support public policy: An overview Jeff Valentine University of Louisville AfrEA – NONIE – 3ie Cairo.
Maria E. Fernandez, Ph.D. Associate Professor Health Promotion and Behavioral Sciences University of Texas, School of Public Health.
Why a CPCRN? CDC Expectations Katherine M. Wilson, PhD, MPH CPCRN Technical Monitor Division of Cancer Prevention and Control CDC.
Evidence-based Education and the Culture of Special Education Chair: Jack States, Wing Institute Discussant: Teri Palmer, University of Oregon.
Chapter 8 New Wave Research: Contemporary Applied Approaches.
Developing a Review Protocol. 1. Title Registration 2. Protocol 3. Complete Review Components of the C2 Review Process.
Evidence Based Practice RCS /9/05. Definitions  Rosenthal and Donald (1996) defined evidence-based medicine as a process of turning clinical problems.
META-ANALYSIS, RESEARCH SYNTHESES AND SYSTEMATIC REVIEWS © LOUIS COHEN, LAWRENCE MANION & KEITH MORRISON.
+ NASP’s Position Statement on Prevention and Intervention Research in the Schools Training School Psychologists to be Experts in Evidence Based Practices.
Sifting through the evidence Sarah Fradsham. Types of Evidence Primary Literature Observational studies Case Report Case Series Case Control Study Cohort.
An Expanded Model of Evidence-based Practice in Special Education Randy Keyworth Jack States Ronnie Detrich Wing Institute.
1 Lecture 10: Meta-analysis of intervention studies Introduction to meta-analysis Selection of studies Abstraction of information Quality scores Methods.
Finding, Evaluating, and Presenting Evidence Sharon E. Lock, PhD, ARNP NUR 603 Spring, 2001.
Massachusetts Cancer Prevention Community Research Network (MCPCRN) CPCRN Atlanta Meeting October 15-16, 2009.
1 Lecture 10: Meta-analysis of intervention studies Introduction to meta-analysis Selection of studies Abstraction of information Quality scores Methods.
Is a meta-analysis right for me? Jaime Peters June 2014.
Evidence-Based Public Health in Action: Strategies from New York Moderator: Amy Ramsay Association of State and Territorial Health Officials Speakers from.
Presentation Developed for the Academy of Managed Care Pharmacy
Evidence Review Methodological Framework: The Case of Teen Pregnancy Prevention Research April 2011 Presentation to the Inter-American Development Bank.
Copyright © 2010, 2006, 2002 by Mosby, Inc., an affiliate of Elsevier Inc. Chapter 10 Evidence-Based Practice Sharon E. Lock.
Evidence-Based Mental Health PSYC 377. Structure of the Presentation 1. Describe EBP issues 2. Categorize EBP issues 3. Assess the quality of ‘evidence’
Research Methods & Design Outline
Dr. Aidah Abu Elsoud Alkaissi An-Najah National University Employ evidence-based practice: key elements.
1 Copyright © 2012 by Mosby, an imprint of Elsevier Inc. Copyright © 2008 by Mosby, Inc., an affiliate of Elsevier Inc. Chapter 15 Evidence-Based Practice.
David M. Murray, Ph.D. Associate Director for Prevention Director, Office of Disease Prevention Multilevel Intervention Research Methodology September.
Building an Evidence-Based Nursing Practice
SYNTHESIS OF EVALUATION CAPACITY BUILDING LITERATURE
Evidence Based Practice Process
A Meta Analysis of the Impact of SBI on Healthcare Utilization
Independent International Consultant, Child Protection & Disability.
Randomized Trials: A Brief Overview
MeOTa fall conference October 22, 2016
H676 Meta-Analysis Brian Flay WEEK 1 Fall 2016 Thursdays 4-6:50
What are Mixed Methods? Donna M. Mertens, Ph.D., Professor Emeritus, Gallaudet University Economic and Social Research Council National Centre for Research.
Dr. Matthew Keough August 8th, 2018 Summer School
A Meta Analysis of the Impact of SBI on Healthcare Utilization
EVIDENCE-BASED PRACTICE
The CIT Model: Can We Call It Evidence-Based?
Meta-analysis, systematic reviews and research syntheses
Presentation transcript:

RESEARCH SYNTHESIS: The Core Methodology of Evidence-Based Reviews Susan N. Labin, Ph.D.

Susan N. Labin-AEA Research Synthesis: What & Why ? Secondary data analysis-aggregates findings from primary research Secondary data analysis-aggregates findings from primary research Establishes base of existing knowledge Establishes base of existing knowledge Informs future policies, programs, practices, & research Informs future policies, programs, practices, & research

Susan N. Labin-AEA Types of Syntheses Broad-Based: from variety of designs & types of data Broad-Based: from variety of designs & types of data –GAO –CDC: Guide to Community Preventive Services Meta-Analysis: only from randomized control trials (RCT) Meta-Analysis: only from randomized control trials (RCT)

Susan N. Labin-AEA Broad-Based Research Synthesis: GAO & CDC Moderately systematic Moderately systematic Qualitative & quantitative data Qualitative & quantitative data Draw from a variety of designs Draw from a variety of designs Consensus building-between academics and practitioners Consensus building-between academics and practitioners

Susan N. Labin-AEA Research Synthesis: Systematic Decision Rules Research Synthesis Research Synthesis –Broad-based and meta-analysis –Systematic decision rules –Six basic steps, e.g. selection & inclusion criteria Traditional Literature Reviews Traditional Literature Reviews –Broad-based –Less systematic decision rules

Susan N. Labin-AEA Brief History: Primary & Secondary Evaluation 1960’s – RCT “gold standard” from physical and biological sciences Evaluation 1960’s – RCT “gold standard” from physical and biological sciences 1970’s and 1980’s interest in synthesis and meta-analysis for social sciences 1970’s and 1980’s interest in synthesis and meta-analysis for social sciences Resolve contradictory findings Resolve contradictory findings Improve external validity or generalizability of small sample RCTs Improve external validity or generalizability of small sample RCTs Meta-Analysis physical & biological roots Meta-Analysis physical & biological roots

Susan N. Labin-AEA Evidence-Based Practice Reviews (EBPR) As synthesis-assess evidence to inform policy & practice As synthesis-assess evidence to inform policy & practice Who: predominately federal Who: predominately federal How same & different from synthesis? How same & different from synthesis? How can synthesis inform EBPR? How can synthesis inform EBPR?

Susan N. Labin-AEA EBPR Who Is Doing Them? 1. Agency Healthcare Research & Quality 2. Centers for Disease Control & Prevention-2 3. Office Juvenile Justice & Delinquency Prevention 4. Substance Abuse & Mental Health Services Administration 5. US Dept of Ed 6. Coalition for Evidence-Based Policy (non- federal)

Susan N. Labin-AEA Syntheses & EBPR: Common Steps & Decision-rules Many of same steps Many of same steps Quality ratings Quality ratings Inclusion-Design Inclusion-Design –Some restrict to only RCT –Others include quasi-experimental, observational –Some only RCT can get highest rating, most RCT highest ratings

Susan N. Labin-AEA EBPR Differs from Synthesis Inclusion-some EBPR include only those with positive outcomes Inclusion-some EBPR include only those with positive outcomes Only a few EBPR aggregate analyses across studies Only a few EBPR aggregate analyses across studies Less emphasis w EBPR on reliability between reviewers Less emphasis w EBPR on reliability between reviewers

Susan N. Labin-AEA Addressing Practice Challenges to RCT Social interventions linked with reinforcing social interactions or culture- address w group/place/cluster/saturation trials Social interventions linked with reinforcing social interactions or culture- address w group/place/cluster/saturation trials Assumptions re replicability of protocol or intervention Assumptions re replicability of protocol or intervention Feasibility Feasibility

Susan N. Labin-AEA RCT Challenges for Social & Behavioral Sciences Inherent differences between physical and biological sciences and social and behavioral sciences Inherent differences between physical and biological sciences and social and behavioral sciences Lack of blinding and double blinding Lack of blinding and double blinding Relevance of individual choice Relevance of individual choice

Susan N. Labin-AEA Comparing Results: RCT versus Quasi Designs Complex: quasi may underestimate (Shadish) Complex: quasi may underestimate (Shadish) Same labor market, same questionnaire quasi & exp Same labor market, same questionnaire quasi & exp give “fairly close” results give “fairly close” results

Susan N. Labin-AEA Comparing Results: RCT vs Quasi Designs cont. Lipsey & Wilson (2001) Lipsey & Wilson (2001) –> 300 M-A; > 16,000 ind studies –Exp vs Quasi not much difference –One-group pre-post over estimates compared to quasi –Measurement =.07 proportion of variance –Exp vs Quasi=.04

Susan N. Labin-AEA How To Strengthen EBPR? Include negative results to inform counter- factual Include negative results to inform counter- factual –How consistent results –What else could be contributing to the effects Participatory and transparent reviews Participatory and transparent reviews –Quality criteria and decision rules –Reliability & resolving discrepancies Common measurement-include criteria of respondent meanings Common measurement-include criteria of respondent meanings

Susan N. Labin-AEA How To Strengthen EBPR cont. Aggregate across projects to improve Aggregate across projects to improve –Generalizability –> Confidence in “best practice” designation –Choice of projects for replication –Knowledge Base Develop & use technologies for categorizing studies and depicting patterns of results. Develop & use technologies for categorizing studies and depicting patterns of results.

Susan N. Labin-AEA References Labin, S Research Syntheses: Toward Broad-Based Evidence in Fundamental Issues in Evaluation, (eds.) Nick L. Smith and Paul R. Brandon, Guilford Press, NY, NY. Labin, S Research Syntheses: Toward Broad-Based Evidence in Fundamental Issues in Evaluation, (eds.) Nick L. Smith and Paul R. Brandon, Guilford Press, NY, NY. GAO-30-10, Program Evaluation: A Variety of Methods Can Help Identify Effective Interventions GAO-30-10, Program Evaluation: A Variety of Methods Can Help Identify Effective Interventions Websites for the Effective Practice Reviews Websites for the Effective Practice Reviews

Susan N. Labin-AEA THANK YOU Susan N. Labin, Ph.D