Practical Meta-Analysis -- D. B. Wilson 1 Practical Meta-Analysis David B. Wilson.

Slides:



Advertisements
Similar presentations
EVAL 6970: Meta-Analysis Introduction to Meta-Analysis Dr. Chris L. S. Coryn Kristin A. Hobson Fall 2013.
Advertisements

Practical Meta-Analysis
Effect Size Overheads1 The Effect Size The effect size (ES) makes meta-analysis possible. The ES encodes the selected research findings on a numeric scale.
Introduction to Research Methodology
Reading the Dental Literature
Funded through the ESRC’s Researcher Development Initiative Department of Education, University of Oxford Session 2.3 – Publication bias.
MSS 905 Methods of Missiological Research
Introduction to Meta-Analysis Joseph Stevens, Ph.D., University of Oregon (541) , © Stevens 2006.
Critical appraisal of the literature Michael Ferenczi Head of Year 4 Head of Molecular Medicine Section, National Heart and Lung Institute.
Research Synthesis (Meta-Analysis) Research Synthesis (Meta-Analysis) CHAPTER 1 CHAPTER 10.
Evidenced Based Practice; Systematic Reviews; Critiquing Research
Friday, November 14 and Monday, November 17 Evaluating Scientific Argument: Peer Review IPHY 3700 Writing Process Map.
PSY 1950 Meta-analysis December 3, Definition “the analysis of analyses... the statistical analysis of a large collection of analysis results.
Meta Analysis & Programmatic Research Re-introduction to Programmatic research Importance of literature reviews Meta Analysis – the quantitative contribution.
PPA 415 – Research Methods in Public Administration Lecture 1 – Research Design.
Practical Meta-Analysis -- D. B. Wilson
Personality, 9e Jerry M. Burger
Today Concepts underlying inferential statistics
Using Statistics in Research Psych 231: Research Methods in Psychology.
Chapter 7. Getting Closer: Grading the Literature and Evaluating the Strength of the Evidence.
Meta-Analysis. Why Meta-Analysis? There is an urgent need for reliable summaries of primary research in music therapy. 1. Music therapists can not keep.
Three Common Misinterpretations of Significance Tests and p-values 1. The p-value indicates the probability that the results are due to sampling error.
Practical Meta-Analysis -- D. B. Wilson1 Practical Meta-Analysis David B. Wilson Evaluators’ Institute July 16-17, 2010.
Chapter 4 Principles of Quantitative Research. Answering Questions  Quantitative Research attempts to answer questions by ascribing importance (significance)
Chapter 2: The Research Enterprise in Psychology
CHAPTER 3: DEVELOPING LITERATURE REVIEW SKILLS
Chapter 2: The Research Enterprise in Psychology
Chapter 2 The Research Enterprise in Psychology. n Basic assumption: events are governed by some lawful order  Goals: Measurement and description Understanding.
OKU 9 Chapter 15: ORTHOPAEDIC RESEARCH Brian E. Walczak.
Meta Analysis MAE Course Meta-analysis The statistical combination and analysis of data from separate and independent studies to determine if there.
Chapter 3 An Overview of Quantitative Research
Practical Meta-Analysis -- The Effect Size -- D. B. Wilson 1 The Effect Size The effect size (ES) makes meta-analysis possible The ES encodes the selected.
Unanswered Questions in Typical Literature Review 1. Thoroughness – How thorough was the literature search? – Did it include a computer search and a hand.
Advanced Statistics for Researchers Meta-analysis and Systematic Review Avoiding bias in literature review and calculating effect sizes Dr. Chris Rakes.
Program Evaluation. Program evaluation Methodological techniques of the social sciences social policy public welfare administration.
The Effect of Computers on Student Writing: A Meta-Analysis of Studies from 1992 to 2002 Amie Goldberg, Michael Russell, & Abigail Cook Technology and.
PTP 560 Research Methods Week 8 Thomas Ruediger, PT.
Epistemology and Methods Small-N and Large-N Studies May
Systematic Reviews.
Introduction to Meta-Analysis a bit of history definitions, strengths & weaknesses what studies to include ??? “choosing” vs. “coding & comparing” studies.
Copyright © Allyn & Bacon 2008 Locating and Reviewing Related Literature Chapter 3 This multimedia product and its contents are protected under copyright.
Chapter 3 Copyright © Allyn & Bacon 2008 Locating and Reviewing Related Literature This multimedia product and its contents are protected under copyright.
User Study Evaluation Human-Computer Interaction.
Evidence Based Medicine Meta-analysis and systematic reviews Ross Lawrenson.
Today: Our process Assignment 3 Q&A Concept of Control Reading: Framework for Hybrid Experiments Sampling If time, get a start on True Experiments: Single-Factor.
LECTURE 2 EPSY 642 META ANALYSIS FALL CONCEPTS AND OPERATIONS CONCEPTUAL DEFINITIONS: HOW ARE VARIABLES DEFINED? Variables are operationally defined.
Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved.
Systematic reviews to support public policy: An overview Jeff Valentine University of Louisville AfrEA – NONIE – 3ie Cairo.
LITERATURE REVIEW  A GENERAL GUIDE  MAIN SOURCE  HART, C. (1998), DOING A LITERATURE REVIEW: RELEASING THE SOCIAL SCIENCE RESEARCH IMAGINATION.
META-ANALYSIS, RESEARCH SYNTHESES AND SYSTEMATIC REVIEWS © LOUIS COHEN, LAWRENCE MANION & KEITH MORRISON.
From description to analysis
 Descriptive Methods ◦ Observation ◦ Survey Research  Experimental Methods ◦ Independent Groups Designs ◦ Repeated Measures Designs ◦ Complex Designs.
Retain H o Refute hypothesis and model MODELS Explanations or Theories OBSERVATIONS Pattern in Space or Time HYPOTHESIS Predictions based on model NULL.
EBM --- Journal Reading Presenter :呂宥達 Date : 2005/10/27.
1 Lecture 10: Meta-analysis of intervention studies Introduction to meta-analysis Selection of studies Abstraction of information Quality scores Methods.
URBDP 591 I Lecture 4: Research Question Objectives How do we define a research question? What is a testable hypothesis? How do we test an hypothesis?
Systematic Reviews and Meta-analyses. Introduction A systematic review (also called an overview) attempts to summarize the scientific evidence related.
What Have We Learnt About Attendance Demand in Sports During the Last Decades? A Meta-Regression Analysis X IASE ANNUAL CONFERENCE
 Bullying is a significant problem for ~ 20% of youths › Numerous associated issues:  Internal & external psychological disorders  Physical and psychosomatic.
1 Lecture 10: Meta-analysis of intervention studies Introduction to meta-analysis Selection of studies Abstraction of information Quality scores Methods.
Cedric D. Murry APT Instructor of Applied Technology in research and development.
Research methods revision The next couple of lessons will be focused on recapping and practicing exam questions on the following parts of the specification:
This multimedia product and its contents are protected under copyright law. The following are prohibited by law: any public performance or display, including.
NURS 306, Nursing Research Lisa Broughton, MSN, RN, CCRN RESEARCH STATISTICS.
Week Seven.  The systematic and rigorous integration and synthesis of evidence is a cornerstone of EBP  Impossible to develop “best practice” guidelines,
The Research Design Continuum
Rehabilitation Research July 27, 2005
Introduction to Meta-Analysis
Meta-analysis April 11, 2006.
Meta-analysis, systematic reviews and research syntheses
Presentation transcript:

Practical Meta-Analysis -- D. B. Wilson 1 Practical Meta-Analysis David B. Wilson

Practical Meta-Analysis -- D. B. Wilson 2 The Great Debate 1952: Hans J. Eysenck concluded that there were no favorable effects of psychotherapy, starting a raging debate 20 years of evaluation research and hundreds of studies failed to resolve the debate 1978: To proved Eysenck wrong, Gene V. Glass statistically aggregate the findings of 375 psychotherapy outcome studies Glass (and colleague Smith) concluded that psychotherapy did indeed work Glass called his method “meta-analysis”

Practical Meta-Analysis -- D. B. Wilson 3 The Emergence of Meta-Analysis Ideas behind meta-analysis predate Glass’ work by several decades –R. A. Fisher (1944) “When a number of quite independent tests of significance have been made, it sometimes happens that although few or none can be claimed individually as significant, yet the aggregate gives an impression that the probabilities are on the whole lower than would often have been obtained by chance” (p. 99). Source of the idea of cumulating probability values –W. G. Cochran (1953) Discusses a method of averaging means across independent studies Laid-out much of the statistical foundation that modern meta-analysis is built upon (e.g., inverse variance weighting and homogeneity testing)

Practical Meta-Analysis -- D. B. Wilson 4 The Logic of Meta-Analysis Traditional methods of review focus on statistical significance testing Significance testing is not well suited to this task –highly dependent on sample size –null finding does not carry to same “weight” as a significant finding Meta-analysis changes the focus to the direction and magnitude of the effects across studies –Isn’t this what we are interested in anyway? –Direction and magnitude represented by the effect size

Practical Meta-Analysis -- D. B. Wilson 5 When Can You Do Meta-Analysis? Meta-analysis is applicable to collections of research that –are empirical, rather than theoretical –produce quantitative results, rather than qualitative findings –examine the same constructs and relationships –have findings that can be configured in a comparable statistical form (e.g., as effect sizes, correlation coefficients, odds-ratios, etc.) –are “comparable” given the question at hand

Practical Meta-Analysis -- D. B. Wilson 6 Forms of Research Findings Suitable to Meta-Analysis Central Tendency Research –prevalence rates Pre-Post Contrasts –growth rates Group Contrasts –experimentally created groups comparison of outcomes between treatment and comparison groups –naturally occurring groups comparison of spatial abilities between boys and girls Association Between Variables –measurement research validity generalization –individual differences research correlation between personality constructs

Practical Meta-Analysis -- D. B. Wilson 7 Effect Size: The Key to Meta-Analysis The effect size makes meta-analysis possible –it is the “dependent variable” –it standardizes findings across studies such that they can be directly compared Any standardized index can be an “effect size” (e.g., standardized mean difference, correlation coefficient, odds- ratio) as long as it meets the following –is comparable across studies (generally requires standardization) –represents the magnitude and direction of the relationship of interest –is independent of sample size Different meta-analyses may use different effect size indices

Practical Meta-Analysis -- D. B. Wilson 8 The Replication Continuum Pure Replications Conceptual Replications You must be able to argue that the collection of studies you are meta-analyzing examine the same relationship. This may be at a broad level of abstraction, such as the relationship between criminal justice interventions and recidivism or between school- based prevention programs and problem behavior. Alternatively it may be at a narrow level of abstraction and represent pure replications. The closer to pure replications your collection of studies, the easier it is to argue comparability.

Practical Meta-Analysis -- D. B. Wilson 9 Which Studies to Include? It is critical to have an explicit inclusion and exclusion criteria (see handout) –the broader the research domain, the more detailed they tend to become –developed iteratively as you interact with the literature To include or exclude low quality studies –the findings of all studies are potentially in error (methodological quality is a continuum, not a dichotomy) –being too restrictive may restrict ability to generalize –being too inclusive may weaken the confidence that can be placed in the findings –must strike a balance that is appropriate to your research question

Practical Meta-Analysis -- D. B. Wilson 10 Searching Far and Wide The “we only included published studies because they have been peer-reviewed” argument Significant findings are more likely to be published than nonsignificant findings Critical to try to identify and retrieve all studies that meet your eligibility criteria Potential sources for identification of documents –computerized bibliographic databases –authors working in the research domain –conference programs –dissertations –review articles –hand searching relevant journal –government reports, bibliographies, clearinghouses

Practical Meta-Analysis -- D. B. Wilson 11 Strengths of Meta-Analysis Imposes a discipline on the process of summing up research findings Represents findings in a more differentiated and sophisticated manner than conventional reviews Capable of finding relationships across studies that are obscured in other approaches Protects against over-interpreting differences across studies Can handle a large numbers of studies (this would overwhelm traditional approaches to review)

Practical Meta-Analysis -- D. B. Wilson 12 Weaknesses of Meta-Analysis Requires a good deal of effort Mechanical aspects don’t lend themselves to capturing more qualitative distinctions between studies “Apples and oranges”; comparability of studies is often in the “eye of the beholder” Most meta-analyses include “blemished” studies Selection bias posses continual threat –negative and null finding studies that you were unable to find –outcomes for which there were negative or null findings that were not reported Analysis of between study differences is fundamentally correlational