Reviewing systematic reviews: meta- analysis of What Works Clearinghouse computer-assisted interventions. November 2011 American Evaluation Association.

Slides:



Advertisements
Similar presentations
National Reading Panel. Formation Congress requested its formation in Asked to assess the status of research-based knowledge about reading and the.
Advertisements

Synthesizing the evidence on the relationship between education, health and social capital Dan Sherman, PhD American Institutes for Research 25 February,
Magia G. Krause Ph.D. Candidate School of Information University of Michigan Society of American Archivists Annual Meeting August 14, 2009 Undergraduates.
Funded through the ESRC’s Researcher Development Initiative Department of Education, University of Oxford Session 2.3 – Publication bias.
The Campbell Collaborationwww.campbellcollaboration.org Moderator analyses: Categorical models and Meta-regression Terri Pigott, C2 Methods Editor & co-Chair.
The Behavioural/Developmental Continuum of Interventions for Autism Spectrum Disorders: A Systematic Review Many Faces of Childhood Well Being: The Early.
Introduction to Meta-Analysis Joseph Stevens, Ph.D., University of Oregon (541) , © Stevens 2006.
EVAL 6970: Meta-Analysis Fixed-Effect and Random- Effects Models Dr. Chris L. S. Coryn Spring 2011.
EVAL 6970: Meta-Analysis Fixed-Effect and Random- Effects Models Dr. Chris L. S. Coryn Spring 2011.
How Do We Identify Causes? The criteria of causation © Pine Forge Press, an imprint of Sage Publications, 2006.
Meta-analysis & psychotherapy outcome research
Coaches as Critical Consumers of Research Statewide Coaches Meeting Oregon Reading First Center Rachell Katz Jeanie Mercier-Smith April 24, 2008.
Overview of Meta-Analytic Data Analysis. Transformations Some effect size types are not analyzed in their “raw” form. Standardized Mean Difference Effect.
Robert delMas (Univ. of Minnesota, USA) Ann Ooms (Kingston College, UK) Joan Garfield (Univ. of Minnesota, USA) Beth Chance (Cal Poly State Univ., USA)
READING RECOVERY KIMBERLEE HICKS KAREN JEWETTE CHARLOTTE WRIGHT RANDY BACHMEIER GROUP 1 KIMBERLEE HICKS KAREN JEWETTE CHARLOTTE WRIGHT RANDY BACHMEIER.
Meta-findings from the Best Evidence Encyclopaedia Robert E Slavin University of York and Johns Hopkins University.
Are the results valid? Was the validity of the included studies appraised?
Overview of Meta-Analytic Data Analysis
9.0 A taste of the Importance of Effect Size The Basics of Effect Size Extraction and Statistical Applications for Meta- Analysis Robert M. Bernard Philip.
Robert McCabe, Chief Education Officer. Company Confidential A National Consensus.
Advanced Statistics for Researchers Meta-analysis and Systematic Review Avoiding bias in literature review and calculating effect sizes Dr. Chris Rakes.
Reviewing systematic reviews: meta- analysis of What Works Clearinghouse computer-assisted reading interventions. October 2012 Improving Education through.
Program Evaluation. Program evaluation Methodological techniques of the social sciences social policy public welfare administration.
T tests comparing two means t tests comparing two means.
The Effect of Computers on Student Writing: A Meta-Analysis of Studies from 1992 to 2002 Amie Goldberg, Michael Russell, & Abigail Cook Technology and.
Systematic Reviews.
Evidence Based Medicine Meta-analysis and systematic reviews Ross Lawrenson.
Introduction to Systematic Reviews Afshin Ostovar Bushehr University of Medical Sciences Bushehr, /9/20151.
The effects of within class grouping on reading achievement: A meta-analytic synthesis Kelly Puzio & Glenn Colby Vanderbilt University.
Statistical Applications for Meta-Analysis Robert M. Bernard Centre for the Study of Learning and Performance and CanKnow Concordia University December.
Funded through the ESRC’s Researcher Development Initiative Prof. Herb MarshMs. Alison O’MaraDr. Lars-Erik Malmberg Department of Education, University.
What Works Clearinghouse Susan Sanchez Institute of Education Sciences.
IMPRINT Developer’s Workshop December 6-7, 2005 Meta-analytic Reviews of the Effects of Temperature and Vibration on Performance J.L. Szalma & G. Conway.
Systematic reviews to support public policy: An overview Jeff Valentine University of Louisville AfrEA – NONIE – 3ie Cairo.
Meta-analysis and “statistical aggregation” Dave Thompson Dept. of Biostatistics and Epidemiology College of Public Health, OUHSC Learning to Practice.
Clinical Writing for Interventional Cardiologists.
The Campbell Collaborationwww.campbellcollaboration.org C2 Training: May 9 – 10, 2011 Introduction to meta-analysis.
Effect Size Calculation for Meta-Analysis Robert M. Bernard Centre for the Study of Learning and Performance Concordia University February 24, 2010 February.
Using Regression Discontinuity Analysis to Measure the Impacts of Reading First Howard S. Bloom
1 Introduction Description of the video series Meta-analysis in R with Metafor.
Advanced Meta-Analyses Heterogeneity Analyses Fixed & Random Efffects models Single-variable Fixed Effects Model – “Q” Wilson Macro for Fixed Efffects.
Developing a Review Protocol. 1. Title Registration 2. Protocol 3. Complete Review Components of the C2 Review Process.
Analysis Overheads1 Analyzing Heterogeneous Distributions: Multiple Regression Analysis Analog to the ANOVA is restricted to a single categorical between.
META-ANALYSIS, RESEARCH SYNTHESES AND SYSTEMATIC REVIEWS © LOUIS COHEN, LAWRENCE MANION & KEITH MORRISON.
Robert E. Slavin Johns Hopkins University and University of York.
1 Lecture 10: Meta-analysis of intervention studies Introduction to meta-analysis Selection of studies Abstraction of information Quality scores Methods.
Systematic Synthesis of the Literature: Introduction to Meta-analysis Linda N. Meurer, MD, MPH Department of Family and Community Medicine.
Interventions Identifying and Implementing. What is the purpose of providing interventions? To verify that the students difficulties are not due to a.
Chapter Eight: Quantitative Methods
Replication in Prevention Science Valentine, et al.
Characteristics of Studies that might Meet the What Works Clearinghouse Standards: Tips on What to Look For 1.
Funded through the ESRC’s Researcher Development Initiative Department of Education, University of Oxford Session 2.1 – Revision of Day 1.
Systematic Reviews and Meta-analyses. Introduction A systematic review (also called an overview) attempts to summarize the scientific evidence related.
1 Lecture 10: Meta-analysis of intervention studies Introduction to meta-analysis Selection of studies Abstraction of information Quality scores Methods.
Copyright © 2012 Wolters Kluwer Health | Lippincott Williams & Wilkins Chapter 27 Systematic Reviews of Research Evidence: Meta-Analysis, Metasynthesis,
Effects of Child Skills Training in Preventing Antisocial Behavior By: Friedrich Losel & Andreas Beelmann Angelique Marshall Radford University.
Wim Van den Noortgate Katholieke Universiteit Leuven, Belgium Belgian Campbell Group Workshop systematic reviews.
An Application of Multilevel Modelling to Meta-Analysis, and Comparison with Traditional Approaches Alison O’Mara & Herb Marsh Department of Education,
Educational Research Inferential Statistics Chapter th Chapter 12- 8th Gay and Airasian.
June 25, Regional Educational Laboratory - Southwest Review of Evidence on the Effects of Teacher Professional Development on Student Achievement:
Systematic Reviews of Evidence Introduction & Applications AEA 2014 Claire Morgan Senior Research Associate, WestEd.
Do Adoptees Have Lower Self Esteem?
Exercise 2-Effect Size Coding
RDI Meta-analysis workshop - Marsh, O'Mara, & Malmberg
Supplementary Table 1. PRISMA checklist
The Big Picture Teaching Reading Sourcebook 2nd edition
Publication Bias in Systematic Reviews
MBA 231 RESEARCH METHODOLOGY Course Introduction Dr. Mohammed Al-Abed.
META-ANALYSIS PROCEDURES
Presentation transcript:

Reviewing systematic reviews: meta- analysis of What Works Clearinghouse computer-assisted interventions. November 2011 American Evaluation Association Annual Meeting, Anaheim Andrei Streke ● Tsze Chan

Session Title: Advanced Analytic Techniques in Educational Evaluation Multipaper Session 244 Thursday, Nov 3 2

 What Works Clearinghouse systematic reviews  Meta-analysis of computer-assisted programs across WWC topic areas, reading outcomes  Meta-analysis of computer-assisted programs within Beginning Reading topic area Presentation Overview 3

 A clearly stated set of objectives with pre- defined eligibility criteria for studies  An explicit reproducible methodology  A systematic search that attempts to identify all studies that would meet the eligibility criteria  An assessment of the validity of the findings of the included studies  A systematic presentation, and synthesis, of the characteristics and findings of the studies WWC Systematic Review 4

Normative documents ( ) : (  WWC Procedures and Standards Handbook  WWC topic area review protocol WWC products:  Intervention reports  Practice guides  Quick reviews WWC Systematic Review 5

 Manuscript is written in English and published 1983 or later  Both published and unpublished reports are included  Eligible designs: RCT; QED with statistical controls for pretest and/or a comparison group matched on pretest; regression discontinuity; SCD  At least one relevant quantitative outcome measure  Manuscript focuses on beginning reading  Focus is on students ages 5-8 and/or in grades K-3.  Primary language of instruction is English Selection Criteria for Beginning Reading Topic Area 6

 Designs that confound study condition and study site –Programs that were tested with only one treatment and one control classroom or school  Non-comparable groups –Study designs that compared struggling readers to average or good readers to test a program’s effectiveness Examples of problematic study designs that do not meet WWC criteria 7

 Program description  Intervention rating  Technical Appendices – Study characteristics – Outcomes characteristics – Study findings: effect sizes and improvement indices ccelreader_app_ pdf ccelreader_app_ pdf WWC Intervention reports 8

9

 Effect Sizes  Aggregation Method  Testing for Homogeneity  Fixed and Random Effects Models  Moderator Analysis -- ANOVA type -- Regression type Meta-Analysis procedures 10

Effect Size 11 (1) Effect size (Hedges & Olkin, 1985):

Flowchart for calculation of effect size (Tobler et al., 2000) 12

Aggregation of Effect Sizes 13 (1) Effect size (Hedges): (2) Effect size variance: (3) Weighted average effect size: (4) Weighted average effect size variance: Weight (w)= (Variance) -1

Meta-analysis of computer-assisted programs across WWC topic areas, reading outcomes  Does the evidence in WWC reports indicate that computer-assisted programs increase student reading achievement? 14

Computer-assisted interventions 15

 Earobics ® is interactive software that provides students in pre-K through third grade with individual, systematic instruction in early literacy skills as students interact with animated characters. The program builds children’s skills in phonemic awareness, auditory processing, and phonics, as well as the cognitive and language skills required for comprehension. Examples of computer-assisted programs 16

 Lexia Reading is a computerized reading program that provides phonics instruction and gives students independent practice in basic reading skills. Lexia Reading is designed to supplement regular classroom instruction. It is designed to support skill development in the five areas of reading instruction identified by the National Reading Panel. Examples of computer-assisted programs 17

Number of students and effect sizes by topic area 18

Computer-assisted programs, fixed effects 19

 Homogeneity analysis tests whether the assumption that all of the effect sizes are estimating the same population mean is a reasonable assumption.  If homogeneity is rejected, the distribution of effect sizes is assumed to be heterogeneous. Homogeneity Testing 20

Tests for Homogeneity of Weighted Effect Sizes by Topic Area 21

 Fixed effects model assume: (1) there is one true population effect that all studies are estimating (2) all of the variability between effect sizes is due to sampling error  Random effects model assume: (1) there are multiple (i.e., a distribution) of population effects that the studies are estimating (2) variability between effect sizes is due to sampling error + variability in the population of effects (Lipsey and Wilson, 2001) Random versus Fixed Effects Models 22

 Fixed effects model weights each study by the inverse of the sampling variance.  Random effects model weights each study by the inverse of the sampling variance plus a constant that represents the variability across the population effects (Lipsey & Wilson, 2001). Random Effects Model weights 23 This is the random effects variance component.

Computer-assisted programs, random effects 24

Computer-assisted programs, random and fixed effects 25

Computer-assisted reading interventions, topic area effects and 95% CIs 26

Meta-analysis of computer-assisted programs within Beginning Reading topic area  Are computer-assisted reading programs more effective than non-computer reading programs in improving student reading achievement? 27

Number of students and effect sizes by type of program: BR topic area 28

Beginning Reading Topic Area 29

 Reading Recovery® is a short-term tutoring intervention intended to serve the lowest- achieving first-grade students. The goals of Reading Recovery® are to promote literacy skills, reduce the number of first-grade students who are struggling to read, and prevent long-term reading difficulties. Reading Recovery® supplements classroom teaching with one-to-one tutoring sessions, generally conducted as pull-out sessions during the school day. Other reading programs 30

Beginning Reading programs, fixed effects 31

Tests for Homogeneity of Weighted Effect Sizes by Type of Program, BR 32

Beginning Reading programs, random effects 33

Beginning Reading programs, random and fixed effects 34

Beginning Reading Interventions, Fixed Effects, 95% Confidence Intervals 35

Beginning Reading Interventions, Random Effects, 95% Confidence Intervals 36

Modeling between study variability:  Categorical models (analogous to a one- way ANOVA)  Regression models (continuous variables and/or multiple variables with weighted multiple regression) Moderator Analysis, random effects 37

 Population  Design  Sample size  Control group  Reading domain Categorical analysis: moderators of program effectiveness 38

Weighted mean Effect Sizes for moderators: 80 studies, Beginning Reading, random effects 39

Weighted mean Effect Sizes for moderators: 80 studies, Beginning Reading, random effects 40

Dummy Variables for Regressions 41

Regression Statistics for BR Programs, Random effects 42

Regression Statistics for BR Programs, Random effects 43

Regression Statistics for BR Programs, Random Effects 44

Meta-Analytic Multiple Regression Results From the Wilson/Lipsey SPSS Macro 45

 The present work appears to lend some support to the proposition that computer-assisted interventions in reading are effective. For example, the average effect for beginning reading computer-based programs is positive and substantively important (that is >0.25).  For the Beginning Reading topic area, the effect appears smaller than the effect achieved by non- computer reading programs. Conclusions 46

 Borenstein, M., Hedges, L.V., Higgins, J.P., and Rothstein, H.R. (2009). Introduction to Meta-Analysis. John Wiley and Sons.  Hedges, L. V. and Olkin I. (1985). Statistical Methods for Meta-Analysis. New York: Academic Press.  Lipsey, M.W., & Wilson, D.B. (2001). Practical Meta-Analysis. Thousand Oaks, CA: Sage.  Tobler, N.S., Roona, M.R., Ochshorn, P., Marshall, D.G., Streke, A.V., & Stackpole, K.M. (2000). School-based adolescent drug prevention programs: 1998 meta-analysis. Journal of Primary Prevention, 20(4), References 47

Mathematica ® is a registered trademark of Mathematica Policy Research.  Please contact: –Andrei Streke –Tsze Chan For More Information 48