1. Systematic Reviews of Evidence Introduction & Applications AEA 2014 Claire Morgan Senior Research Associate, WestEd

2. Systematic Review The goal of a systematic review is to use transparent, explicit methods in conducting a comprehensive review of research on the effects of an intervention. What works to increase school enrollment in developing nations? What are the impacts of school fee abolition policies? What are the impacts of education voucher programs in developing countries? Overview

3. Systematic Review A review of the evidence on a clear formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant primary research and to extract and analyze data from the studies included in the review “ Overview

4. Systematic Review A systematic review must have: Clear inclusion/ exclusion criteria An explicit search strategy Systematic coding and analysis of included studies Meta-analysis (where possible) Overview

5. Systematic Review Peer review is a key part of the process. Published Protocol: 1. A relevant research question developed in consultation with users. 2. A search strategy to find all the available studies, including journals, grey literature and unpublished studies. 3. A set of inclusion and exclusion criteria to select the studies for review (for example only studies of adolescents in South Asia). Overview

6. Systematic Review Published Protocol: 4. A quality appraisal strategy that is relevant to the review question and the types of studies under review. 5. Methods for synthesizing the studies, according to the type of data available. Overview

7. Steps for Completing a Systematic Review 1. Define the question 2. Assemble a team 3. Create a review protocol Conceptual discussion of the problem Review question Search strategy Study inclusion and exclusion criteria Study quality assessment Data extraction procedure Data synthesis procedure

8. Steps for Completing a Systematic Review 4.Comprehensive search Electronic searches of bibliographic databases “Hand searches” of relevant journals Citation chasing Contacting researchers Manage search results 5. Screen studies

9. Steps for Completing a Systematic Review 6.Quality Appraisal How the groups were equated and whether any problems with equating were reported. Whether the researchers report a loss of participants from the initial assigned sample at the end of the study, how much attrition is reported, and whether the attrition differentially affects one group or the other. Whether the program experienced significant implementation and fidelity problems

10. Steps for Completing a Systematic Review 7. Extract information from each study Coding document: Researcher and study characteristics Study methods and methodological quality Intervention/control conditions data Participants in the study Outcome data

11. Steps for Completing a Systematic Review 7.Analyze and present results Researcher and study characteristics Tabulate results from individual studies Examine forest plot Explore sources of heterogeneity Thematic synthesis for non-empirical studies

12. Steps for Completing a Systematic Review 8. Interpret results Consider limitations, including publication and related biases Consider strength of evidence Consider applicability Consider economic implications Consider implications for future research

13. Recent WestEd International Education Systematic Reviews Interventions in Developing Nations for Improving Primary and Secondary School Enrollment of Children: A Systematic Review http://www.3ieimpact.org/media/filer_public/2013/10/28/ interventions_in_developing_nations_for_improving_p rimary_and_secondary_school_enrolment_of_children _a_systematic_review.pdf A Systematic Review of the Evidence of the Impact of School Voucher Programmes in Developing Countries http://r4d.dfid.gov.uk/pdf/outputs/systematicreviews/Vo ucherProgrammes2013Morgan.pdf A Systematic Review of the Evidence of the Impact of Eliminating School User Fees in Low-income Developing Countries http://r4d.dfid.gov.uk/PDF/Outputs/SystematicReviews/ School-fees-2012-Morgan-report.pdf

14. Challenges Ensuring thoroughness of search Classifying poorly-described studies Subjectivity in screening For certain areas, focus on rigorous experimental and quasi-experimental studies will result in a very low yield There may be disagreement on how interventions are categorized Need for rigorous evaluations and cost-benefit analyses Highly resource-intensive Generating useful policy recommendations (context/process)

15. Resources The Campbell Collaboration http://www.campbellcollaboration.org The Cochrane Collaboration http://www.cochrane.org/ UK Department for International Development (DFID) https://www.gov.uk/government/organisati ons/department-for-international- development The Collaboration for Environmental Evidence http://www.environmentalevidence.org International Initiative for Impact Evaluation (3ie) http://www.3ieimpact.org ePPI Centre http://eppi.ioe.ac.uk PRISMA http://www.prisma-statement.org/

16. Contact Claire Morgan cmorgan@wested.org