1. Systematic reviews to support public policy: An overview Jeff Valentine University of Louisville jeff.valentine@louisville.edu AfrEA – NONIE – 3ie Cairo Conference April 2009

2. Overview of the Session What is a systematic review? How can a systematic review help support decision making? What limitations need to be considered? Recommendations for using systematic reviews to support decision making in developing contexts

3. What is a Systematic Review? Systematic review –Organized, transparent (replicable) approach to all phases of a review process –Systematic effort to identify, analyze, and synthesize results of previous studies –Follows a pre-determined plan (ideally) –Synonyms: research synthesis, scientific literature review Meta-analysis –Quantitative synthesis of the results of previous studies

4. Overarching Goals in Systematic Reviewing Limit bias in literature reviews by using … –a systematic approach to data collection, analysis, and evaluation –both published and unpublished research (minimizing publication bias) Enhance methodological rigor by… –holding reviewers to high scientific standards Employ better and more appropriate statistical tools by… –avoiding decision rules with poor inferential characteristics –using statistics that are appropriate for the analysis of collections of studies

5. What is a Systematic Review? (cont.) Compared to traditional procedures, systematic reviews... use a more systematic approach to data collection have more transparent and explicit procedures, –thus have less potential for hidden assumptions or biases to drive results, and focus on the magnitude of effects rather than statistical significance result in conclusions that are less bound by context than conclusions that arise from individual studies

6. Non-Systematic Reviews Non-systematic reviews tend to… –rely on a convenience sample of studies, –use decision rules that are not transparent, e.g., rarely state a priori what constitutes “good” evidence, –rely on statistical significance to gauge what a study “says”, and –often use a “vote count” of results to draw conclusions about the literature as a whole.

7. Comparison of Systematic and Non-Systematic Reviews Systematic ReviewsNon-Systematic Reviews Exhaustive search for relevant studiesConvenience sample of studies, often published only Usually not well specified Extensive and structured “survey” of the studies for information about the quality of the studies the intervention’s effects how and for whom the intervention worked ? – Usually not well specified Often appear to be subjective assessments of effectiveness Appropriate statistical methods for analyzing the results of studies Usually not well specified Inappropriate statistical methods for analyzing the results of studies

8. The Campbell Collaboration (C2) C2 quality control methods include –A dedicated editor who shepherds the research team through the systematic review process –The creation of a review “protocol” that operationally defines the review methods, before data collection –Peer review by at least one methodological and one substantive expert of both protocol and the final review

9. How Can Systematic Reviews Support Decision Making?: The Case for School Feeding Programs The research question “…to determine the effectiveness of school feeding programs in improving physical and psychosocial health for disadvantaged school pupils.” The research method –Systematic review

10. Specific Methods for School Feeding Programs Review Literature search –26 electronic databases Some of which specialize in unpublished literature –Hand searches of 5 academic journals –Bibliography searches –Personally contacted 5 individuals who might know of research that is hard to obtain Exact literature search strategy reported in detail in the report (Cochrane library) 400 potential articles identified

11. Specific Methods for School Feeding Programs Review Of the 400 potential articles –30 appeared to meet criteria for review –18 actually did meet criteria for review All decisions were made by two people who worked independently 18 articles were coded –Again, two researchers working independently extracted information from studies Study quality was appraised for all included studies –Criteria were established before data collection began

12. Specific Methods for School Feeding Programs Review Data Analysis –Random effects meta-analysis Outcomes –Weight/height gain –Psychosocial –Behavioral Confounders –Some evidence for substitution effects »Increase in daily calorie intake far less than that given by the meal –Some evidence of compliance problems

13. Specific Methods for School Feeding Programs Review Recommendations –Discussed implications for policy and future research “a school nutrition intervention is not just about confirming palatability, but about identifying specific nutritional issues / needs in the program's target population” “all primary researchers [should] provide breakdowns of effects by socio-economic status of parents as well as by sex and baseline nutritional status” “attention needs to be paid to ensuring that the intervention is delivered as intended” “Substitution should also be monitored”

14. What Are Some Limitations of Systematic Reviews? A systematic review presumes that a body of relevant literature exists to support a review –However, sometimes “empty reviews” can be informative The “quality” of studies in a systematic review has a very large influence on the validity of that review It is often unclear how to generalize results from a systematic review to contexts that have not been studied –In the context of developing countries, this is an important point –Need to consider plausible theoretical models for how the intervention works

15. Recommendations for Using Systematic Reviews to Support Decision Making in Developing Contexts Consider the relevance of existing (high-quality) systematic reviews for the policy question and context Support additional high-quality and contextually- relevant research –The conclusions from any individual study are highly bound by very specific factors associated with the study and relatively more likely to be influenced by sampling error so be careful about using individual studies to support decision making Support high-quality and relevant systematic reviews on important policy questions