Evidence-Based Decision Making: The Contribution of Systematic Reviews in Synthesizing Evidence

General introduction to systematic reviews 1.What is a systematic review? 2.Why do/fund one? 3.For which policy questions is SR useful? 4.What are the main international bodies?

Systematic reviews as defined by the experts “A systematic, up-to-date summary of reliable evidence of the benefits and risks of an intervention” (The Cochrane Collaboration) Reviews “sum up the best available research on a specific question” (The Campbell Collaboration) Defines a process, not a product. The process is a systematic and transparent approach to all phases of a literature review (Jeff Valentine).

Systematic reviews are research Systematic reviews are a form of research – Secondary observations – In which studies are the unit of analysis Follow basic steps in the research process Aim to minimize bias and error

Example: microcredit Evidence used to justify development policy is often anecdotal: “Small loans enable people in poverty to earn an income and provide for their families… Each successful business feeds a family, employs more people and eventually helps empower a whole community” (Opportunity International) Where evidence is used it is often based on single studies: “Impact studies done on the Grameen Bank by independent researchers find that 5% of borrowers come out of poverty every year…the status of women has been enhanced” (Yunus, 2005) Where a range of studies are used, not clear how representative: “Women participants in microcredit programs often experience important self-empowerment… there is a strong indication from borrowers that microcredit improves their lives” (Microfinance Gateway)

So what do reviews synthesizing all the existing high quality evidence say? Duvendack et al. (2011): ‘All impact evaluations of microfinance suffer from weak methodologies and inadequate data [which] can lead to misconceptions about the actual effects of a microfinance programme’ ‘It remains unclear under what circumstances, and for whom, microfinance has been and could be of real, rather than imagined, benefit to poor people’ Vaessen et al. (2012): ‘From those studies deemed comparable and of minimum acceptable quality, we can conclude that overall the effect of microcredit on women’s control over household spending is weak’.

Sheer amount and flow of information/ research Variable quality of research outputs Need to ‘separate the wheat from the chaff’ Problems of publication bias Limitations of single studies Why do we need systematic reviews?

1.Scoping: defining answerable question, methods set out in study protocol 2.Rigorous search to identify published and unpublished sources, in any language 3.Application of study inclusion criteria (PICOS) 4.Critical appraisal of study quality, to assess how reliable is the evidence 5.Data extraction and organisation 6.Synthesis of evidence (outcomes along causal chain) 7.Interpreting results (policy and practice, research recommendations) 8.Improving and updating reviews as new evidence emerges What makes a systematic review ‘systematic’?

Types of (answerable) questions Policy areaExample question Effectiveness: what ‘difference’ does an intervention make? What is the comparative effectiveness of multiple interventions in attaining an outcome? What are the impacts of daycare on child health, nutrition and cognitive development? What is the effectiveness of water, sanitation and hygiene interventions in terms of reducing child diarrhoea in LMICs? Drivers of change: determinants of compliance or adherence What are the barriers and facilitators of patient adherence to tuberculosis treatment in Africa? What are the barriers to discontinuation of female genital mutilation/ cutting in Africa? Views: What are stakeholders’ experiences or preferences? What is the willingness to pay for clean water in developing countries? Cost-effectiveness or cost-benefitWhat is the effectiveness and cost-effectiveness of conditional and unconditional cash transfers in improving education outcomes? Risk factors associated with a condition or problem?What are the predictive factors for youth involvement in gang violence?

Resources and international bodies Cochrane Collaboration; 1993; – Producing high quality information about the effectiveness of health care (> 5000 published online – Cochrane library) Campbell Collaboration; 2000; – Producing systematic reviews of the effects of social interventions (>200 published online – Campbell library) International Development Coordinating Group (IDCG); – Producing systematic reviews of high policy-relevance focusing on social and economic development interventions in LMICs International Initiative for Impact Evaluation, 3ie; EPPI Centre - An Institute of Education centre focusing on systematic reviews in education, health and social policy EPPI Centre Collaboration for Environmental Evidence producing systematic reviews for environmental management Collaboration for Environmental Evidence