International Initiative for Impact Evaluation TWO RECENT SYSTEMATIC REVIEWS FOR DEVELOPMENT Annette N. Brown, Deputy Director for AIES

International Initiative for Impact Evaluation Outline Review questions Inclusion criteria Theory of change Search pipeline Results Fun methods finding

International Initiative for Impact Evaluation “Interventions in Developing Nations for Improving Primary and Secondary School Enrollment of Children: A Systematic Review” By Anthony Petrosino, Claire Morgan, Trevor A. Fronius, Emily E. Tanner-Smith, Robert F. Boruch, November 2012

International Initiative for Impact Evaluation Review questions What are the effects of interventions implemented in developing countries on measures of students’ enrollment, attendance, graduation, and progression? Within those studies that report the effects of an intervention on [the above measures], what are the ancillary effects on learning outcomes?

International Initiative for Impact Evaluation Inclusion criteria Population/participants: Primary and secondary school aged children in LMICs Interventions: Intended to affect one of the four primary outcomes Comparison: No specific program comparison Outcomes: Enrollment, attendance, graduation, and progression Studies: RCTs and QEDs

International Initiative for Impact Evaluation Typology of education interventions Demand Reducing costs CCTs, scholarships and non-fee subsidies Vouchers Abolishing school fees and capitation grants Providing information Increasing preparedness Early child development Health/ nutrition School feeding Supply Buildings Teachers Materials Management

International Initiative for Impact Evaluation

International Initiative for Impact Evaluation

International Initiative for Impact Evaluation

International Initiative for Impact Evaluation

International Initiative for Impact Evaluation

International Initiative for Impact Evaluation “Farmer field schools for improving farming practices and farmer outcomes in low- and middle-income countries: a systematic review” By Hugh Waddington, Birte Snilstveit, Jorge Hombrados, Martina Vojtkova, Daniel Phillips, and Howard White, December 2012

International Initiative for Impact Evaluation Review questions What is the impact of farmer field schools on their objectives in terms of ‘endpoint’ outcomes such as increased yields, net revenues and farmer empowerment, and intermediate outcomes such as capacity building and adoption of improved practices? Under which circumstances and why: what are the facilitators and barriers to FFS effectiveness and sustainability?

International Initiative for Impact Evaluation Inclusion criteria Population/participants: Farm households in low and middle income countries Intervention: Programs explicitly referred to as ‘farmer field school’ Comparison: No specific program comparison Outcomes: effectiveness across the causal chain –Knowledge → adoption → –Impact on yields, revenues, environment, health, empowerment Studies: –Effects: experimental, quasi-experimental with controlled comparison –Barriers/facilitators: qualitative (Critical Appraisal Skills Programme checklist 2006)

International Initiative for Impact Evaluation T of Change Input 1 Training of trainers Input 2 Field school Adoption (FFS participants) Capacity building (FFS participants) Capacity building (FFS neighbours) Adoption (FFS neighbours) Measured impacts: Yield, input-output ratio, income, empowerment, environmental outcomes, health

International Initiative for Impact Evaluation Input 1 Training of trainers Input 2 Field school Adoption (FFS participants) Capacity building (FFS participants) Capacity building (neighbours) Adoption (neighbours) - Facilitators adequately trained - Farmers and facilitators attend sufficient meetings - FFS synchronised with planting season - Curriculum relevant to problems facing farmers -Farmer attitudes changed (convinced message appropriate) - Relative advantage over old techniques - Field days/follow-up - High degree of social cohesion - Geographical proximity to other farmers (observation) or market (communication) Measured impacts: Yield, input-output ratio, income, empowerment, environmental outcomes. health - New technology appropriate - Market access - Favorable prices - Environmental factors including weather, soil fertility

International Initiative for Impact Evaluation 1,112 abstracts screened 751 excluded 312 full text sought49 no access 183 Extension impact papers: 134 FFS 49 non-FFS 257 excluded 1453 abstracts screened 27,866 titles screened 369 full text obtained126 no access 186 excluded: 128 on relevance 58 on design (no comparison) 134 FFS impact papers 80 individual FFS studies 25 qualitative papers Causal Chain Analysis Effectiveness 20 individual FFS studies 30 IE and sister papers 11 individual FFS studies Qualitative Synthesis BB+ Synthesis

International Initiative for Impact Evaluation Positive impacts on knowledge among participants

International Initiative for Impact Evaluation Increased yields among FFS- beneficiaries not neighbours

International Initiative for Impact Evaluation Reduced environmental risk factors NOTE: Weights are from random effects analysis.. FFS participants Pananurak, 2010 (India) Praneetvatakul & Waibel, 2006 (Thailand) Pananurak, 2010 (Pakistan) Cavatassi et al., 2011 (Ecuador) Subtotal (I-squared = 8.0%, p = 0.353) FFS neighbours Pananurak, 2010 (India) Pananurak, 2010 (Pakistan) Cavatassi et al., 2011 (Ecuador) Praneetvatakul & Waibel, 2006 (Thailand) Subtotal (I-squared = 0.0%, p = 0.878) ID Study 0.52 (0.32, 0.85) 0.54 (0.39, 0.76) 0.55 (0.41, 0.75) 0.82 (0.55, 1.23) 0.59 (0.49, 0.71) 0.58 (0.24, 1.41) 0.64 (0.37, 1.10) 0.69 (0.43, 1.11) 1.04 (0.32, 3.40) 0.68 (0.49, 0.93) ES (95% CI) 0.52 (0.32, 0.85) 0.54 (0.39, 0.76) 0.55 (0.41, 0.75) 0.82 (0.55, 1.23) 0.59 (0.49, 0.71) 0.58 (0.24, 1.41) 0.64 (0.37, 1.10) 0.69 (0.43, 1.11) 1.04 (0.32, 3.40) 0.68 (0.49, 0.93) ES (95% CI) Favours intervention

International Initiative for Impact Evaluation Summary of quantitative findings FFS increase knowledge and improve adoption of the FFS practices On average increasing yields and/or incomes Suggestions of farmers feeling empowered Limited, if any, spillovers Neighbours do not adopt the practices consistently

International Initiative for Impact Evaluation Sensitivity analysis: Yields by risk of bias status High risk of bias studies over- estimate impacts

International Initiative for Impact Evaluation THANK YOU!