Impact evaluations at IFAD-IOE Fabrizio Felloni, Deputy Director Independent Office of Evaluation of IFAD (IOE) 19 May 2017 – Presentation to the Independent Evaluation Division, UNIDO

Background IFAD Management launched in 2013 an impact assessment initiative (over 30 IAs 2013-15), with quantitative techniques to attribute impact to project activities IOE-IFAD requested by Executive Board to review the above initiative. Since 2013 IOE conducted 1 impact evaluation per year to: (i) upgrade its technical skills; (ii) better engage in IFAD and external fora; and (iii) provide hands-on assessment on impact assessment initiative

Impact Evaluations conducted so far Sri Lanka (2013). Quantitative survey (~2,500hh) + focus group discussion and follow-up technical mission India (2014). Quantitative survey (~8,800hh) + focus group discussion Mozambique. Quantitative survey (~1,500hh) + focus group discussion Ongoing: Georgia: Quantitative survey (~4,000hh) + focus group discussion

Characteristics Include both: Quantitative and Cover all evaluation criteria Relevance Effectiveness Efficiency Impact Sustainability Gender equality Innovation Scaling Up Natural Resource Management Climate Change Adaptation Performance of partners (IFAD, Government) Include both: Quantitative and Qualitative data collection and analysis Primary data collection

General approach Quantitative part compares between household with and without project support (treatment vs non- treatment). Focuses on whether and what changed Qualitative part focus on understanding why (mechanisms) Technical validation mission: covers other evaluation criteria and further validates findings

Main constraints Non-existent or poor baseline surveys or missing databases Can not use “difference in difference” methods Non-random selection of project beneficiary Used propensity score matching (non-parametric) and Heckman selection procedure (parametric) to correct for sample selection bias They do not strictly require baseline data Included some recall questions in the questionnaire

Ideal situation (so far not found) Compare a sample of units of observations (persons, households) with and without project. And observe the differences before and after Income pc with project without project Project’s contribution Total change 2000 Before 2006 After Time

Other methodological issues Reconstruct a theory of change  highlight causal chain and key assumptions Level of analysis. Useful to foresee two questionnaires for quantitative part: (i) household level; (ii) community characteristics level Helps understand village fixed effects on final outcomes / impacts Beware of possible: (i) spill-over effects (spreading to non- treated groups); (ii) contamination effects (an external programme affecting project results) Model specification is time consuming and in your final statistical output, you may have signs and levels of significance that you can not explain

Practical organization Data collection and analysis conducted by consulting companies (national and international) But IOE retained full leadership of evaluation (design of methodology, questionnaires, oversight of testing of field instruments, oversight of analysis, report drafting) Better not to have an external company dictate main methodological choices IOE conducted two-three missions: (i) reconnaissance; (ii) oversight of instrument testing in the field; (iii) follow-up technical mission Budget US$ 200,000 all inclusive

Practical organization Important to retain former project manager / senior staff as key informants (check sampling strategy) Important for consulting companies to invest on training of enumerators and quality of collection (vs. incentives to fill in many questionnaires) ICT technology to reduce coding time and reduce risk of “fakes” (timing, GPS)

Documents IFAD Manual: https://www.ifad.org/documents/10180/bfec198c-62fd-46ff-abae-285d0e0709d6 Sri Lanka: https://www.ifad.org/documents/10180/2b8f1c99-16be-4b30-969b-d93f03ccca41 India: https://www.ifad.org/evaluation/reports/impact_evaluation/tags/india/1063/7854528 Mozambique: https://www.ifad.org/evaluation/reports/impact_evaluation/tags/mozambique/1517/36805916