Seminar on selected evaluation methodology issues 评估方法论研讨会 Independent Office of Evaluation of IFAD Beijing, 16 July 2014 国际农业发展基金独立评估办公室 2014 年 7 月 16 日，北京

Objectives Discuss selected methodology issues for evaluation of agriculture development interventions Collect feedback for the development of the new IFAD/IOE evaluation manual 2

Context IOE and Ministry of Finance joint Statement of Intend for Evaluation Capacity Development Development of new evaluation manual, to be issued in 2015 Joint PPA of the Environment Conservation and Poverty-Reduction Programme in Ningxia and Shanxi 3

Key IOE products Corporate-level Evaluations Country Programme Evaluations Project Evaluations Evaluation Synthesis Reports Impact Evaluations 4

Evaluation criteria used Use of six point rating scale to assess performance 5 RelevanceGender EffectivenessInnovation and Scaling-up EfficiencyBorrower performance ImpactLender performance Sustainability

Methodology Issues (1) Data Quality Challenges: No baseline survey; weak M&E system; data mostly on outputs; deficiencies in project log frame; too many indicators Options: Evaluability assessments; reconstruct log frame and underlying theories of change at evaluation; primary data collection (RRAs, surveys, focus groups, case studies); triangulation 6

Methodology Issues (2) Assessing efficiency (A measure of how economically resources/inputs (funds, expertise, time, etc.) are converted into results) Challenges: Lack of data on outcomes; difficulty in quantifying benefits; weak data on costs and inputs; inadequate capacity to measure EIRR; not all projects have ex-ante EIRR Options: use of proxy indicators (delays from loan approval to effectiveness; time-over-run, project management costs, cost per beneficiary, final disbursements; etc.); benchmark unit costs (e.g., for infrastructure; CBAs 7

Methodology Issues (3) Evaluation and hindsight (How to evaluate a project in an evolving context? Is it fair to evaluate a project using today’s standard?) Challenges: New priorities emerge during implementation (e.g., climate change, scaling-up, etc.); evolution of policy and institutional context Options: Learning accountability; retrofitting design during implementation; If project objectives and/or outcome targets were changed during implementation, both the original and revised objectives are taken into account 8

Methodology Issue (4) Impact Evaluation Definition of impact: The changes that have occurred – as perceived at the time of evaluation – in the lives of the rural people (whether positive or negative, direct or indirect, intended or unintended) as a result of IFAD interventions” IOE Evaluation Manual

Methodology Issue (4) – cont. IOE and impact evaluation IOE has introduced impact evaluations as new independent evaluation products in impact evaluation finalized (Sri Lanka); 1 ongoing (India); 1 planned in the IOE 2015 Work Programme Selectivity framework guiding the identification of the (project) impact evaluation 10

Methodology Issue (4) – cont. IOE Objectives for Impact Evaluations Assess the results and impact of the programme in a more quantitative manner; Generate recommendations; and Contribute to innovative mixed evaluation methodology development 11

Methodology Issues (4) – cont. How do impact evaluations provide evidence of the changes caused by an operation? Impact evaluations compare the situation of the beneficiaries who participated in the program (treatment group) against those who did not participate (comparison group), as well as “before and after” the program Mix-methods approach applying quantitative quasi-experimental and qualitative participatory methods:  Pre-Post  Simple Difference  Difference-in-difference  Matching  Multivariate (Multiple) Regression 12

Methodology Issues (4) – cont. Impact evaluations: key challenges Complexity of IFAD’s programmes and scarce availability of baseline data Incubation time for impact measurement Establishment of a counterfactual that will effectively allow the attribution of impact to the program 13

Methodology Issues (4) – cont. Impact evaluations: opportunities To improve accountability  through assessing not only “if”, but also “how” and “why” the program has, or has not, had an impact and its magnitude To improve systematic and dynamic learning through:  enhancing IFAD’s ongoing programs and ultimately better allocate funds across programs  feeding results into relevant CPEs and CLEs undertaken by IOE  strengthening the internal debate on impact evaluations 14