Africa RISING M&E Expert Meeting Addis Ababa, 5-7 September 2012

Outline Timeline Objectives of M&E Targeting vs Research Monitoring and Evaluation Theory of Change Evaluation designs Open issues

3-9 Months 9-12 Months Timeline ✔ 1-3 Months Site selection/ Characterization Component Inventory Activity -> Indicator List Survey Design Baseline Survey Component DB ✔ Evaluation Design

Objectives of M&E Keep track of the AR’s outputs (M) Assess the impact of AR (E) Create knowledge of what works (M&E): -assess effectiveness -rank policy/project alternatives -appraise scale-up and external validity -identify problems in the project -inform project management

What is AR M&E after? M could be an easy(ier) task but: -we need constant and timely information -we need close collaboration among CG, implementers, and IFPRI ->Solution: Outcome mapping? Cost-benefit analysis? Surveys? E must be well thought in advance…

Targeting vs Research 1. Do we need to select the poorest/hungriest farmers to benefit from AR? 2. Or, being AR a research project, is it possible to use a stratified sample (based on different criteria)? 2a. common E approach? 2b. project-specific E approach? It depends on the question(s)!

Evaluation Determine the causal effect of AR on outcomes (not only on outputs): –Farmers’ wellbeing? –Land productivity? –Input supply, labor productivity, environment, women’s conditions, health and nutrition,…? –…all of the above plus-> for whom? For which development domain? For which type of households? For which livelihood? What would be the impact with a different technology package?

Monitoring Evaluation ‘Traditional’ M&E: monitoring to track implementation efficiency (input - output) Impact Evaluation: estimate causal effectiveness on outcomes (output - outcome) INPUTS OUTCOMES/IMPACTS OUTPUTS MONITOR EFFICIENCY EVALUATE EFFECTIVENESS $$$ BEHAVIOUR Note: Diagram from WorldBank training material produced by Arianna Legovini, Lead Economist - AIEI

Monitoring Evaluation Indicators Number of individuals who have received USG supported long-term agricultural sector productivity or food security training Number of individuals who have received USG supported short-term agricultural sector productivity or food security training Number of food security private enterprises (for profit), producers organizations, […] receiving USG assistance Number of members of producer organizations and community based organizations receiving USG assistance Number of new technologies or management practices in one of the following phases of development: …in Phase I: under research as a result of USG assistance …in Phase II: under field testing as a result of USG assistance …in Phase III: made available for transfer as a result of USG assistance Number of public-private partnerships formed as a result of FTF assistance *Number of rural households benefiting directly from USG interventions *Number of vulnerable households benefiting directly from USG interventions Number of farmers and others who have applied new technologies or management practices as a result of USG assistance Number of private enterprises (for profit), producers organizations, […] that applied new technologies or management practices as a result of USG assistance * *Number of hectares under improved technologies or management practices as a result of USG assistance *Number of stakeholders implementing risk-reducing practices/actions to improve resilience to climate change as a result of USG assistance *Value of incremental sales (collected at farm-level) attributed to FTF implementation *Farmer satisfaction with quantity, quality and timeliness of extension and input supply services *Value of new private sector investment in the agriculture sector or food chain leveraged by FTF implementation * Indicators proposed for discussion by IFPRI

Theory of Change Impact evaluation must be based on a set of hypotheses on the change that can be achieved as a consequence of AR How would you think that AR can affect the life of (beneficiary) farmers?

Livelihood Strategies /Coping Strategies /Vulnerability to Shocks AR implementation Productivity increase Income and Expenditure Saving/ Investment General Household Expenditure School Enrolment and Attendance Expenditure on Health and Education for children School Progression Health Status Food Intake Dietary Diversity Food Security Psychologica l well being * Other Expenditures for children: Food, Clothing, Recreation Targeting ACTIVITIES OUTPUTS FIRST ORDER OUTCOMES SECOND ORDER OUTCOMES Asset Building THIRD ORDER OUTCOMES Labour Participation Child Labour Remittances Access to Credit Time Allocation of Children Time Allocation/ Caring arrangements/ Migration of Adults and Caregivers * Intra-household decision making * Utilization of Health Services Time and risk preferences

Impact Evaluation How would you go about measuring the causal impact of AR on … -productivity?

Y t Impact? Impact Evaluation - Method AfterBefore Bens

Y t Impact Impact Evaluation - Method AfterBefore Bens Non Bens RCTs

Y t Pre-existing Difference Impact Impact Evaluation - Method AfterBefore Bens Non Bens Diff in Diff

Y t Propensity Score Matching Impact Impact Evaluation - Method AfterBefore Bens Non Bens Diff in Diff with

Impact Evaluation How would you go about measuring the causal impact of AR on … -Gross margin per unit of land? [assume we are not interested in farmers with <.3 ha]

RDD

Impact Evaluation - Method Causal effect: change that is due to AR and not to other actors or factors (confounders) –… taking into account any other factors also changing during the program period –… taking into account any systematic differences between beneficiaries and non- beneficiaries of AR intervention It is very important that the “control group” is comparable to the “treatment group”

Evaluation Design Treatment SitesControl Sites How can we ensure that treatment and control sites are comparable?

Evaluation Design How can we ensure that treatment and control farmers are comparable?

Evaluation Design Random Treatment Assignment

Evaluation Design Random Treatment Assignment

Evaluation Design Treatment sites Control Sites BENEFICIARY FARMERS WOULD BE BENEFICIARY FARMERS A B

Y t Pre-existing Difference Impact Where do we stand? AfterBefore BASELINE A B

BENEFICIARY FARMERS Evaluation Design Treatment sitesControl sites WOULD BE BENEFICIARY FARMERS A B WOULD BE NON BENEFICIARY FARMERS NON BENEFICIARY FARMERS C D

Open issues/1 What questions we would like AR to answer? What R4D lessons can we learn from jumpstart projects? What would you carry over to longer-term AR activities? Do we need to target AR to specific farmers? Role of FtF indicators -which data to collect-?

Open issues/2 Choice of outcome indicators/variables Ethics for control (same as placebo effect in medicine) Sampling frame for randomization Sample design Statistical power for causal impact Site 1Site 2