Sampling for Impact Evaluation -theory and application- Carlo Azzarri, IFPRI November 27, 2018 University of Rome ”La Sapienza”

Outline Introduction Targeting analysis Objective Theory of change Impact Evaluation (IE) Evaluation design Sample size Randomization Type I and II errors Panel data 11. Conclusions

Introduction This is a basic introduction to sampling for impact evaluation. Focuses mainly on Impact Evaluation and simple random sample design, but basic ideas are transferrable to more complex randomized designs and non-random sample designs. Main Question: how do we construct a sample to credibly detect a given effect size within our evaluation budget constraints?

What we are not going to talk about… Being an expert in sample design for IE Non-random designs (matching, Regression Discontinuity Designs, etc.)

Targeting Analysis Do we need to select the poorest/hungriest farmers to benefit from the project? 2. Do we need to select the most promising farmers? If so, how would we select them under 1 and 2?

Targeting Analysis: selection of farmers Is there any system already in place to target the poorest/hungriest vs. promising farmers? Collect basic information on demographics and, possibly, land assets (listing ALL farmers in project areas) Assign farmers to 2 levels Subsistence farmers (poorest)? Market-driven and most advanced (promising) farmers? Which farmers are expected to benefit more from the project?

Targeting Analysis: Community validation Lists of eligible farmers are brought to the community Only farmers whose status is confirmed by the community can be beneficiaries Target group meet jointly the two conditions: Being subsistence farmers AND poor according to the community

Objective Determine the causal effect of the project on outcomes (not only on outputs): Farmers’ wellbeing? Land productivity? Input supply, labor productivity, environment, women’s conditions, health and nutrition,…? …all of the above plus-> for whom? For which development domain? For which type of households? For which livelihood? What would be the impact with a different intervention? Is the intervention sustainable, especially at scale?

EVALUATE EFFECTIVENESS Objective (cont’d) Monitoring to track implementation efficiency (input - output) Evaluation to estimate causal effectiveness on outcomes (output - outcome) INPUTS OUTCOMES/IMPACTS OUTPUTS MONITOR EFFICIENCY EVALUATE EFFECTIVENESS $$$ BEHAVIOUR Note: Diagram from World Bank training material produced by Arianna Legovini, Lead Economist - AIEI

Theory of Change/1 Impact evaluation must be based on a set of hypotheses on the change that can be achieved as a consequence of the intervention How would you think the project can affect the life of the beneficiaries?

Theory of Change/2 ACTIVITIES OUTPUTS FIRST ORDER OUTCOMES Livelihood Strategies /Coping Strategies /Vulnerability to Shocks Project implementation Production increase Income and Expenditure Saving/ Investment General Household Expenditure School Enrolment and Attendance Expenditure on Health and Education for children School Progression Health Status Food Intake Dietary Diversity Food Security Psychological well being * Other Expenditures for children: Food, Clothing, Recreation Targeting ACTIVITIES OUTPUTS FIRST ORDER OUTCOMES SECOND ORDER OUTCOMES Asset Building THIRD ORDER OUTCOMES Labour Participation Child Labour Remittances Access to Credit Time Allocation of Children Time Allocation/ Caring arrangements/ Migration of Adults and Caregivers * Intra-household decision making * Utilization of Health Services Time and risk preferences

Impact Evaluation How would you go about measuring the causal impact of a project on … -productivity?

Impact Evaluation - Method Y Bens Impact? t Before After 13

Impact Evaluation - Method Y Bens Non Bens RCTs t Before After 14

Impact Evaluation - Method Diff in Diff Impact Y Pre-existing Difference Bens Non Bens When panel data are not available or when there is additional need to account for baseline differences between treatment and control groups, propensity score matching or propensity score weighting can be applied. The details for applying these techniques are developed in the next section. t Before After 15

Impact Evaluation - Method Diff in Diff Impact Y with Propensity Score Matching Bens Non Bens When panel data are not available or when there is additional need to account for baseline differences between treatment and control groups, propensity score matching or propensity score weighting can be applied. The details for applying these techniques are developed in the next section. Gary King: Why Propensity Scores Should Not Be Used for Matching https://www.youtube.com/watch?v=rBv39pK1iEs&t=3186s t Before After 16

Impact Evaluation What about if we have a sharp eligibility cut-off point? -assume the project targets only farmers with <.3 ha

Regression Discontinuity Design

Impact Evaluation - Method Causal effect: change that is due to our project and not to other actors or factors (confounders) … taking into account any other factors also changing during the program period … taking into account any systematic differences between beneficiaries and non- beneficiaries of AR intervention It is crucial that the “control group” is comparable to the “treatment group”

Evaluation Design How can we ensure that treatment and control villages are comparable? Treatment villages Control villages

Evaluation Design How can we ensure that treatment and control villages are comparable?

Random Treatment Assignment Evaluation Design Random Treatment Assignment

Random Treatment Assignment Evaluation Design Random Treatment Assignment

Evaluation Design A B C D Treatment villages Control villages BENEFICIARY FARMERS WOULD BE BENEFICIARY FARMERS A B NON BENEFICIARY FARMERS WOULD BE NON BENEFICIARY FARMERS C D

X Randomized assignment with multiple interventions 4. Randomize intervention 2 3. Randomize intervention 1 1. Eligible Population 2. Evaluation sample = Ineligible X = Eligible

Spillovers: How to know about direct and indirect impacts? Treatment Group A Non-Treatment Group C Affected by Spillovers Pure Control Group B E.g. Sharing food. Even when the comparison group is not directly provided with the program, it may indirectly be affected by spillovers from the treatment group. Spillovers themselves are often of policy interest because they constitute indirect program impacts. Direct program impact = A – B Spillover/indirect impact = C - B