Download presentation
Presentation is loading. Please wait.
Published byMartin Hansson Modified over 5 years ago
1
Sampling for Impact Evaluation -theory and application-
Carlo Azzarri, IFPRI November 27, 2018 University of Rome ”La Sapienza”
2
Outline Introduction Targeting analysis Objective Theory of change
Impact Evaluation (IE) Evaluation design Sample size Randomization Type I and II errors Panel data 11. Conclusions
3
Introduction This is a basic introduction to sampling for impact evaluation. Focuses mainly on Impact Evaluation and simple random sample design, but basic ideas are transferrable to more complex randomized designs and non-random sample designs. Main Question: how do we construct a sample to credibly detect a given effect size within our evaluation budget constraints?
4
What we are not going to talk about…
Being an expert in sample design for IE Non-random designs (matching, Regression Discontinuity Designs, etc.)
5
Targeting Analysis Do we need to select the poorest/hungriest farmers to benefit from the project? 2. Do we need to select the most promising farmers? If so, how would we select them under 1 and 2?
6
Targeting Analysis: selection of farmers
Is there any system already in place to target the poorest/hungriest vs. promising farmers? Collect basic information on demographics and, possibly, land assets (listing ALL farmers in project areas) Assign farmers to 2 levels Subsistence farmers (poorest)? Market-driven and most advanced (promising) farmers? Which farmers are expected to benefit more from the project?
7
Targeting Analysis: Community validation
Lists of eligible farmers are brought to the community Only farmers whose status is confirmed by the community can be beneficiaries Target group meet jointly the two conditions: Being subsistence farmers AND poor according to the community
8
Objective Determine the causal effect of the project on outcomes (not only on outputs): Farmers’ wellbeing? Land productivity? Input supply, labor productivity, environment, women’s conditions, health and nutrition,…? …all of the above plus-> for whom? For which development domain? For which type of households? For which livelihood? What would be the impact with a different intervention? Is the intervention sustainable, especially at scale?
9
EVALUATE EFFECTIVENESS
Objective (cont’d) Monitoring to track implementation efficiency (input - output) Evaluation to estimate causal effectiveness on outcomes (output - outcome) INPUTS OUTCOMES/IMPACTS OUTPUTS MONITOR EFFICIENCY EVALUATE EFFECTIVENESS $$$ BEHAVIOUR Note: Diagram from World Bank training material produced by Arianna Legovini, Lead Economist - AIEI
10
Theory of Change/1 Impact evaluation must be based on a set of hypotheses on the change that can be achieved as a consequence of the intervention How would you think the project can affect the life of the beneficiaries?
11
Theory of Change/2 ACTIVITIES OUTPUTS FIRST ORDER OUTCOMES
Livelihood Strategies /Coping Strategies /Vulnerability to Shocks Project implementation Production increase Income and Expenditure Saving/ Investment General Household Expenditure School Enrolment and Attendance Expenditure on Health and Education for children School Progression Health Status Food Intake Dietary Diversity Food Security Psychological well being * Other Expenditures for children: Food, Clothing, Recreation Targeting ACTIVITIES OUTPUTS FIRST ORDER OUTCOMES SECOND ORDER OUTCOMES Asset Building THIRD ORDER OUTCOMES Labour Participation Child Labour Remittances Access to Credit Time Allocation of Children Time Allocation/ Caring arrangements/ Migration of Adults and Caregivers * Intra-household decision making * Utilization of Health Services Time and risk preferences
12
Impact Evaluation How would you go about measuring the causal impact of a project on … -productivity?
13
Impact Evaluation - Method
Y Bens Impact? t Before After 13
14
Impact Evaluation - Method
Y Bens Non Bens RCTs t Before After 14
15
Impact Evaluation - Method
Diff in Diff Impact Y Pre-existing Difference Bens Non Bens When panel data are not available or when there is additional need to account for baseline differences between treatment and control groups, propensity score matching or propensity score weighting can be applied. The details for applying these techniques are developed in the next section. t Before After 15
16
Impact Evaluation - Method
Diff in Diff Impact Y with Propensity Score Matching Bens Non Bens When panel data are not available or when there is additional need to account for baseline differences between treatment and control groups, propensity score matching or propensity score weighting can be applied. The details for applying these techniques are developed in the next section. Gary King: Why Propensity Scores Should Not Be Used for Matching t Before After 16
17
Impact Evaluation What about if we have a sharp eligibility cut-off point? -assume the project targets only farmers with <.3 ha
18
Regression Discontinuity Design
19
Impact Evaluation - Method
Causal effect: change that is due to our project and not to other actors or factors (confounders) … taking into account any other factors also changing during the program period … taking into account any systematic differences between beneficiaries and non- beneficiaries of AR intervention It is crucial that the “control group” is comparable to the “treatment group”
20
Evaluation Design How can we ensure that treatment and control villages are comparable? Treatment villages Control villages
21
Evaluation Design How can we ensure that treatment and control villages are comparable?
22
Random Treatment Assignment
Evaluation Design Random Treatment Assignment
23
Random Treatment Assignment
Evaluation Design Random Treatment Assignment
24
Evaluation Design A B C D Treatment villages Control villages
BENEFICIARY FARMERS WOULD BE BENEFICIARY FARMERS A B NON BENEFICIARY FARMERS WOULD BE NON BENEFICIARY FARMERS C D
25
X Randomized assignment with multiple interventions
4. Randomize intervention 2 3. Randomize intervention 1 1. Eligible Population 2. Evaluation sample = Ineligible X = Eligible
26
Spillovers: How to know about direct and indirect impacts?
Treatment Group A Non-Treatment Group C Affected by Spillovers Pure Control Group B E.g. Sharing food. Even when the comparison group is not directly provided with the program, it may indirectly be affected by spillovers from the treatment group. Spillovers themselves are often of policy interest because they constitute indirect program impacts. Direct program impact = A – B Spillover/indirect impact = C - B
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.