Measuring the Effects of an Irrigation and Land Tenure Security Initiative in the Senegal River Valley Baseline findings and evaluation challenges March 22, 2017 Aravind Moorthy

Overview Will discuss two evaluation challenges: Finding a comparison group of similar households Obtaining reliable data on key outcomes Goal: Start a dialogue about the challenges of measuring impacts for land-related programs

Background: The Senegal River Valley Produces 80 percent of the rice in Senegal Bordered on the north by the Senegal River

Background: Our research Study the effects of a Millennium Challenge Corporation (MCC) initiative to: Rehabilitate existing irrigation infrastructure and construct new infrastructure Create a land inventory Build local government capacity: Training in land management and conflict resolution Land allocation principles Database of land-related information Allocate land Did the initiative increase land utilization, productivity, and investment?

Background: Previous steps taken Begun by another research firm Planned a difference-in-differences study Treatment group: random sample of households in affected area Comparison group: selected comparison areas and households through a largely unknown process Used propensity score matching to increase similarity Unable to match on outcome variables

Background: The evaluation Baseline survey Two post-implementation follow-up surveys

Baseline findings Comparison households were different: More educated Less likely to live in poverty More likely to plant, harvest and receive crop revenue in cold season Comparison areas had better farming conditions: Better access to irrigation Public institutions to assist farmers Developed irrigation perimeters Selection of comparison areas may have been intentional

Baseline findings Key outcome data have unusually high variance: yields, revenue per hectare, costs per hectare Example: hot season rice yield * Diagne, M., Demont, M., Seck, P. A., & Diaw, A. (2013). Self-sufficiency policy and irrigated rice productivity in the Senegal River Valley. Food security, 5(1), 55-68. Baseline data Another recent study* Mean (kg/ha) 14,434 5,320 Median (kg/ha) 4,889 -- Standard deviation (kg/ha) 413,710 1,840 Coefficient of variation (std dev / mean) 28.7 0.35

Challenges 1) How can we find a similar comparison group? Affects validity of our findings Not feasible to get baseline information for a new set of comparison households 2) Can we clean the baseline data? Measurement error reduces ability to detect project effects

Finding a similar comparison group Solution: Propensity score matching Choose the comparison households that are most similar to treatment households. Match on baseline outcomes and characteristics Concern: Unobservables Are the worst-off households in a good farming environment really similar to the average households in a poor farming environment?

Finding a similar comparison group Solution: Qualitative investigation Interviews with village leaders in comparison areas with matched households. Are these areas similar to treatment areas, or is something else causing households to appear similar? Solution: Sensitivity tests Compare results from propensity score matching to results from an unmatched difference-in-differences analysis.

Cleaning the baseline data Considerations: Should not impose subjective ideas of what “looks clean” Selectively cleaning portions of the data could add bias Had previously looked for obvious errors (eg, “999999999”). Looked for patterns in value distributions: A few records with extreme values? Concentrations of records at particular values?

Cleaning the baseline data Next, examined full records for outliers Found incorrect unit conversions for land areas and production amounts Are they really errors? Checked consistency with other fields (eg, revenue and cost data). Checked all data – not just the most extreme values. Done by hand

Follow-up survey design Can we learn retrospectively about respondents’ baseline farming environments? Hope to identify clear questions from qualitative research Can we do better in terms of measurement error?

Thank you Questions? Suggestions? Please email me: Aravind Moorthy Have you used propensity score matching in a land evaluation? How did it go? Any data cleaning experiences to share? Please email me: Aravind Moorthy amoorthy@mathematica-mpr.com