1. Measuring the Effects of an Irrigation and Land Tenure Security Initiative in the Senegal River Valley
Baseline findings and evaluation challenges
March 22, 2017
Aravind Moorthy

2. Overview Will discuss two evaluation challenges:
Finding a comparison group of similar households
Obtaining reliable data on key outcomes
Goal: Start a dialogue about the challenges of measuring impacts for land-related programs

3. Background: The Senegal River Valley
Produces 80 percent of the rice in Senegal
Bordered on the north by the Senegal River

4. Background: Our research
Study the effects of a Millennium Challenge Corporation (MCC) initiative to:
Rehabilitate existing irrigation infrastructure and construct new infrastructure
Create a land inventory
Build local government capacity:
Training in land management and conflict resolution
Land allocation principles
Database of land-related information
Allocate land
Did the initiative increase land utilization, productivity, and investment?

5. Background: Previous steps taken
Begun by another research firm
Planned a difference-in-differences study
Treatment group: random sample of households in affected area
Comparison group: selected comparison areas and households through a largely unknown process
Used propensity score matching to increase similarity
Unable to match on outcome variables

6. Background: The evaluation
Baseline survey
Two post-implementation follow-up surveys

7. Baseline findings Comparison households were different:
More educated
Less likely to live in poverty
More likely to plant, harvest and receive crop revenue in cold season
Comparison areas had better farming conditions:
Better access to irrigation
Public institutions to assist farmers
Developed irrigation perimeters
Selection of comparison areas may have been intentional

8. Baseline findings
Key outcome data have unusually high variance: yields, revenue per hectare, costs per hectare
Example: hot season rice yield
* Diagne, M., Demont, M., Seck, P. A., & Diaw, A. (2013). Self-sufficiency policy and irrigated rice
productivity in the Senegal River Valley. Food security, 5(1),
Baseline data
Another recent study*
Mean (kg/ha)
14,434
5,320
Median (kg/ha)
4,889 --
Standard deviation (kg/ha)
413,710
1,840
Coefficient of variation (std dev / mean)
28.7
0.35

9. Challenges 1) How can we find a similar comparison group?
Affects validity of our findings
Not feasible to get baseline information for a new set of comparison households
2) Can we clean the baseline data?
Measurement error reduces ability to detect project effects

10. Finding a similar comparison group
Solution: Propensity score matching
Choose the comparison households that are most similar to treatment households.
Match on baseline outcomes and characteristics
Concern: Unobservables
Are the worst-off households in a good farming environment really similar to the average households in a poor farming environment?

11. Finding a similar comparison group
Solution: Qualitative investigation
Interviews with village leaders in comparison areas with matched households.
Are these areas similar to treatment areas, or is something else causing households to appear similar?
Solution: Sensitivity tests
Compare results from propensity score matching to results from an unmatched difference-in-differences analysis.

12. Cleaning the baseline data
Considerations:
Should not impose subjective ideas of what “looks clean”
Selectively cleaning portions of the data could add bias
Had previously looked for obvious errors (eg, “ ”).
Looked for patterns in value distributions:
A few records with extreme values?
Concentrations of records at particular values?

13. Cleaning the baseline data
Next, examined full records for outliers
Found incorrect unit conversions for land areas and production amounts
Are they really errors? Checked consistency with other fields (eg, revenue and cost data).
Checked all data – not just the most extreme values.
Done by hand

14. Follow-up survey design
Can we learn retrospectively about respondents’ baseline farming environments?
Hope to identify clear questions from qualitative research
Can we do better in terms of measurement error?

15. Thank you Questions? Suggestions? Please email me: Aravind Moorthy
Have you used propensity score matching in a land evaluation? How did it go?
Any data cleaning experiences to share?
Please me:
Aravind Moorthy