1. Better Household Survey Data for Better Development Policies
The Living Standard Measurement Study
Gero Carletto
Manager, Center for Development Data
Development Data Group
The World Bank
Summer School in Development Economics
Prato, June 22nd, 2017

2. Today’s plan …
Show some of the (cool) things we are doing in survey work to improve the way we measure development outcomes
Entice you to use our data for your work, master, PhD thesis or your next research article
Lure you to possibly work in the field of data production
Make you wary of the data you are working with, whether collected by others, or by yourself

3. It is cool: you can be in the movies!
The Crowd & The Cloud – A PBS Documentary on Citizen Data Science
LSMS Segment on Episode 4: Citizens4Earth
Every data point has a human story

4. Still not convinced? Let’s look at some facts…
Background on household survey data landscape
What the Survey Unit at the World Bank does
LSMS and LSMS-Integrated Surveys on Agriculture
Center for Development Data (C4D2) – based in Rome
Zoom in on data and research work:
Gender disaggregated data
“Minding the (Data) Gap”: Measuring agricultural productivity
Land Area Measurement

5. Poverty data availability is better than it used to be…
MDGs
WDR Poverty
First LSMS Survey
Serajuddin et al., 2016

6. Deprivation in Household Survey Data
Data points
92 low/middle income countries do not have a multi-topic survey every 3 years, as per the President’s commitment
No data: mainly in EAP and LAC small countries
Only 1 point: mainly in AFR
77 with “extreme” deprivation (> 5-year interval)
Irregular (ad hoc) survey implementation
But also, beyond data deprivation, issues with:
Uncertainty of funding: many more (IDA) countries “at risk”
Data quality (reliability, comparability) and accessibility
E.g., only 27 of 48 SSA countries have at least two comparable surveys between
Countries
3 or more
2, interval <=5 years
2, interval >=6 years
Extreme deprivation defined as having less than a regular survey every 5 years (it applies to 77 countries).
This is an old slide which does not reflect the agreement at the Oct DC to focus on only 1 target (1 survey every 3 years). Now that we have agreed on this target, there are another 15 countries (in addition to the 77) which are also data deprived (as they only have 2 surveys in a 10-year period).
Only 1
No data
Note: number of data deprived countries estimated based on surveys conducted during

7. The SDG provide a unique opportunity, but …
SDG provides a unique opportunity to think of new data architecture for countries to be able to deliver on international data agenda while also meeting domestic demands

8. … need to go beyond indicators!
=F( ; ;
; …)
Need an Integrated approach involving …
Integration within same instrument
Cost saving
Analytical advantages … but also drawbacks!
Integration across data sources
To the plethora of problems related to increase availability and improve accuracy of LHS, also need data system to provide additional indicators in an integrated fashion …

9. Integration within same instrument: Multi-topic, multi-level
Household
Expenditures – Food & Nonfood
Education
Health
Labour
Nonfarm Enterprises
Durable Assets
Anthropometry
Food Security
Shocks
Agriculture
Plot Details
Inputs – Use & Access; Labor & Non-Labor Alike
Crops – Cultivation & Production
Implements & Machinery
Extension services
Livestock, Fisheries
Forestry?
Community
Demographics
Services
Facilities
Infrastructure
Governance
Organizations & Groups
Prices

10. … and across data sources …
Geographically linked
Geo-referenced
Validate and improve accuracy of spatial data
Sampling Units
Admin Units
Thematically linked
Small area estimation
Survey-to-survey imputation (“cross walking”)
High-frequency mobile phone surveys
Common nomenclature/coding
Administrative data
Schools, health centers, tax records, …
Market data

11. That’s what we do, since 1980, and more!
Country-owned, nationally representative surveys
Multi-topic, household-level and community data
Monitor, but more importantly understand, analyze
Typically every 3-5 years
Technical assistance to NSOs, capacity building & knowledge transfer
Support on survey design/implementation to World Bank teams
Research on best practices in survey methods & latest technology
Documents & publicly releases duly anonymized micro-data
Data Production: LSMS-ISA
Methodological & Policy Research
Capacity Building & Dissemination
Partnerships

12. LSMS Surveys are innovative …
Computer-assisted: Using Survey Solutions’ CAPI platform
Use of mobile phones to integrate face-to-face data collection
Geo-referenced: Integration with other spatially explicit data sources
Use of sensors for ‘direct measurement’:
GPS of land area
Soil testing
Water quality testing
Rain gauges
Accelerometers for physical activity
Phone apps for commuting in urban areas

13. …and open access
100+ LSMS Surveys available on the World Bank’s Microdata Catalog
Enhance data usability via automated analytical tools (ADePT)

14. Attention to data quality and timeliness
Relatively small samples size
Training, training, training…
Data entry in the field (CAFE)
Computer Assisted personal Interviews (CAPI)
Supervision, supervision, supervision…

15. Capacity Building
Annual LSMS Courses on Multi-Topic Survey Design & Sampling
Ad-hoc Workshops & Training Courses
On-the-job training with NSOs
Statistical Seminar Series
LSMS E-Learning Course
Household Survey Clinics

16. WB Household Survey Support

17. Communications Journal Articles & Special Issues
Publications, including LSMS Guidebooks
Website
Conferences, Workshops, Seminars
Quarterly LSMS E-newsletter
Facebook

18. HOW ARE WE ORGANIZED?

19. LSMS in the World Bank’s structure
Development Data Group (DECDG)
Survey Unit
LSMS
Computational Tools
Microdata Catalog
C4D2-ROME
HQS-DC

20. C4D2: The LSMS hub in Rome
The Center’s work program is built on two pillars
Agriculture
One-stop shop for integrated agricultural survey methods & data (FAO, IFAD, USDA, USAID…)
Expand usability and use of agricultural survey data
Poverty & Inequality
Partners: Bank of Italy
Initial focus on improving consumption measurement methods
About half the LSMS team (13 staff)
International training program on household surveys (with several

21. PUSHING THE MICRODATA AGENDA IN AFRICA: THE LSMS-ISA PROGRAM

22. LSMS–Integrated Surveys on Agriculture (LSMS-ISA)
Technical and financial assistance for the design and implementation of multi-topic panel household surveys, with a focus on African agriculture. The surveys:
Are integrated into national statistical systems
Are regionally and nationally representative
Panel: Tracking households and individuals
Generate individual- and plot-level data
Use field-based data entry (CAPI, CAFE)
Are geo-referenced
Are open access
20+ surveys since 2009

23. LSMS-ISA: Survey Schedule

24. LSMS-ISA research and downloads by country
Publications using LSMS-ISA: 888
Data downloads (as of March 2016)

25. LSMS-ISA Research by subject area
*Note: Results could be coded as including research on multiple topics

26. Special Issues Agriculture in Africa: Telling Facts from Myths
Partners: AfDB, World Bank Africa CE, Yale, Cornell, Maastricht
World Bank Policy Research Working Papers
Food Policy Special Issue
Nutrition & Agriculture
Partners: BMGF, IFPRI
Journal of Development Studies Special Issue
Gender & Agriculture
Partners: IFAD, Africa Gender Innovation Lab, IFPRI, FAO
World Bank-ONE Campaign Report – Leveling the Field
Agricultural Economics Special Issue

27. GENDER

28. What Do (Don’t) We Know?
Estimates of the gender gap in agricultural productivity in Africa range from 4 to 40 percent, the majority around 20 to 30 percent
However….
Household level analyses overlook within household differentiation of individual roles
Plot level analyses are largely based on non-representative case study data.

29. Putting New Data & Old Methods to Good Use
New (sex-disaggregated) data from the Living Standards Measurement Study – Integrated Surveys on Agriculture (LSMS-ISA)
Nationally- & regionally- representative multi-topic household panel surveys with an emphasis on agriculture
Geo-referenced household & plot locations for improved links to geospatial data sources, allowing better understanding of environmental constraints
Cross-country focus: Ethiopia, Malawi, Niger, Nigeria, Tanzania & Uganda
Apply decomposition methods to identify contributions of gender differences in
Levels of factors of production: Endowment effect vs.
Returns to factors of production: Structure effect
Compute relative contributions of each factor, across & within countries

30. Measuring the Gender Gap in Malawi
26% of agricultural plots managed by females
On average, female-managed plots are 25% less productive
Gap is larger for more productive farmers
80% of the average gap is driven by unequal endowments
Household adult male labor input
High-value export crop cultivation
Access to agricultural implements
Inorganic fertilizer
20% structure effect comes from:
Lower returns to adult male labor & inorganic fertilizer on female-managed plots
Domestic duties (children in particular) that lower agricultural productivity
Source: Kilic et al. (2015).

31. LSMS Data Facilitate the Verification of Common Wisdoms but …
“….[W]omen are responsible for [percent] of the agricultural labour supplied on the continent of Africa.” (UNECA, 1972; FAO, 1995)
Women produce 60 to 80 percent of the food in developing countries and 50 percent of the world’s food supply (Momsen, 1991)

32. … Often Tell a Different (More Nuanced) Story
Source: Palacios-Lopez et al. (2017).

33. METHODOLOGICAL RESEARCH

34. Overview Broad scope of LSMS methodological research since 2005
Consumption, Survey-to-Survey imputation, Labor, Income, Subjective Welfare, Asset Ownership…
Focus under the LSMS-ISA: Improving agricultural production & productivity measurement
Methodological survey experiments under
LSMS Methodological Validation Program (MVP), funded by UK Aid
Partnerships w/ (FAO) Global Strategy to Improve Agricultural & Rural Statistics & CGIAR
Livestock Data Innovation in Africa

35. Overview (2)
Completed/on-going/planned randomized household survey experiments under MVP on:
Land area
Soil fertility
Crop production
Agricultural labor inputs
Livestock production
Cognitive, non-cognitive and technical skills
Approach:
Test (old & new) methods in tandem with a gold standard
Assess relative accuracy & scale-up feasibility
Cost effectiveness, skill & training requirements, respondent burden
Document results, best practices & protocols for scale-up
Integrate validated & cost-effective methods into LSMS operations

36. LAND AREA MEASUREMENT

37. Measuring Land Area: Methodological Options
Farmer self-reported estimate
PROS
- Inexpensive
- Less missingness
CONS
- Subjective
- Complicated by traditional units
-Potential ulterior motives
Compass and rope
(aka traversing)
-Traditional gold standard for accuracy
- Eliminates subjectivity
- Time/labor intensive
- Requires travel to plot
GPS
- Significantly quicker than traversing with advantages of objective measurement
- Questions of accuracy on small plots
Remote Sensing (?)
- Potential to eliminate plot visits
- Resolution limitations
- Feasibility of boundary identification
3 Methodological experiments:
Ethiopia (n=1798), Tanzania (n=1945), Nigeria (n=494) – Total N=4237

38. Comparison of Methods (National Surveys)
Farmer self-reported estimates …
Potentially sensitive to:
Respondent characteristics
Perceived use of the data (taxation, program eligibility)
Traditional/local units of measurement
Rounding
May result in large errors, systematic biases
Source: Carletto, Savastano, Zezza (2013). “Fact or Artifact: the Impact of Measurement Errors on the Farm size - Productivity Relationship”, Journal of Development Economics.

39. Also “Heaping” in Farmers’ Self-reporting
Clear evidence of heaping/loss of precision in self-reported measurements!
Can have a significant impact on estimates of yield and productivity
Subjective farmer self-reported estimates are potentially sensitive to:
Respondent characteristics
Perceived use of the data
Traditional/local units of measurement
Rounding

40. Farmer’s self-reporting vs. GPS
Mean Plot Area (acres)
Difference
LSMS-ISA
Self-Reported Estimate
GPS %
Malawi
0.98
1.04
6.1%
Uganda
2.28
2.03
-11.0%
Tanzania
2.64
2.35
-11.4%
Niger
4.80
5.15
7.3%
source: Carletto et al., 2015

41. Farmer’s self-reporting vs. GPS
Levels & trends affected by systematic differences by plot size
Pooled Data - Plot Area Mean (acres)
Level
Plot Size Group
GPS SR
Difference
Correlation 1
< 0.5
0.30
0.61
103.3%
0.08 2
0.72
0.89
23.6%
0.11 3
1.35
1.43
5.9%
0.19 4
2 - 5
2.92
2.90
-0.7% 5
> 5
11.50
7.76
-32.5%
0.54
Total
1.59
1.55
-2.5%
0.67
source: Carletto et al., 2015

42. Farmer’s self-reporting vs. GPS by farm size
Pooled Data – Value of Output per Acre (USD, farm level)
Level
GPS Farm Size Group
Yield (GPS denominator)
Yield (SR denominator)
Difference 1
< 0.5
197.1
119.1
-40% 2
131.2
119.8
-9% 3
113.0
115.6 2% 4
2 - 5
101.4
115.7
14% 5
> 5
52.6
69.5
32%
Total
0.02 – 200.3
114.1
111.5
-2%
In this slide, comparison of yields is made across the same farms (as clustered by GPS farm size), rather than farm size as measured by GPS and SR separately
source: Carletto et al., 2015

43. Just in case you are wondering …
GPS stacks up against gold-standard, without evidence of systematic error!
LSMS Methodological Validation Studies (Ethiopia, Nigeria, Tanzania)
Plot Size Group
GPS
Compass
& Rope
Difference
Correlation
< 0.05
0.02
-1%
0.95
<0.15
0.10 2%
0.92
<0.35
0.24
0.23
0.93
<0.75
0.51
0.50
<1.25
0.97
0.96 1%
>=1.25
2.44
2.43 0%
1.00
Total
0.44
0.997
source: Carletto et al., 2016

44. GPS vs. Compass & Rope vs. Subjective
Correlation between GPS and CR measurements: 0.997
(about 0.5 between SR and CR)

45. GPS much, much faster (cheaper) than CR
Ethiopia:
GPS = 13.7 minutes
CR = 56.8 minutes
Tanzania:
GPS = 7.4 minutes
CR = 29.3 minutes

46. So GPS is the way to go, except…
Collecting GPS-based land areas not always feasible – field work protocols, lack of physical access, refusals
Substantial presence of missing values (up to 30 percent or more): Missingness shown to be non-random, with a clear bearing on agricultural productivity analyses (Kilic et al. 2017)
Survey
Rate of
Missingness
Required Spatial Coverage of
GPS-Based Plot Area Measurements
Niger Enquête Nationale sur les Conditions de Vie des nages et l’Agriculture 2011
29%
Measure all plots in the same enumeration area as the household.
Nigeria General Household Survey - Panel
2012/2013
13%
Measure all plots in the same district of the household and within 3 hours of travel, regardless of mode of transportation.
Tanzania National Panel Survey
2010/2011
22%
Measure all plots within 1 hour of travel from the household, regardless of mode of transportation.
Uganda National Panel Survey
2011/2012
44%

47. Key Take-Away Messages
Clear evidence of systematic bias in farmer self-reported area estimates
GPS serves as a time- and cost-efficient substitute for CR (in most cases)
GPS + SR: When GPS measurements are missing, use Multiple Imputation to predict missing values with a model that includes SR area
Critical rates of missing(ness) that MI can overcome is context specific and can be used to efficiently plan survey operations

48. LSMS: LOOKING AHEAD

49. LSMS: Looking Ahead
Sustained focus on high-quality household survey data production, integrated into national statistical systems
Increase non-agricultural focus (energy, water and sanitation, urban…)
LSMS +: Individually disaggregated data
Leverage existing in-country infrastructure for further methodological validation
Invest in increasing data usability
Data dissemination
Analytical tools