Better Household Survey Data for Better Development Policies The Living Standard Measurement Study Gero Carletto Manager, Center for Development Data Development Data Group The World Bank Summer School in Development Economics Prato, June 22nd, 2017

Today’s plan … Show some of the (cool) things we are doing in survey work to improve the way we measure development outcomes Entice you to use our data for your work, master, PhD thesis or your next research article Lure you to possibly work in the field of data production Make you wary of the data you are working with, whether collected by others, or by yourself

It is cool: you can be in the movies! The Crowd & The Cloud – A PBS Documentary on Citizen Data Science LSMS Segment on Episode 4: Citizens4Earth Every data point has a human story

Still not convinced? Let’s look at some facts… Background on household survey data landscape What the Survey Unit at the World Bank does LSMS and LSMS-Integrated Surveys on Agriculture Center for Development Data (C4D2) – based in Rome Zoom in on data and research work: Gender disaggregated data “Minding the (Data) Gap”: Measuring agricultural productivity Land Area Measurement

Poverty data availability is better than it used to be… MDGs WDR Poverty First LSMS Survey Serajuddin et al., 2016

Deprivation in Household Survey Data Data points 2002-2011 92 low/middle income countries do not have a multi-topic survey every 3 years, as per the President’s commitment No data: mainly in EAP and LAC small countries Only 1 point: mainly in AFR 77 with “extreme” deprivation (> 5-year interval) Irregular (ad hoc) survey implementation But also, beyond data deprivation, issues with: Uncertainty of funding: many more (IDA) countries “at risk” Data quality (reliability, comparability) and accessibility E.g., only 27 of 48 SSA countries have at least two comparable surveys between 1990-2012 Countries 3 or more 2, interval <=5 years 2, interval >=6 years Extreme deprivation defined as having less than a regular survey every 5 years (it applies to 77 countries). This is an old slide which does not reflect the agreement at the Oct DC to focus on only 1 target (1 survey every 3 years). Now that we have agreed on this target, there are another 15 countries (in addition to the 77) which are also data deprived (as they only have 2 surveys in a 10-year period). Only 1 No data Note: number of data deprived countries estimated based on surveys conducted during 2002-2011

The SDG provide a unique opportunity, but … SDG provides a unique opportunity to think of new data architecture for countries to be able to deliver on international data agenda while also meeting domestic demands

… need to go beyond indicators! =F( ; ; ; …) Need an Integrated approach involving … Integration within same instrument Cost saving Analytical advantages … but also drawbacks! Integration across data sources To the plethora of problems related to increase availability and improve accuracy of LHS, also need data system to provide additional indicators in an integrated fashion …

Integration within same instrument: Multi-topic, multi-level Household Expenditures – Food & Nonfood Education Health Labour Nonfarm Enterprises Durable Assets Anthropometry Food Security Shocks Agriculture Plot Details Inputs – Use & Access; Labor & Non-Labor Alike Crops – Cultivation & Production Implements & Machinery Extension services Livestock, Fisheries Forestry? Community Demographics Services Facilities Infrastructure Governance Organizations & Groups Prices

… and across data sources … Geographically linked Geo-referenced Validate and improve accuracy of spatial data Sampling Units Admin Units Thematically linked Small area estimation Survey-to-survey imputation (“cross walking”) High-frequency mobile phone surveys Common nomenclature/coding Administrative data Schools, health centers, tax records, … Market data

That’s what we do, since 1980, and more! Country-owned, nationally representative surveys Multi-topic, household-level and community data Monitor, but more importantly understand, analyze Typically every 3-5 years Technical assistance to NSOs, capacity building & knowledge transfer Support on survey design/implementation to World Bank teams Research on best practices in survey methods & latest technology Documents & publicly releases duly anonymized micro-data Data Production: LSMS-ISA Methodological & Policy Research Capacity Building & Dissemination Partnerships

LSMS Surveys are innovative … Computer-assisted: Using Survey Solutions’ CAPI platform Use of mobile phones to integrate face-to-face data collection Geo-referenced: Integration with other spatially explicit data sources Use of sensors for ‘direct measurement’: GPS of land area Soil testing Water quality testing Rain gauges Accelerometers for physical activity Phone apps for commuting in urban areas

…and open access 100+ LSMS Surveys available on the World Bank’s Microdata Catalog Enhance data usability via automated analytical tools (ADePT)

Attention to data quality and timeliness Relatively small samples size Training, training, training… Data entry in the field (CAFE) Computer Assisted personal Interviews (CAPI) Supervision, supervision, supervision…

Capacity Building Annual LSMS Courses on Multi-Topic Survey Design & Sampling Ad-hoc Workshops & Training Courses On-the-job training with NSOs Statistical Seminar Series LSMS E-Learning Course Household Survey Clinics

WB Household Survey Support

Communications Journal Articles & Special Issues Publications, including LSMS Guidebooks Website Conferences, Workshops, Seminars Quarterly LSMS E-newsletter Facebook

HOW ARE WE ORGANIZED?

LSMS in the World Bank’s structure Development Data Group (DECDG) Survey Unit LSMS Computational Tools Microdata Catalog C4D2-ROME HQS-DC

C4D2: The LSMS hub in Rome The Center’s work program is built on two pillars Agriculture One-stop shop for integrated agricultural survey methods & data (FAO, IFAD, USDA, USAID…) Expand usability and use of agricultural survey data Poverty & Inequality Partners: Bank of Italy Initial focus on improving consumption measurement methods About half the LSMS team (13 staff) International training program on household surveys (with several

PUSHING THE MICRODATA AGENDA IN AFRICA: THE LSMS-ISA PROGRAM

LSMS–Integrated Surveys on Agriculture (LSMS-ISA) Technical and financial assistance for the design and implementation of multi-topic panel household surveys, with a focus on African agriculture. The surveys: Are integrated into national statistical systems Are regionally and nationally representative Panel: Tracking households and individuals Generate individual- and plot-level data Use field-based data entry (CAPI, CAFE) Are geo-referenced Are open access 20+ surveys since 2009

LSMS-ISA: Survey Schedule

LSMS-ISA research and downloads by country Publications using LSMS-ISA: 888 Data downloads (as of March 2016)

LSMS-ISA Research by subject area *Note: Results could be coded as including research on multiple topics

Special Issues Agriculture in Africa: Telling Facts from Myths Partners: AfDB, World Bank Africa CE, Yale, Cornell, Maastricht World Bank Policy Research Working Papers Food Policy Special Issue Nutrition & Agriculture Partners: BMGF, IFPRI Journal of Development Studies Special Issue Gender & Agriculture Partners: IFAD, Africa Gender Innovation Lab, IFPRI, FAO World Bank-ONE Campaign Report – Leveling the Field Agricultural Economics Special Issue

GENDER

What Do (Don’t) We Know? Estimates of the gender gap in agricultural productivity in Africa range from 4 to 40 percent, the majority around 20 to 30 percent However…. Household level analyses overlook within household differentiation of individual roles Plot level analyses are largely based on non-representative case study data.

Putting New Data & Old Methods to Good Use New (sex-disaggregated) data from the Living Standards Measurement Study – Integrated Surveys on Agriculture (LSMS-ISA) Nationally- & regionally- representative multi-topic household panel surveys with an emphasis on agriculture Geo-referenced household & plot locations for improved links to geospatial data sources, allowing better understanding of environmental constraints Cross-country focus: Ethiopia, Malawi, Niger, Nigeria, Tanzania & Uganda Apply decomposition methods to identify contributions of gender differences in Levels of factors of production: Endowment effect vs. Returns to factors of production: Structure effect Compute relative contributions of each factor, across & within countries

Measuring the Gender Gap in Malawi 26% of agricultural plots managed by females On average, female-managed plots are 25% less productive Gap is larger for more productive farmers 80% of the average gap is driven by unequal endowments Household adult male labor input High-value export crop cultivation Access to agricultural implements Inorganic fertilizer 20% structure effect comes from: Lower returns to adult male labor & inorganic fertilizer on female-managed plots Domestic duties (children in particular) that lower agricultural productivity Source: Kilic et al. (2015).

LSMS Data Facilitate the Verification of Common Wisdoms but … “….[W]omen are responsible for 60-80 [percent] of the agricultural labour supplied on the continent of Africa.” (UNECA, 1972; FAO, 1995) Women produce 60 to 80 percent of the food in developing countries and 50 percent of the world’s food supply (Momsen, 1991)

… Often Tell a Different (More Nuanced) Story Source: Palacios-Lopez et al. (2017).

METHODOLOGICAL RESEARCH

Overview Broad scope of LSMS methodological research since 2005 Consumption, Survey-to-Survey imputation, Labor, Income, Subjective Welfare, Asset Ownership… Focus under the LSMS-ISA: Improving agricultural production & productivity measurement Methodological survey experiments under LSMS Methodological Validation Program (MVP), funded by UK Aid Partnerships w/ (FAO) Global Strategy to Improve Agricultural & Rural Statistics & CGIAR Livestock Data Innovation in Africa

Overview (2) Completed/on-going/planned randomized household survey experiments under MVP on: Land area Soil fertility Crop production Agricultural labor inputs Livestock production Cognitive, non-cognitive and technical skills Approach: Test (old & new) methods in tandem with a gold standard Assess relative accuracy & scale-up feasibility Cost effectiveness, skill & training requirements, respondent burden Document results, best practices & protocols for scale-up Integrate validated & cost-effective methods into LSMS operations

LAND AREA MEASUREMENT

Measuring Land Area: Methodological Options Farmer self-reported estimate PROS - Inexpensive - Less missingness CONS - Subjective - Complicated by traditional units -Potential ulterior motives Compass and rope (aka traversing) -Traditional gold standard for accuracy - Eliminates subjectivity - Time/labor intensive - Requires travel to plot GPS - Significantly quicker than traversing with advantages of objective measurement - Questions of accuracy on small plots Remote Sensing (?) - Potential to eliminate plot visits - Resolution limitations - Feasibility of boundary identification 3 Methodological experiments: Ethiopia (n=1798), Tanzania (n=1945), Nigeria (n=494) – Total N=4237

Comparison of Methods (National Surveys) Farmer self-reported estimates … Potentially sensitive to: Respondent characteristics Perceived use of the data (taxation, program eligibility) Traditional/local units of measurement Rounding May result in large errors, systematic biases Source: Carletto, Savastano, Zezza (2013). “Fact or Artifact: the Impact of Measurement Errors on the Farm size - Productivity Relationship”, Journal of Development Economics.

Also “Heaping” in Farmers’ Self-reporting Clear evidence of heaping/loss of precision in self-reported measurements! Can have a significant impact on estimates of yield and productivity Subjective farmer self-reported estimates are potentially sensitive to: Respondent characteristics Perceived use of the data Traditional/local units of measurement Rounding

Farmer’s self-reporting vs. GPS Mean Plot Area (acres) Difference LSMS-ISA Self-Reported Estimate GPS % Malawi 0.98 1.04 6.1% Uganda 2.28 2.03 -11.0% Tanzania 2.64 2.35 -11.4% Niger 4.80 5.15 7.3% source: Carletto et al., 2015

Farmer’s self-reporting vs. GPS Levels & trends affected by systematic differences by plot size Pooled Data - Plot Area Mean (acres) Level Plot Size Group GPS SR Difference Correlation 1 < 0.5 0.30 0.61 103.3% 0.08 2 0.5 - 0.99 0.72 0.89 23.6% 0.11 3 1 - 1.99 1.35 1.43 5.9% 0.19 4 2 - 5 2.92 2.90 -0.7% 5 > 5 11.50 7.76 -32.5% 0.54 Total 0.01 - 97.02 1.59 1.55 -2.5% 0.67 source: Carletto et al., 2015

Farmer’s self-reporting vs. GPS by farm size Pooled Data – Value of Output per Acre (USD, farm level) Level GPS Farm Size Group Yield (GPS denominator) Yield (SR denominator) Difference 1 < 0.5 197.1 119.1 -40% 2 0.5 - 0.99 131.2 119.8 -9% 3 1 - 1.99 113.0 115.6 2% 4 2 - 5 101.4 115.7 14% 5 > 5 52.6 69.5 32% Total 0.02 – 200.3 114.1 111.5 -2% In this slide, comparison of yields is made across the same farms (as clustered by GPS farm size), rather than farm size as measured by GPS and SR separately source: Carletto et al., 2015

Just in case you are wondering … GPS stacks up against gold-standard, without evidence of systematic error! LSMS Methodological Validation Studies (Ethiopia, Nigeria, Tanzania) Plot Size Group GPS Compass & Rope Difference Correlation < 0.05 0.02 -1% 0.95 <0.15 0.10 2% 0.92 <0.35 0.24 0.23 0.93 <0.75 0.51 0.50 <1.25 0.97 0.96 1% >=1.25 2.44 2.43 0% 1.00 Total 0.44 0.997 source: Carletto et al., 2016

GPS vs. Compass & Rope vs. Subjective Correlation between GPS and CR measurements: 0.997 (about 0.5 between SR and CR)

GPS much, much faster (cheaper) than CR Ethiopia: GPS = 13.7 minutes CR = 56.8 minutes Tanzania: GPS = 7.4 minutes CR = 29.3 minutes

So GPS is the way to go, except… Collecting GPS-based land areas not always feasible – field work protocols, lack of physical access, refusals Substantial presence of missing values (up to 30 percent or more): Missingness shown to be non-random, with a clear bearing on agricultural productivity analyses (Kilic et al. 2017) Survey Rate of Missingness Required Spatial Coverage of GPS-Based Plot Area Measurements Niger Enquête Nationale sur les Conditions de Vie des nages et l’Agriculture 2011 29% Measure all plots in the same enumeration area as the household. Nigeria General Household Survey - Panel 2012/2013 13% Measure all plots in the same district of the household and within 3 hours of travel, regardless of mode of transportation. Tanzania National Panel Survey 2010/2011 22% Measure all plots within 1 hour of travel from the household, regardless of mode of transportation. Uganda National Panel Survey 2011/2012 44%

Key Take-Away Messages Clear evidence of systematic bias in farmer self-reported area estimates GPS serves as a time- and cost-efficient substitute for CR (in most cases) GPS + SR: When GPS measurements are missing, use Multiple Imputation to predict missing values with a model that includes SR area Critical rates of missing(ness) that MI can overcome is context specific and can be used to efficiently plan survey operations

LSMS: LOOKING AHEAD

LSMS: Looking Ahead Sustained focus on high-quality household survey data production, integrated into national statistical systems Increase non-agricultural focus (energy, water and sanitation, urban…) LSMS +: Individually disaggregated data Leverage existing in-country infrastructure for further methodological validation Invest in increasing data usability Data dissemination Analytical tools