Welcome to Fantasyland: Comparing Approaches to Land Area Measurement in Household Surveys Sydney Gourlay Survey Specialist Living Standards Measurement Study Living Standards Measurement Study Team Development Research Group, The World Bank 2015 World Bank Conference on Land and Poverty

Motivation & Key Questions Land area is critical in: – Measuring yields – Land distribution – Titling schemes – Anything agriculture… Gaps in literature: – How do measurement methodologies stack-up? Is GPS as accurate as the gold- standard? – What characteristics influence measurement accuracy? – How does the measurement methodology employed influence analytical outcomes?

Measuring Land Area: Methodological Options Farmer self-reported estimate PROS - Inexpensive - Less missingness CONS - Subjective - Complicated by traditional units -Potential ulterior motives Compass and rope (aka traversing) PROS -Traditional gold standard for accuracy - Eliminates subjectivity CONS - Time/labor intensive (leading to higher costs) - Requires travel to plot GPS PROS - Significantly quicker than traversing with advantages of objective measurement CONS - Questions of accuracy on small plots (?) - Requires travel to plot Remote Sensing (?) PROS - Potential to eliminate plot visits CONS - Resolution limitations - Feasibility of boundary identification

LSMS Methodological Validation Program (MVP) UK Aid-funded project: Improving Measurement of Agricultural Productivity through Methodological Validation and Research To date, 3 highly-supervised methodological studies on land area measurement: Ethiopia Tanzania Nigeria

Methodological Validation Experiments Land and Soil Experimental Research (LASER) Improving the Measurement of Cassava Productivity Area Measurement Validation LocationEthiopiaZanzibar, TanzaniaNigeria Components: Land Area; Soil Testing; Crop Cutting Land Area; Production Diaries; Crop Cutting Land Area Partnerships: World Agroforestry Centre (ICRAF), Central Statistical Agency Ministry of Agriculture and Natural Resources, Office of the Chief Government Statistician National Bureau of Statistics Fieldwork CompleteMarch 2014July 2014May Ethiopia 1945 Tanzania 494 Nigeria 4237 Total Plots with GPS and Compass & Rope These studies included farmer self-reported estimate, GPS, and compass & rope measurement

ElevationRainfallAEZ 85 EAs 12 HH Each LASER - Ethiopia

Methodologies tested: Land area Traversing (i.e. compass-and-rope) GPS measurement Farmer self-reported area Cassava production Crop-cutting with balance scales Crop diaries with enumerator visits twice a week Crop diaries with telephone calls twice a week Farmer self-reported harvest (12-month recall) Farmer self-reported harvest (6-month recall) Cassava Productivity – Zanzibar, Tanzania

Scope of (Preliminary) Analysis Comparison of competing measurements: – Bias: Difference between two measurements (in acres) For example: GPS - CR – Absolute Value of Bias For example: |GPS – CR| – Relative Bias: Difference between two measurements (in terms of %) For example: (GPS- CR)/CR * 100% – Absolute Value of Relative Bias Analysis of systematic measurement error between measurements via OLS regression Determinants of “High Bias” plots

Comparison of Measurements: Subjective vs Objective (1) SR estimates are sensitive to respondent characteristics, including the tendency to round off numbers:

Comparison of Measurements: Subjective vs Objective (2) SR vs CR Acres Level (CR) EthiopiaTanzaniaNigeria NSRCRBias Mean Bias / Mean CRNSRCRBias Mean Bias / Mean CRNSRCRBias Mean Bias / Mean CR 1 (< 0.05 acres) % % (< 0.15 acres) % % % 3 (< 0.35 acres) % % % 4 (< 0.75 acres) % % % 5 (< 1.25 acres) % % % 6 (>= 1.25 acres) % % % Total % % % Pooled data reveals self-reported estimates are over-estimated by 51% on average (over CR measurement)

Comparison of Measurements: Subjective vs Objective (3) Ethiopia Tanzania Nigeria

Comparison of Measurements: Objective vs Objective (1) GPS vs CR Means; Acres Level (CR) Ethiopia TanzaniaNigeria GPSCRBias Mean Bias / Mean CRGPSCRBias Mean Bias / Mean CRGPSCRBias Mean Bias / Mean CR 1 (< 0.05 acres) % % (< 0.15 acres) % % % 3 (< 0.35 acres) % % % 4 (< 0.75 acres) % % % 5 (< 1.25 acres) % % % 6 (>= 1.25 acres) % % % Total % % % Pooled data reveals GPS measurements are over-stated by 1% on average (over CR measurement)

Comparison of Measurements: Objective vs Objective (2) Correlation between GPS and CR measurements: (pooled data)

Comparison of Measurements: Objective vs Objective (3) Despite high correlation of GPS and CR measurements, evidence of “high bias” or problem plots exists: High bias plots are generally: smaller than average, have a higher closing error (suggesting bias is at least partially attributable to CR), and have a more complex shape (proxied by perimeter:area ratio).

Comparison of Measurements: Objective vs Objective (4) We attempt to explain the discrepancy between GPS and CR by: Y i =L i +C i +S i +SAT i +T i +W i +e i - Y is one of the four measures of bias, -L is the measure of the plot taken using CR, -C is the closing error of the CR measure, -S is a vector of proxies for the shape of the plot, -SAT is the number of satellites the GPS device had acquired, -T is a vector of dummy variables related to tree canopy cover, -W is a vector of dummy variables related to weather conditions, and -e is a random error.

Comparison of Measurements: Objective vs Objective (5) The small difference is difficult to capture, but: Heavy tree cover & cloudy conditions slightly increase absolute value of percent deviation between measurements (in Ethiopia and Tanzania) Perimeter : Area ratio is statistically significant in all countries, suggesting that in plots with more complex shapes the absolute value of percent bias is higher. Plot area is the most significant factor…

Bias (acres)| Bias (acres)|Relative Bias (%)| Relative Bias (%) |

Comparison of Measurements: Objective vs Objective (7) Ethiopia: – GPS = 13.7 minutes – CR = 56.8 minutes Tanzania: – GPS = 7.4 minutes – CR = 29.3 minutes

Final Thoughts Lower limit to GPS use not yet clearly defined (But lower than most literature alludes to) GPS + SR – When GPS measurements are missing, impute them using the self-reported area estimates a la Kilic et al: Kilic, Zezza, Carletto, and Savastano. “Missing(ness) in Action: Selectivity Bias in GPS-Based Land Area Measurements.” World Bank Policy Research Working Paper No

Welcome to Fantasyland: Comparing Approaches to Land Area Measurement in Household Surveys Sydney Gourlay Survey Specialist Living Standards Measurement Study Living Standards Measurement Study Team Development Research Group, The World Bank 2015 World Bank Conference on Land and Poverty