Cassava Production Measurement and Variety Identification in Household Surveys: Results from a Randomized Survey Experiment in Malawi HEATHER MOYLAN Survey Specialist Living Standards Measurement Study Development Data Group – Survey Unit – World Bank hmoylan@worldbank.org Co-Authors: TALIP KILIC, JOHN ILUKOR and INNOCENT PANGAPANGA PHIRI World Bank Land and Poverty Conference Washington, D.C. – March 24, 2017

Background: LSMS Methodological Research Broad scope of LSMS methodological research since 2005 Consumption, Labor, Income, Subjective Welfare, Asset Ownership Focus under the LSMS-Integrated Surveys on Agriculture (LSMS-ISA): Improving agricultural production & productivity measurement Methodological survey experiments under Minding the (Agricultural) Data Gap Research Program, funded by UK Aid Partnerships w/ (FAO) Global Strategy to Improve Agricultural and Rural Statistics; CGIAR Standing Panel on Impact Assessment; World Bank Innovations in Big Data Analytics; Stanford University; Livestock Data Innovation (LDI) in Africa Project

Background: LSMS Methodological Research (2) Completed/on-going/planned randomized household survey experiments on: Land Area, Soil Fertility, Rainfall, Crop Production, Agricultural Labor, Livestock Production, Cognitive, Non-cognitive & Technical skills Approach: Test (old & new) methods in tandem with a gold standard Assess relative accuracy & scale-up feasibility Cost effectiveness, skill & training requirements, respondent burden Document results, best practices & protocols for scale-up Integrate validated & cost-effective methods into LSMS operations Today’s focus: Randomized household survey experiment in Malawi on measuring cassava production, productivity, and variety identification

The Root of the Measurement Problem : Cassava is an important staple crop in many countries, particularly in sub-Saharan Africa, and is often planted for purposes of food security. However, cassava and other continuous crops that are often harvested in small quantities over extended periods of time pose particular challenges for the estimation of crop production and yield, due to the length and frequency of their harvesting. What are the difficulties associated with measuring production in smallholder production systems? First, farmers do not keep records. Second, recall is widely used in household survey operations but does not always work. Third, crop cutting is more expensive as well as time & resource-intensive. Fourth, use of non-standard measurement units of varying sizes is rampant. Fifth, there are different measurement units along the value chain, crops appearing in different conditions. Sixth, the development of conversion factors for expressing product-condition-non-standard unit combinations in KG-condition-equivalent terms in its infancy. Finally, improved approaches to farmer-reported production measurement need validation.

Summary of Main Findings Measurement error in self-reported plot areas vis-à-vis their GPS-based counterparts is worrisome, in line with the recent literature Collecting daily cassava harvest information via monthly self-administered diaries with bi-weekly phone calls generates arguably the most accurate annual household cassava production estimates Collecting daily cassava harvest information via two 6-month recall surveys performs as well as self-administered monthly diaries with bi-weekly in-person visits (traditionally-considered as gold-standard) Crop cutting provides an upper-bound household-level annual cassava yields, in line with expectations Measurement error in self-reported cassava variety information vis-à-vis DNA fingerprinting is prohibitively high, in line with the emerging empirical evidence on alternative crops and countries

CVIP: Methodological Experiment on Measuring Cassava Production, Productivity, and Variety Identification Funding LSMS Minding the (Agricultural) Data Gap Research Program, funded by UKAid Global Strategy to Improve Agricultural and Rural Statistics Objectives Test subjective approaches to measurement vis-à-vis objective methods for land area; cassava production and productivity; & cassava variety identification Technical Partnerships National Statistical Office (Implementing Agency) CGIAR Standing Panel on Impact Assessment (Variety Identification) CVIP: Methodological Experiment on Measuring Cassava Production, Productivity, and Variety Identification

CVIP: Context Along lakeshore in North & Central Regions: Cassava is used as a staple food crop along the shores of Malawi (Nkhatabay, Nkhotakota) Non-cassava belt areas in Southern & Central: Cassava is mainly grown for sale or use as a snack (Lilongwe, Zomba, Mulanje) 5 of the top cassava-producing districts were chosen within Malawi Note that differences wrt Nkhatabay Notes: Reference group for the test of mean differences is Nkhatabay. ***/**/* denote statistical significance at the 1/5/10 percent level, respectively.

CVIP: Sampling District Selection Consultations Ministry of Agriculture & Food Security (MoAFS) – Department of Agricultural Research Services Lilongwe University of Agriculture & Natural Resources International Institute of Tropical Agriculture (IITA) 5 top cassava-producing districts identified in the Central and Southern regions of Malawi 1 district from each agro-ecological zone (Agricultural Development Division) Extension Planning Area (EPA) Selection District agricultural development officers, crop specialists in each of the 5 districts Cassava-producing EPAs identified based MoAFS production figures Enumeration Area (EAs) Selection Identified universe of census EAs in each cassava EPA of interest 9 EAs selected with PPS in each district, based on 2008 census EA-level household counts

Households per treatment CVIP: Sampling (2) 5 Districts 45 Enumeration Areas 1218 Households ~ 305 Households per treatment Household Selection Listing exercise conducted in each sampled EA to identify cassava-cultivating households 28 cassava households randomly selected in each EA; 7 households randomly assigned to each of the 4 treatment arms Randomization successful Note - attrition

CVIP: Methods Methods Tested: Cassava Production Crop-cutting (5mx5m subplot) with balance scales One plot/household for all households Crop diaries with enumerator visits twice/week (D1) Crop diaries with telephone calls twice/ week (D2) Farmer-reported harvest (two visits, 6-month recall) (R1) Farmer-reported harvest (single visit, 12-month recall) (R2) Land area GPS measurement Farmer-reported area Variety DNA fingerprinting of leaf samples obtained at post-planting Farmer-reported variety names, types & attributes w/ photo aid Idea here is to assume that the extrapolate mark as an upper bound or potential production over the 12 month period and use this as a reference point for

A B C D CVIP: Crop Cutting Crop-Cutting Strategy 1 Sub-Plot per Plot: Survey Solutions CAPI Application selects 1 plot at random for crop cutting Cassava weighed fresh A B |---------- 5 m ------------| C D |------------- 5 m --------------|

CVIP: Crop Cutting - Laying the 5x5 Sub-Plot D

CVIP: Crop Cutting - Harvesting B C D Timing based on when RESPONDENT is ready for harvest, Enumerator uses crop cut assistant to help in communication with household, helps to harvest and then weighs the cassava with the respondent, uses balance scale like the one distributed to all diary households

CVIP: Cassava Production Diaries B C D Taught them how to use the scales, selected a literate household member as the keeper of the diary to assist plot managers in filling

CVIP: Recall Interviews B C D

Fieldwork Implementing agency: Malawi National Statistical Office (NSO) 5 mobile teams, 1 per district (1 Supervisor, 3 enumerators) Computer-assisted personal interviewing (CAPI) application designed in Survey Solutions Questionnaire Survey Solutions CAPI Mode Special CAPI Features Household Questionnaire Census Random selection of crop cutting plot GPS coordinates taken on tablet Crop Cutting Questionnaire Sample Scanned barcodes for leaf samples Monthly Diaries (Paper) & Sample Enumerators entered data after paper diaries collected each month 6-Month Recall First visit data on gardens & plots fed forward 12-Month Recall

Fieldwork (2) Timeline: Initial Visits Laying Crop Cuts Leaf Sample Collection July 2015 – September 2016 Diary Visits Diary Phone Calls Starting July 2015 Crop cutting 6-month recall February 2016 12-Month recall July–August 2016

Land Area Measurement

Self-Reported vs. GPS-Based Cassava Plot Area

Production & Yield Measurement

Overview Production measured as total kilograms harvested per household across the 12-month period D1 & D2 Treatment Arms: Diary households all received scales; reported daily (fresh) harvest as measured in non-standard harvest units as well as in KGs R1 & R2 Treatment Arms: Total production solicited in recall interviews – two visits to R1, 6-months apart, with a 6-month reference period at each visit; single visit to R2 with a 12-month reference period Production typically captured in non-standard harvest units Results robust to various formulations conversion factors created from the diary sample

Estimation of Survey Treatment Effects 1 𝑃𝑟𝑜𝑑𝑢𝑐𝑡𝑖𝑜𝑛 𝑖 =𝛼+ 𝛽 1 𝐷2 𝑖 + 𝛽 2 𝑅2 𝑖 + 𝛽 3 𝑅1 𝑖 + 𝛾𝐶+ 𝜀 𝑖 2 𝑌𝑖𝑒𝑙𝑑 𝑖 =𝛼+ 𝛽 1 𝐷1 𝑖 +𝛽 2 𝐷2 𝑖 + 𝛽 3 𝑅2 𝑖 + 𝛽 4 𝑅1 𝑖 + 𝛾𝐶+ 𝜀 𝑖 i represents household, ∝& 𝜀 are constant & error terms, respectively Equation 1 – Outcome: Annual Cassava Production (KGs) – Comparison Category: D1 Equation 2 – Outcome: Annual Cassava Yield (KGs/Ha) – Comparison Category: CC Crop cutting-based yield extrapolated from sub-plot to cassava farm (measured with GPS) D2, R1 & R2 represent identifiers for diary-phone, 6-month recall & 12-month recall, respectively C is a vector of household attributes – inclusion of which has no bearing on our findings Standard errors clustered at the EA-level for Equation 1, at the household-level for Equation 2 C includes household size, dependency ratio, highest grade in household, wealth index, land area cultivated with cassava, cassava sales, gender and age of household head, household non-farm enterprise, household paid employment. The fact that the control variables are included/excluded does not have a bearing on the estimated survey treatment effects – in support of successful randomization.

Core Results You need to note at some point, our failed attempts at unpacking the heterogeneity of treatment effects, explored through interactions of the survey treatment identifiers with household attributes. . And please make sure to note that this heterogeneity analysis is only descriptive, given the endogenous control variables that you are interacting with your survey treatment dummies.

Monthly & Semi-Annual Production Estimates by Treatment Arm

Variety Identification

Why Variety Identification Matters? Justifying investment in crop technology investments Are farmers adopting modern agricultural technologies? Are these technologies performing as expected? Are technologies being used correctly? Assessing the performance of the extension system Do farmers know what they are planting? Are farmers properly using the agricultural technologies? Assessing the performance of the seed system Are the seeds /cultivars of required standard or quality? Are vitro plants in the gene bank true to their type?

Methods Subjective Methods Farmer-Reported variety names, local vs. improved variety classification Farmer-Reported Morphological Attributes based on a photo aid of 13 attributes Objective Gold Standard DNA Fingerprinting of Leaf Samples from crop cut sub-plots

How Do Subjective Methods Perform vis-à-vis DNA Fingerprinting? All the farmers could uniquely state variety planted but only 30% were correctly identified Farmer-reported morph. attributes perform poorly in unique & correct variety identification

Considerable Measurement Error in Variety Identification Based on Farmer-Reporting Popular varieties (Beatrice & Manyokola): More likely to be correctly identified by farmers Less popular varieties: High rate of mis-identification Only 0.18% of the field samples were actually improved, while the farmer-reported incidence of improved variety cultivation was 21% Likelihood of correct variety identification Negatively correlated with varietal mixing Positively correlated with access to extension NO correlation with commercial seed acquisition DNA Fingerprinting Farmer-Reporting

Key Take-Away Messages LAND AREA Measurement error in self-reported plot areas vis-à-vis their GPS-based counterparts is worrisome, in line with the recent literature At the mean, farmer-reported plot area is twice as much as its GPS-based counterpart PRODUCTION Diary-Phone generates arguably the most accurate annual cassava production estimates 6-Month Recall performs as well as Diary-Visit; traditionally-considered as gold-standard 12-Month Recall underestimates cassava production by a significant margin Crop cutting provides an upper-bound for annual cassava yields, in line with expectations Results in line with an earlier, near-identical methodological experiment in Zanzibar 6-Month Recall: A viable alternative to existing methods for cassava production measurement

Key Take-Away Messages (2) VARIETY IDENTIFICATION Error in self-reported cassava variety information vis-à-vis DNA fingerprinting is prohibitively high, in line with the emerging empirical evidence on alternative crops & countries Call for increased consideration for integrating DNA fingerprinting into survey operations Dramatically improves the extent of correct varietal identification Reveals weaknesses in seed and extension systems Cost per 1 sample per household: USD 25 (2016 Prices) Need assess the effect of variety mis-identification on adoption & impact analyses

Next Steps Better understand mechanisms for treatment effects: estimations of differences by district/production systems, (primary) cassava variety, socioeconomic attributes Better document relative costs of each approach to production measurement Analysis of the data from the Malawi (mini) consumption experiment Feed the research into the FAO Global Strategy to Improve Agricultural and Rural Statistics Handbook on Measuring Crop Area, Yield, and Production

Cassava Production Measurement and Variety Identification in Household Surveys: Results from a Randomized Survey Experiment in Malawi HEATHER MOYLAN Survey Specialist Living Standards Measurement Study Development Data Group – Survey Unit – World Bank hmoylan@worldbank.org Co-Authors: TALIP KILIC, JOHN ILUKOR and INNOCENT PANGAPANGA PHIRI World Bank Land and Poverty Conference Washington, D.C. – March 24, 2017