1. Cassava Production Measurement and Variety Identification in Household Surveys: Results from a Randomized Survey Experiment in Malawi
HEATHER MOYLAN
Survey Specialist
Living Standards Measurement Study
Development Data Group – Survey Unit – World Bank
Co-Authors: TALIP KILIC, JOHN ILUKOR and INNOCENT PANGAPANGA PHIRI
World Bank Land and Poverty Conference
Washington, D.C. – March 24, 2017

2. Background: LSMS Methodological Research
Broad scope of LSMS methodological research since 2005
Consumption, Labor, Income, Subjective Welfare, Asset Ownership
Focus under the LSMS-Integrated Surveys on Agriculture (LSMS-ISA): Improving agricultural production & productivity measurement
Methodological survey experiments under
Minding the (Agricultural) Data Gap Research Program, funded by UK Aid
Partnerships w/ (FAO) Global Strategy to Improve Agricultural and Rural Statistics; CGIAR Standing Panel on Impact Assessment; World Bank Innovations in Big Data Analytics; Stanford University; Livestock Data Innovation (LDI) in Africa Project

3. Background: LSMS Methodological Research (2)
Completed/on-going/planned randomized household survey experiments on:
Land Area, Soil Fertility, Rainfall, Crop Production, Agricultural Labor, Livestock Production, Cognitive, Non-cognitive & Technical skills
Approach:
Test (old & new) methods in tandem with a gold standard
Assess relative accuracy & scale-up feasibility
Cost effectiveness, skill & training requirements, respondent burden
Document results, best practices & protocols for scale-up
Integrate validated & cost-effective methods into LSMS operations
Today’s focus: Randomized household survey experiment in Malawi on measuring cassava production, productivity, and variety identification

4. The Root of the Measurement Problem
: Cassava is an important staple crop in many countries, particularly in sub-Saharan Africa, and is often planted for purposes of food security. However, cassava and other continuous crops that are often harvested in small quantities over extended periods of time pose particular challenges for the estimation of crop production and yield, due to the length and frequency of their harvesting. What are the difficulties associated with measuring production in smallholder production systems? First, farmers do not keep records. Second, recall is widely used in household survey operations but does not always work. Third, crop cutting is more expensive as well as time & resource-intensive. Fourth, use of non-standard measurement units of varying sizes is rampant. Fifth, there are different measurement units along the value chain, crops appearing in different conditions. Sixth, the development of conversion factors for expressing product-condition-non-standard unit combinations in KG-condition-equivalent terms in its infancy. Finally, improved approaches to farmer-reported production measurement need validation.

5. Summary of Main Findings
Measurement error in self-reported plot areas vis-à-vis their GPS-based counterparts is worrisome, in line with the recent literature
Collecting daily cassava harvest information via monthly self-administered diaries with bi-weekly phone calls generates arguably the most accurate annual household cassava production estimates
Collecting daily cassava harvest information via two 6-month recall surveys performs as well as self-administered monthly diaries with bi-weekly in-person visits (traditionally-considered as gold-standard)
Crop cutting provides an upper-bound household-level annual cassava yields, in line with expectations
Measurement error in self-reported cassava variety information vis-à-vis DNA fingerprinting is prohibitively high, in line with the emerging empirical evidence on alternative crops and countries

6. CVIP: Methodological Experiment on Measuring Cassava Production, Productivity, and Variety Identification
Funding
LSMS Minding the (Agricultural) Data Gap Research Program, funded by UKAid
Global Strategy to Improve Agricultural and Rural Statistics
Objectives
Test subjective approaches to measurement vis-à-vis objective methods for land area; cassava production and productivity; & cassava variety identification
Technical Partnerships
National Statistical Office (Implementing Agency)
CGIAR Standing Panel on Impact Assessment (Variety Identification)
CVIP: Methodological Experiment on Measuring Cassava Production, Productivity, and Variety Identification

7. CVIP: Context
Along lakeshore in North & Central Regions: Cassava is used as a staple food crop along the shores of Malawi (Nkhatabay, Nkhotakota)
Non-cassava belt areas in Southern & Central: Cassava is mainly grown for sale or use as a snack (Lilongwe, Zomba, Mulanje)
5 of the top cassava-producing districts were chosen within Malawi
Note that differences wrt Nkhatabay
Notes: Reference group for the test of mean differences is Nkhatabay. ***/**/* denote statistical significance at the 1/5/10 percent level, respectively.

8. CVIP: Sampling District Selection Consultations
Ministry of Agriculture & Food Security (MoAFS) – Department of Agricultural Research Services
Lilongwe University of Agriculture & Natural Resources
International Institute of Tropical Agriculture (IITA)
5 top cassava-producing districts identified in the Central and Southern regions of Malawi
1 district from each agro-ecological zone (Agricultural Development Division)
Extension Planning Area (EPA) Selection
District agricultural development officers, crop specialists in each of the 5 districts
Cassava-producing EPAs identified based MoAFS production figures
Enumeration Area (EAs) Selection
Identified universe of census EAs in each cassava EPA of interest
9 EAs selected with PPS in each district, based on 2008 census EA-level household counts

9. Households per treatment
CVIP: Sampling (2) 5
Districts 45
Enumeration Areas
1218
Households
~ 305
Households per treatment
Household Selection
Listing exercise conducted in each sampled EA to identify cassava-cultivating households
28 cassava households randomly selected in each EA; 7 households randomly assigned to each of the 4 treatment arms
Randomization successful
Note - attrition

10. CVIP: Methods Methods Tested: Cassava Production
Crop-cutting (5mx5m subplot) with balance scales
One plot/household for all households
Crop diaries with enumerator visits twice/week (D1)
Crop diaries with telephone calls twice/ week (D2)
Farmer-reported harvest (two visits, 6-month recall) (R1)
Farmer-reported harvest (single visit, 12-month recall) (R2)
Land area
GPS measurement
Farmer-reported area
Variety
DNA fingerprinting of leaf samples obtained at post-planting
Farmer-reported variety names, types & attributes w/ photo aid
Idea here is to assume that the extrapolate mark as an upper bound or potential production over the 12 month period and use this as a reference point for

11. A B C D CVIP: Crop Cutting Crop-Cutting Strategy 1 Sub-Plot per Plot:
Survey Solutions CAPI Application selects 1 plot at random for crop cutting
Cassava weighed fresh A B
| m | C D
| m |

12. CVIP: Crop Cutting - Laying the 5x5 Sub-PlotD

13. CVIP: Crop Cutting - HarvestingB C D
Timing based on when RESPONDENT is ready for harvest, Enumerator uses crop cut assistant to help in communication with household, helps to harvest and then weighs the cassava with the respondent, uses balance scale like the one distributed to all diary households

14. CVIP: Cassava Production DiariesB C D
Taught them how to use the scales, selected a literate household member as the keeper of the diary to assist plot managers in filling

15. CVIP: Recall InterviewsB C D

16. Fieldwork
Implementing agency: Malawi National Statistical Office (NSO)
5 mobile teams, 1 per district (1 Supervisor, 3 enumerators)
Computer-assisted personal interviewing (CAPI) application designed in Survey Solutions
Questionnaire
Survey Solutions CAPI Mode
Special CAPI Features
Household Questionnaire
Census
Random selection of crop cutting plot
GPS coordinates taken on tablet
Crop Cutting Questionnaire
Sample
Scanned barcodes for leaf samples
Monthly Diaries
(Paper) & Sample
Enumerators entered data after paper diaries collected each month
6-Month Recall
First visit data on gardens & plots fed forward
12-Month Recall

17. Fieldwork (2) Timeline: Initial Visits Laying Crop Cuts
Leaf Sample Collection
July 2015 – September 2016
Diary Visits
Diary Phone Calls
Starting July 2015
Crop cutting
6-month recall
February 2016
12-Month recall
July–August 2016

18. Land Area Measurement

19. Self-Reported vs. GPS-Based Cassava Plot Area

20. Production & Yield Measurement

21. Overview
Production measured as total kilograms harvested per household across the 12-month period
D1 & D2 Treatment Arms: Diary households all received scales; reported daily (fresh) harvest as measured in non-standard harvest units as well as in KGs
R1 & R2 Treatment Arms: Total production solicited in recall interviews – two visits to R1, 6-months apart, with a 6-month reference period at each visit; single visit to R2 with a 12-month reference period
Production typically captured in non-standard harvest units
Results robust to various formulations conversion factors created from the diary sample

22. Estimation of Survey Treatment Effects
1 𝑃𝑟𝑜𝑑𝑢𝑐𝑡𝑖𝑜𝑛 𝑖 =𝛼+ 𝛽 1 𝐷2 𝑖 + 𝛽 2 𝑅2 𝑖 + 𝛽 3 𝑅1 𝑖 + 𝛾𝐶+ 𝜀 𝑖
2 𝑌𝑖𝑒𝑙𝑑 𝑖 =𝛼+ 𝛽 1 𝐷1 𝑖 +𝛽 2 𝐷2 𝑖 + 𝛽 3 𝑅2 𝑖 + 𝛽 4 𝑅1 𝑖 + 𝛾𝐶+ 𝜀 𝑖
i represents household, ∝& 𝜀 are constant & error terms, respectively
Equation 1 – Outcome: Annual Cassava Production (KGs) – Comparison Category: D1
Equation 2 – Outcome: Annual Cassava Yield (KGs/Ha) – Comparison Category: CC
Crop cutting-based yield extrapolated from sub-plot to cassava farm (measured with GPS)
D2, R1 & R2 represent identifiers for diary-phone, 6-month recall & 12-month recall, respectively
C is a vector of household attributes – inclusion of which has no bearing on our findings
Standard errors clustered at the EA-level for Equation 1, at the household-level for Equation 2
C includes household size, dependency ratio, highest grade in household, wealth index, land area cultivated with cassava, cassava sales, gender and age of household head, household non-farm enterprise, household paid employment. The fact that the control variables are included/excluded does not have a bearing on the estimated survey treatment effects – in support of successful randomization.

23. Core Results
You need to note at some point, our failed attempts at unpacking the heterogeneity of treatment effects, explored through interactions of the survey treatment identifiers with household attributes.
. And please make sure to note that this heterogeneity analysis is only descriptive, given the endogenous control variables that you are interacting with your survey treatment dummies.

24. Monthly & Semi-Annual Production Estimates by Treatment Arm

25. Variety Identification

26. Why Variety Identification Matters?
Justifying investment in crop technology investments
Are farmers adopting modern agricultural technologies?
Are these technologies performing as expected?
Are technologies being used correctly?
Assessing the performance of the extension system
Do farmers know what they are planting?
Are farmers properly using the agricultural technologies?
Assessing the performance of the seed system
Are the seeds /cultivars of required standard or quality?
Are vitro plants in the gene bank true to their type?

27. Methods Subjective Methods
Farmer-Reported variety names, local vs. improved variety classification
Farmer-Reported Morphological Attributes based on a photo aid of 13 attributes
Objective Gold Standard
DNA Fingerprinting of Leaf Samples from crop cut sub-plots

28. How Do Subjective Methods Perform vis-à-vis DNA Fingerprinting?
All the farmers could uniquely state variety planted but only 30% were correctly identified
Farmer-reported morph. attributes perform poorly in unique & correct variety identification

29. Considerable Measurement Error in Variety Identification Based on Farmer-Reporting
Popular varieties (Beatrice & Manyokola): More likely to be correctly identified by farmers
Less popular varieties: High rate of mis-identification
Only 0.18% of the field samples were actually improved, while the farmer-reported incidence of improved variety cultivation was 21%
Likelihood of correct variety identification
Negatively correlated with varietal mixing
Positively correlated with access to extension
NO correlation with commercial seed acquisition
DNA Fingerprinting
Farmer-Reporting

30. Key Take-Away Messages
LAND AREA
Measurement error in self-reported plot areas vis-à-vis their GPS-based counterparts is worrisome, in line with the recent literature
At the mean, farmer-reported plot area is twice as much as its GPS-based counterpart
PRODUCTION
Diary-Phone generates arguably the most accurate annual cassava production estimates
6-Month Recall performs as well as Diary-Visit; traditionally-considered as gold-standard
12-Month Recall underestimates cassava production by a significant margin
Crop cutting provides an upper-bound for annual cassava yields, in line with expectations
Results in line with an earlier, near-identical methodological experiment in Zanzibar
6-Month Recall: A viable alternative to existing methods for cassava production measurement

31. Key Take-Away Messages (2)
VARIETY IDENTIFICATION
Error in self-reported cassava variety information vis-à-vis DNA fingerprinting is prohibitively high, in line with the emerging empirical evidence on alternative crops & countries
Call for increased consideration for integrating DNA fingerprinting into survey operations
Dramatically improves the extent of correct varietal identification
Reveals weaknesses in seed and extension systems
Cost per 1 sample per household: USD 25 (2016 Prices)
Need assess the effect of variety mis-identification on adoption & impact analyses

32. Next Steps
Better understand mechanisms for treatment effects: estimations of differences by district/production systems, (primary) cassava variety, socioeconomic attributes
Better document relative costs of each approach to production measurement
Analysis of the data from the Malawi (mini) consumption experiment
Feed the research into the FAO Global Strategy to Improve Agricultural and Rural Statistics Handbook on Measuring Crop Area, Yield, and Production

33. Cassava Production Measurement and Variety Identification in Household Surveys: Results from a Randomized Survey Experiment in Malawi
HEATHER MOYLAN
Survey Specialist
Living Standards Measurement Study
Development Data Group – Survey Unit – World Bank
Co-Authors: TALIP KILIC, JOHN ILUKOR and INNOCENT PANGAPANGA PHIRI
World Bank Land and Poverty Conference
Washington, D.C. – March 24, 2017