Blowing in the Wind: The Quest for Accurate Crop Variety Identification in Field Research, with an Application to Maize in Uganda TALIP KILIC Senior Economist & Survey Methods Team Leader Living Standards Measurement Study Development Data Group – Survey Unit – World Bank tkilic@worldbank.org Co-Authors: JOHN ILUKOR, JAMES STEVENSON, SYDNEY GOURLAY, FREDERIC KOSMOWSKI, ANDRZEJ KILIAN, JULIUS PYTON SSERUMAGA, AND GODFREY ASEA International Consortium on Applied Bioeconomy Research (ICABR) Conference Berkeley, CA – May 31, 2017

Motivation Accurate identification of crop varieties grown by farmers key to estimating levels of improved variety cultivation ensuing impacts on production, productivity, and a range of welfare and nutrition outcomes Empirical evidence central to justifying investments in crop R&D, support to seed systems Among farmers, correct information essential to their adoption & management decisions

Prevailing Approaches to Variety Identification Extent of underinvestment in methodological innovation for accurate variety identification is puzzling Literature on adoption & impacts have usually relied on expert estimates and/or farmer-reported survey data on Variety names Improved vs. traditional status of a cultivated variety Hybrid vs. OPV status of a cultivated variety Why worry? Weaknesses in extension & formal seed systems Reliance on informal channels of seed acquisition Variety naming systems that exhibit variation across time & space Limited empirical evidence on the accuracy of prevailing approaches to variety identification (& implications of measurement error in impact evaluation) Farmer-elicitation method may work if the cultivated seeds are purchased from a formal seed system that truthfully label purchased seed attributes or that does not suffer from widespread seed adulteration and/or counterfeiting. Given, however, the well-documented weaknesses in extension and seed systems, and the reliance on informal methods of seed acquisition, farmers may misidentify or not be able to identify the varieties they plant, and fail to provide the minimum set of correct varietal information sought in surveys. The variety naming systems that emerge in the absence of formal seed systems and that exhibit variation across time and space is another complicating mediating factor in obtaining reliable information from survey respondents.

Our Contribution Implemented a survey experiment in Eastern Uganda to test the relative accuracy subjective approaches to maize variety identification compared to DNA fingerprinting Compiled a reference library of improved varieties in Uganda that serves a key input into DNA fingerprinting as well as the assessment of commercial seed quality

MAPS: Methodological Experiment on Measuring Maize Productivity, Soil Fertility, and Variety Support LSMS Minding the (Agricultural) Data Gap Research Program, funded by UK Aid Global Strategy to Improve Agricultural and Rural Statistics, housed at FAO World Bank Innovations in Big Data and Analytics Program World Bank Trust Fund for Statistical Capacity Building Primary Objectives Test subjective approaches to measurement vis-à-vis objective methods for maize yield measurement, soil fertility assessment & maize variety identification Partnerships Uganda Bureau of Statistics (Implementing Agency), World Agroforestry Centre (Soil Fertility), CGIAR Standing Panel on Impact Assessment (Variety Identification), Stanford University & Terra Bella (Remote Sensing) Round I (First Agricultural Season of 2015) Post-Planting Fieldwork: April-June 2015 Crop Cutting Fieldwork: June-August 2015 Post-Harvest Fieldwork: September-November 2015 Round II (First Agricultural Season of 2016) Identical timeline & visit structure Follow-up to a subset of Round I households (540 out of 900)

MAPS Sample Round I Enumeration Area (EA) Selection 45 EAs from a 400 Km2 remote sensing tasking area (Iganga & Mayuge) 15 EAs in each of Serere & Sironko districts Household Selection Original Plan: 6 pure stand & 6 intercropping households selected at random in each EA following listing – 450 in each universe Result: 385 vs. 515 split – inadequate # of pure stand HHs in select EAs 249 vs. 291 split in Iganga & Mayuge Plot Selection Survey Solutions CAPI application to randomly select one plot per household Round II Follow-up to 540 households in Iganga & Mayuge Analysis sample: 440 households with crop cuts in Round I & II Attrition does not have a bearing on the analysis Note - attrition

MAPS Remote Sensing Tasking Area

MAPS Methods Methods Tested: Maize Production Crop-cutting 4m x 4m & a 2m x 2m subplot in Round I 8m x 8m sub-plot in Round II Full-plot crop cut in Round II (1/2 of sample) Remote sensing based on high-res imagery First in testing the method in a smallholder production system against an objective measure Self-reported harvest Conversion of quantities in non-standard unit-condition combos into KG-, dried grain terms (“official” methods) Land Area GPS measurement (Garmin eTrex 30 handheld units) Self-reported area Soil Fertility (Round I) Conventional Soil Analysis (subsample) Spectral Soil Analysis Self-reported soil quality & attributes Variety Identification DNA fingerprinting of grain sampled from the crop-cutting subplot harvest (4x4m in Round I, 8x8m in Round II) Self-reported variety name, type & morphological attributes Note on 4x4m vs. 2x2m (lack of) difference in Round 1, and (randomly selected) 4x4m (quadrant) vs. 8x8m (lack of) difference in Round 2

DNA Fingerprinting Diversity Arrays Technology (DArTseq) method that facilitates genome-wide characterizations of large accessions sets compared to existing genotyping-by-sequencing methods using SNP markers Compiled a reference library of 38 maize varieties in circulation during the pre-planting period of the first rainy season of 2015, from NARO & 4 major seed companies, with revealed genotyping intention Genotyped each reference library and field samples to derive two vars Heterogeneity: # of DNA marker variants in the genomic representation - a collection of fragments from the genome selected for sequencing Purity: Computed only for the field samples, represents the extent to which heterogeneity overlaps with that of the matched reference library variety - identified initially as the one with the closest genetic distance to the field sample in question (below a distance threshold of 3).

Recursive Partitioning & Classification Tree Analysis of Morphological Attributes of 38 Reference Library Samples Morphological attributes for the reference library: Obtained by planting out the 38 varieties in NaCCRI fields. Results: Varieties are uniquely identified using 11 attributes. Identification of the varieties in the field: Using these attributes, varieties that the farmers plant were identified based farmer responses on morphological attributes The feasibility and the accuracy of this approach are yet to be tested especially since morphological characteristics (1) are often multi-genic and are not available at all growth stages, and (2) are not stable and may reflect an adaptation to environmental conditions, particularly in the case of maize. The first step in RPCTA is to find an explanatory variable that best splits the data into two groups. This process is repeated in each sub-group, and recursively, until the subgroups either reach a minimum size or until no improvement can be made. The second step is to trim back the full classification tree based on cross-validation (Therneau et al., 2015).

Context 83.9 percent of the population live in rural areas. National rate of poverty = 19.5 percent. Agriculture value added corresponds to 25.8 percent of the GDP. Agricultural employment makes up 71.7 percent of total employmen. Maize is one of the major staple, commercial, and export crops in Uganda. It is the leading cereal crop grown in almost all parts of the country. In Eastern Uganda; the country’s leading maize producing region, the crop accounts for the highest share (25 percent) of crop income. At the same time, Eastern Uganda, following Northern Uganda, is also the region with the highest concentration of the country’s poor, and the latest estimate of the regional absolute poverty rate stands at 24.5 percent. While not shown, the sample descriptive statistics are comparable to those derived for the Eastern region, and Iganga and Mayuge sub-samples of the UNPS 2015/16. The average area was 0.14 hectares for our plots, 46 percent of which was monocropped with maize. The sampled plots were all within a 1 kilometer radius of the dwelling units. The non-labor input use was low (e.g. the incidence of any inorganic fertilizer use was only 9 percent), and the lion share of the plot-level labor input originated from within households. Forty-two percent of the sampled plots were managed by females, and the plot managers were, on average, 41 years old, with slightly above 6 years of education. Regarding specifically the maize grown within the crop cut sub-plots, the seed for 37 percent were acquired commercially in the planting period for the agricultural season of interest. Further, only 45 percent of the farmers could provide a name for the primary variety grown on the plot selected for crop cutting.

How Do Different Methods Perform in Unique Identification of Maize Varieties in Round I? 53 percent of farmers could not state the variety they have planted Farmer-reported morph. attributes does not uniquely identify varieties DNA fingerprinting performs the best for unique varietal identification

Only 2 Percent of the Farmers Correctly Identified the Variety Based on DNA Analysis in Round I With the exception of LONGE 10H, the varieties stated by the farmers (i.e. right panel) are NOT among the varieties identified by DNA fingerprinting (i.e. left panel). Either farmers do not know or the stated names are the ones they were told Source: Ilukor et al. (Forthcoming).

And Our Experts Were No Better!

How Do Different Methods Perform in Identifying Local/Improved & OPV/Hybrid Varieties in Round I? Cultivation of improved & hybrid varieties is under-estimated by farmers Cultivation of open pollinated varieties is over-estimated by farmers

Purity of the Field Samples in Round I Farmer-planted variety according to DNA fingerprinting is the reference library variety that is genetically closest (not necessarily identical) Purity = Overlap between the field sample genetic heterogeneity & the genetic heterogeneity of the identified variety in the reference library Mean 63.2% Median 62.1% Min 46.9% Max 98.4%

Headline Findings from Multivariate Analyses of Variety Identification Outcomes Purity Negatively correlated with farmer’s correct identification of recyclability of the seed NO relationship with commercial acquisition of the seed! Farmer’s correct identification of an improved variety Positively correlated with farmer’s knowledge of the variety & commercial acquisition of the seed

(Unacceptable Levels of) Heterogeneity in Reference Library Samples in Round I Acceptable level of heterogeneity of the samples is 20% but most of the reference library samples are above the threshold. Mean 32.9% Median 24.6% Min 9.8% Max 75.2%

Key Take-Away Messages Variety identification findings reveal: High-levels of improved variety cultivation, despite popular belief But… cultivated varieties are of inferior quality Limited farmer knowledge about the varieties that they plant Weaknesses in & potential implications for extension & seed system Evidence prompts us to think more critically about existing agricultural statistics & survey methods Support for DNA fingerprinting to be the new standard for accurate variety identification in field research Further experimentation & synthesis of evidence from completed survey experiments on other countries & crops – key to formulating guidelines for scale-up Additional costs require more thinking around sub-sampling approaches in existing household & farm surveys

Blowing in the Wind: The Quest for Accurate Crop Variety Identification in Field Research, with an Application to Maize in Uganda TALIP KILIC Senior Economist & Survey Methods Team Leader Living Standards Measurement Study Development Data Group – Survey Unit – World Bank tkilic@worldbank.org Co-Authors: JOHN ILUKOR, JAMES STEVENSON, SYDNEY GOURLAY, FREDERIC KOSMOWSKI, ANDRZEJ KILIAN, JULIUS PYTON SSERUMAGA, AND GODFREY ASEA International Consortium on Applied Bioeconomy Research (ICABR) Conference Berkeley, CA – May 31, 2017