1. The relevance of ‘new metrics’ for the evaluation of SDG2 – data revolution and innovative approaches for assessing human wellbeing Carlo Cafiero and Pietro Gennari FAO Statistics Division Rome, 17 November 2015

2. Guiding questions What is the Theory of Change behind SDG2? The SDG2 attempts to cover multiple dimensions of food availability, people’s access to food, nutrition and agricultural sustainability, do we already have indicators/indices/data to cover all dimensions? How complete and reliable is the global ‘data architecture’ in place to measure key indicators/indices under SDG2 and what are the missing elements? What are the existing data systems to measure relevant indicators under SDG2? What are the gaps? Do we need ‘new metrics’ to measure all dimensions of SDG2?

3. Theory of change If smallholder food producers have secure access to adequate knowledge, resources, and market outlets for their products and all consumers have the means to access adequate food Then we shall eradicate all forms of malnutrition, while improving living conditions in the rural areas, maintaining the genetic diversity of plants and animals, without contributing to environmental degradation.

4. Theory of change The process from Rio and Rio+20 (including the zero hunger challenge) to the 2030 Agenda (with lessons learned from the MDG process) Two possible perspectives The planet serving the people, or The people preserving the planet? A significant change in the “ownership” of the Agenda Country governments in the driving seat A more ambitious, comprehensive agenda May be too comprehensive? The danger of obfuscating priorities and diluting/misplacing resources

5. Increased small-holder Productivity, income and resilience (Target 2.3) Increased food availability & quality (a missing target? ) Better access to food (Target 2.1) Better nutrition (Target 2.2) Sustainable food production systems (Target 2.4) Genetic diversity (Target 2.5) Target 1.4 – Access to land, finance, Target 3.8 – universal health coverage Target 7.1 – Access to energy services Correcting trade restrictions (Target 2.b) Investing in technology, research, infrastructure (Target 2.a) Transparency of food markets (Target 2.c) Goal 15 – Ecosystems sustainability (Targets 15.1, 15.2, 15.3, 15.4 and 15.5) Target 6.1 – access to safe water Target 6.2 – access to sanitation Target 12.2 – Sustainable management of natural resources Target 12.2 – Strengthen resilience to climate change Target 1.1 – extreme poverty Target 1.3 - social protection Target 1.5 – resilience of the poor Target 12.3 – reduce food losses and waste Target 3.2 – end child mortality Target 3.4 – non-communicable diseases The core ToC: from smallholder productivity to better nutritionThe core ToC: from smallholder productivity to better nutrition… constrained by environmental sustainabilityIf we add the means of implementationsIf we add the means of implementations, we have SDG 2But SDG 2 is not in isolation …

6. Do we already have indicators to cover it all? No. But then, given its complexity, can we hope to cover it all? Should we? A first step has been made towards a framework for global monitoring by the IAEG-SDG Still to go on additional indicators on thematic indicators The process of indicators selection has been so far strongly conditioned by the overwhelming scope of the agenda (too many targets) and the limited time available to reach a consensus 169 Targets, each of them being multidimensional, impose a burden that may compromise the sustainability of the monitoring framework

7. The current list of indicators for SDG 2 Target 2.1: 2.1.1.Prevalence of Undernourishment (PoU) 2.1.2.Prevalence of population with moderate or severe food insecurity (based on the Food Insecurity Experience Scale, FIES) Target 2.2: 2.2.1.Prevalence of stunting and wasting among children less than five years of age Target 2.3: 2.3.1.Volume of production per labour unit (measured in constant USD), by classes of farming/pastoral/forestry enterprise size Target 2.4: 2.4.1.Percentage of agricultural area under sustainable agricultural practices

8. The current list of indicators for SDG 2 Target 2.5: 2.5.1.Ex situ Crop Collections Enrichment Index n.a.Number/percentage of local breeds classified as being at-risk, not-at-risk, and unknown-levels of risk of extinction Target 2.a: 2.a.1.Agriculture Orientation Index for Government Expenditures Target 2.b: 2.b.1.Percent change in Import and Export tariffs on agricultural products 2.b.2.Agricultural Export Subsidies Target 2.c: 2.c.1.Indicator of (food) Price Anomalies (IPA)

9. Food availability Rather than a call to increase food availability tout-court, the new agenda points to increased productivity and access to markets by smallholder producers, and to the quality of the food supply. In other words, the concern is not on total food availability, but on how the food supply is composed, both in terms of where it originates (locally, by smallholder producers, etc.) and how it is composed (healthy vs. junk food). Improving food availability data requires more detailed and timely food balances, and better frameworks for their analysis. Major gaps still exist, in this respect, on the ability to precisely capture food storage, food losses and waste and the quality of the food supply.

10. Food access Indicators of food access have been linked to income poverty and food expenditure Use of actual individuals’ food consumption data is been made difficult by the large measurement errors that affect them Individual dietary assessment surveys of the necessary quality are likely to be too costly for timely, continued monitoring Measurement errors are particularly relevant when individual food consumption is indirectly inferred from households’ food acquisition (i.e., from HIES), or when simplified food consumption modules are used (e.g., household food frequency consumption, dietary diversity questionnaires) To measure individual or household food access, use of experience-based food security scales shows very high promise In terms of reliability and cost effectiveness (they can be applied in virtually any existing survey, either at the individual or household level, at very little added cost) In terms of the capacity to truly distinguish between poverty and food insecurity

11. Regression analysis of food security and poverty indicators on child mortality rates Response variable: Logarithm of Child Mortality Rate Model 1Model 2Model 3Model 4 Regressors: Standardized regression coefficient (P-value H o : coefficient = 0) Log-odds(PoU) 0.420 (< 0.001) 0.509 (< 0.001) 0.260 (< 0.001) 0.284 (< 0.001) Log-odds(FI mod+ ) 0.499 (< 0.001) - 0.312 (< 0.001) - Log-odds(FI sev )- 0.409 (< 0.001) - 0.264 ( < 0.001) Log-odds (Extreme poverty (3) )-- 0.351 (<0.001) 0.373 (< 0.001) Adjusted R-squared0.7410.7160.7690.759 N135 103

12. Nutrition While data on nutritional status, based on anthropometry, exist, they need to be triangulated with information on the quality of the diets people eat, while controlling for health and sanitation conditions in order to make the link between people livelihood, food and malnutrition. Ecological association is not sufficient, as differences within a group (by age, sex, position within the household or the community) may be large As noted, we usually lack reliable individual food consumption data to be matched to available data on the nutritional status. Individual dietary intake surveys are costly. Shortcuts as the Individual Dietary Diversity Score are promising to address some of the issues (i.e., risk of micronutrient deficiency). Other shortcuts, such as repurposing household food acquisition data or simplified household Food Consumption Scores for nutrition assessments are problematic, as they provide only indirect and imprecise information on actual regular food consumption by individuals within the households.

13. Agricultural sustainability To measure the environmental, economic and social sustainability of existing agricultural practices is still highly problematic It would require sufficiently detailed data on actual farm operations, and a clear understanding of the economics of small farm operations and the environmental impact of existing practices Too sparse availability of farm surveys is still a problem (and not only in the developed world. Use of administrative data can help in cases where farm support has been linked to the existence of specific sustainable practices, and where mechanisms for certification of compliance with such practice have been put in place (mostly in the developed world), but these are still far from being sufficient to provide a comprehensive view of sustainability. Promoting the conduction of improved farm surveys (e.g., the AGRIS - Agricultural and Rural Integrated Surveys program promoted by FAO, IFAD and the WB) seems the only promising way.

14. What are the gaps? One overall limiting factor is still the level of agricultural statistical capacity in several developing countries. The Global Strategy to Improve Agricultural and Rural Statistics is on the right track but there is still need to invest on statistical literacy, to promote the inclusion of the agricultural domain within the official statistical system, and to reinforce regular farm system data collection. To measure which component of the food supply originates with smallholders, or which practices are common that have environmental quality implications, requires periodic, detailed information from farm surveys, still lacking. In terms of analytic capacity, the lack of consensus on a suitable definition of “small holder” makes the assessment and monitoring of small holder productivity and incomes particularly problematic. Even if we solve the “data” problem, there is still the problem of the analytical capacity to properly validate, interpret and analyze the data

15. Do we need ‘new metrics’ to measure all dimensions of SDG2? “General principles […] determine how the information provided by the senses [i.e., the “data”] is to be treated. It is actually treated in two different ways, which may be called description and inference. Description, in the strict sense, would involve only the cataloguing and classification of sensations already experienced [data]. Inference is the use of sensations already experienced [data] to derive information about sensations not yet experienced, to construct physical [and conceptual] objects, and to describe the past and the future of these […] objects. For pure description only an application of the principles of classification and the properties of classes is required […]. Inference requires much more.” (Harold Jeffreys. Scientific Inference. 1931)

16. The “much more” which is needed (and too often missing) Monitoring and evaluating is “much more” than describing. It calls for application of statistical inference principles to data, in order to verify testable hypotheses of causal links. Too often we speak loosely of “measuring” when what we actually do is simply to describe data which may be highly imprecise or essentially irrelevant Measurement requires: Identification of the measurand, i.e., the attribute being measured (may be an unobservable conceptual object, e.g. “food insecurity”) The selection of data (unambiguously described, e.g. “food consumed” vs. “the amount of food that a person has declared having consumed over a certain period of time”) The definition of a model that links the data to the measurand in a precise causal way: a measurement system is valid when it produce measures that reflect all and only the changes in the magnitude of the attribute being measured (ex.: anthropometry as a “measure” of “food security”)

17. The “much more” which is needed (and too often missing) cont. Measurement requires Due to the uncertainty that inevitably surrounds the data (i.e., sampling and measurement error), the model must be framed in probabilistic terms, i.e., describing the probability of observing a certain range of values for the data, conditional on the magnitude of the attribute being measures The likelihood principle can be used to infer the magnitude of the attribute and to assess the reliability of the data Example: measures of access to food that define “food security” How to infer on food consumption habits from data on episodic food consumption? (The problem of treating intra individual variability of food consumption) Error induced “spurious” correlation may seriously invalidate attempts at seeking concurrent validation of proposed indicators when no gold standard exists

18. Thanks! Carlo.cafiero@fao.org