9. Practical use case 3: Pesticides Use Project Blanaru Cristina Eurostat Unit B5: “Data and metadata services nsad standards” SDMX Basics course, March 2016

PROCESS PHASES - PESTICIDES PROJECT   PHASE 2 PHASE 3 PHASE 4 Each SDMX project has 4 phases.

Phase 1: Preparation The legal acts: Regulation (EC) No 1185/2009 of the European Parliament and of the Council of 25 November 2009 concerning statistics on pesticides Commission Implementing Regulation (EU) No 1264/2014 of 26 November 2014

Phase 1: Preparation The parts involved: ESTAT+ NSIs; Data providers: NSIs; The collection and transmission of data for every 5 year period; One single file per country, covering the whole 5 years period;

Phase 1: Preparation The first deadline for transmission: December 2015 The format of the file: xml, send via eDAMIS; The dataset name: AEI_PESTUSE_5; The use of the MDT(internal database) like production system;

Phase 2: Compliance June 2015:Settle on the matrix and the associated code lists; Re-use of existing code lists and cross-domain code list: CL_FREQ, CL_OBS_STATUS, CL_UNIT_MULT, CL_DECIMALS; Create specific code lists: CL_PESTICIDES, CL_CROPS, CL_MEASURE_PES;

Phase 2: Compliance c) common standards for the collection, processing and dissemination (not always possible) The purpose of statistical standardisation: data about the same concepts are collected and communicated in the same way facilitates data comparisons to be made across domains and over time.

Explanation of Pesticides Matrix - Overview Sheet ● summarises all concepts and code lists used Concept ID Concept Name Description Role Level Usage Status Code list / format FREQ Frequency Frequency of the series (e.g. A, Q, M). Frequency dimension   SDMX+CL_FREQ+2.0 REF_AREA Reference area Reporting Country in ISO code (The country, or geographical/political group of countries that the measured economic phenomenon relates to) Dimension ESTAT+SCL_GEO_EUEFTACC+1.0 TIME_PERIOD Time Period The time period of the data. ObservationalTimePeriod PESTICIDES Pesticides Harmonised classification of substances ESTAT+CL_PESTICIDES+1.1 CROPS Crops ESTAT+CL_CROPS+1.1 OBS_VALUE Value The value of the data Primary measure FLOAT OBS_STATUS Status flag Status of the observation, such as normal, estimated or provisional Attribute Observation M SDMX+CL_OBS_STATUS+2.0 CONF_STATUS Confidentiality flag Confidentiality status of the observation C SDMX+CL_CONF_STATUS+1.1 TITLE Title Series STRING UNIT Unit ESTAT+SCL_UNIT+1.1 UNIT_MULT Unit multiplier Dataset ESTAT+CL_UNIT_MULT+1.0 DECIMALS Decimals ESTAT+CL_DECIMALS+1.0 MEASURE Measure Cross sectional measure Measure Dimension ESTAT+CL_MEASURE_PES+1.0 QUANTITY Quantity Cross-sectional measure AREA_TREATED Area treated

Explanation of Pesticides Matrix – Matrix Sheet Concepts FREQ REF_AREA PESTICIDES CROPS OBS_STATUS CONF_STATUS UNIT UNIT_MULT DECIMALS MEASURE QUANTITY AREA TREATED Legend: # … Code list fully used in concept %… Code list partly used in concept (see detailed sheet) (blank) … Concept not used in table (code) … fixed single code from code list Datasets   Dataset description AEI_PESTUSE_5 % # STATISTICS ON AGRICULTURAL USE OF PESTICIDES

Explanation of Pesticides Matrix – Matrix Sheet Shows the relationship between the dataset(s) and the concepts Each concept has a hyperlink pointing to the corresponding code list sheet The cells link a dataset (row) to a concept (column)

Explanation of Pesticides Matrix – Matrix Sheet The cells contain: A # sign if the code list is fully used in the dataset; A % sign if the code list is partiallly used in the dataset

Explanation of Pesticides Matrix – Code list Sheets Showing the contents of each of the code lists used: CL_FREQ sheet CL_PESTICIDES sheet CL_CROPS sheet CL_OBS_STATUS sheet………

Phase 2: Compliance drawn up the guidelines (B5 provides the draft to the business unit: E1) Reviewed the guidelines by the business unit; Develop a cross-sectional DSD: 2 values reported: for QUANTITY and AREA_TREATED Provide the matrix to the IT production team (for their preparatory work)

Phase 3: Implementation Finalize the guidelines (unit B5); Prepare test files for the package and test the DSD (unit B5); SDMX DSD (USE_PESTICI) is uploaded into the Euro Registry - https://webgate.ec.europa.eu/sdmxregistry/ Send the SDMX package to the pilot countries (Portugal and UK)

The Conversion demo INPUT FILE FORMAT (CSV) Data must be structured according to DSD The header of the CSV file should look like this FREQ;REF_AREA;PESTICIDES;CROPS;MEASURE;TIME_PERIOD;OBS_VALUE;OBS_STATUS;CONF_STATUS;UNIT;UNIT_MULT;DECIMALS A; DE; F01_01; L1000; QTY;2 010;1000.0;A;D;KG;0;0 A;DE;F01_01;L1000;ATRT;2010;2000.0;P;C;HA;0;0 A;DE;F01_01_04;L3000;QTY;2011;1001.0;E;N;KG;0;0 A;DE;F01_01_04;L3000;ATRT;2011;2001.0;F;R;HA;0;0

The Conversion demo OUTPUT FILE FORMAT (xml) Generate SDMX-ML files directly from the production system Generate SDMX-ML files from files in other formats (e.g. CSV), using the SDMX Converter application

The conversion

Conversion – steps: Select Input file :(1) Select Input format: CSV (2) Select Output file (3) Select Output format: CROSS_SDMX (4) Select DSD: downloaded from the Euro Registry "USE_PESTICI+ESTAT+1.1.xml" (5) Select Header file (6) or create the header (7)

Conversion – steps: Select Default Mapping (8) Select CSV delimiter: normally ";" (9) Select CSV Header Row value: USE_COLUMN_HEADER (10) Convert (11)

Phase 3: Implementation Received the test files from Portugal and UK via eDAMIS; Validation of the test files by MDT Team;

Issues on the testing phase: 1 Issues on the testing phase: 1. the header unmarked in the Converter application

Issues on the testing phase: 1. The solution

Issues on the testing phase: 2. aggregation problem

Issues on the testing phase: 3. the wrong extension of the file sent (unk instead of xml);

Phase 4: Production SDMX compliant data exchange used in production; Send the package to all countries;

Thank you for your attention. I am happy to take questions. For further information please contact: ESTAT-SUPPORT-SDMX@ec.europa.eu