Eurostat 1 7a. Practical use case 1: Pesticides Use Project Blanaru Cristina Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, October 2015

Eurostat PROCESS PHASES - PESTICIDES PROJECT 2 PHASE 1 PHASE 2PHASE 3PHASE 4

Eurostat Phase 1: Preparation The legal acts: Regulation (EC) No 1185/2009 of the European Parliament and of the Council of 25 November 2009 concerning statistics on pesticides Commission Implementing Regulation (EU) No 1264/2014 of 26 November 2014

Eurostat The parts involved: ESTAT+ NSIs; Data providers: NSIs; The collection and transmission of data for every 5 year period; One single file per country, covering the whole 5 years period; 4 Phase 1: Preparation

Eurostat The first deadline for transmission: December 2015 The format of the file: xml, send via eDAMIS; The dataset name: AEI_PESTUSE_5; The use of the MDT(internal database) like production system; 5 Phase 1: Preparation

Eurostat June 2015:Settle on the matrix and the associated code lists; a)Re-use of existing code lists and cross-domain code list: CL_FREQ, CL_OBS_STATUS, CL_UNIT_MULT, CL_DECIMALS; b) Create specific code lists: CL_PESTICIDES, CL_CROPS, CL_MEASURE_PES; 6 Phase 2: Compliance

Eurostat c) common standards for the collection, processing and dissemination (not always possible) The purpose of statistical standardisation: data about the same concepts are collected and communicated in the same way facilitates data comparisons to be made across domains and over time. 7 Phase 2: Compliance

Eurostat Explanation of Pesticides Matrix - Overview Sheet ● summarises all concepts and code lists used 8 Concept IDConcept NameDescriptionRoleLevelUsa ge Stat us Code list / format FREQFrequencyFrequency of the series (e.g. A, Q, M). Frequency dimension SDMX+CL_FREQ+2.0 REF_AREAReference area Reporting Country in ISO code (The country, or geographical/political group of countries that the measured economic phenomenon relates to) Dimension ESTAT+SCL_GEO_EUEFTACC+1.0 TIME_PERIODTime PeriodThe time period of the data. Dimension ObservationalTimePeriod PESTICIDESPesticidesHarmonised classification of substances Dimension ESTAT+CL_PESTICIDES+1.1 CROPSCrops Dimension ESTAT+CL_CROPS+1.1 OBS_VALUEValueThe value of the dataPrimary measure FLOAT OBS_STATUSStatus flag Status of the observation, such as normal, estimated or provisionalAttributeObservationMSDMX+CL_OBS_STATUS+2.0 CONF_STATUSConfidentiality flagConfidentiality status of the observationAttributeObservationCSDMX+CL_CONF_STATUS+1.1 TITLETitle AttributeSeriesCSTRING UNITUnit AttributeObservationMESTAT+SCL_UNIT+1.1 UNIT_MULTUnit multiplier AttributeDatasetCESTAT+CL_UNIT_MULT+1.0 DECIMALSDecimals AttributeDatasetCESTAT+CL_DECIMALS+1.0 MEASUREMeasureCross sectional measureMeasure Dimension ESTAT+CL_MEASURE_PES+1.0 QUANTITYQuantity Cross-sectional measure AREA_TREATEDArea treated Cross-sectional measure

Eurostat Explanation of Pesticides Matrix – Matrix Sheet 9 Concepts FREQ REF_AREA PESTICIDES CROPS OBS_STATUS CONF_STATUS UNIT UNIT_MULT DECIMALS MEASURE QUANTITY AREA TREATED Legend: # … Code list fully used in concept %… Code list partly used in concept (see detailed sheet) (blank) … Concept not used in table (code) … fixed single code from code list Datasets Dataset description AEI_PESTUSE_5%##%#%%### STATISTICS ON AGRICULTURAL USE OF PESTICIDES

Eurostat Explanation of Pesticides Matrix – Matrix Sheet Shows the relationship between the dataset(s) and the concepts Each concept has a hyperlink pointing to the corresponding code list sheet The cells link a dataset (row) to a concept (column) 10

Eurostat Explanation of Pesticides Matrix – Matrix Sheet The cells contain: A # sign if the code list is fully used in the dataset; A % sign if the code list is partiallly used in the dataset 11

Eurostat Explanation of Pesticides Matrix – Code list Sheets Showing the contents of each of the code lists used: CL_FREQ sheet CL_PESTICIDES sheet CL_CROPS sheet CL_OBS_STATUS sheet……… 12

Eurostat drawn up the guidelines (B5 provides the draft to the business unit: E1) Reviewed the guidelines by the business unit; Develop a cross-sectional DSD: 2 values reported: for QUANTITY and AREA_TREATED Provide the matrix to the IT production team (for their preparatory work) 13 Phase 2: Compliance

Eurostat Phase 3: Implementation Finalize the guidelines (unit B5); Prepare test files for the package and test the DSD (unit B5); SDMX DSD (USE_PESTICI) is uploaded into the Euro Registry Send the SDMX package to the pilot countries (Portugal and UK) 14

Eurostat The Conversion demo INPUT FILE FORMAT (CSV) Data must be structured according to DSD The header of the CSV file should look like this 15 FREQ;REF_AREA;PESTICIDES;CROPS;MEASURE;TIME_PERIOD;OBS_VALUE;OBS_STATUS; CONF_STATUS;UNIT;UNIT_MULT;DECIMALS A; DE; F01_01; L1000; QTY;2 010;1000.0;A;D;KG;0;0 A;DE;F01_01;L1000;ATRT;2010;2000.0;P;C;HA;0;0 A;DE;F01_01_04;L3000;QTY;2011;1001.0;E;N;KG;0;0 A;DE;F01_01_04;L3000;ATRT;2011;2001.0;F;R;HA;0;0

Eurostat The Conversion demo OUTPUT FILE FORMAT (xml) 1.Generate SDMX-ML files directly from the production system 2. Generate SDMX-ML files from files in other formats (e.g. CSV), using the SDMX Converter application 16

Eurostat The conversion 17

Eurostat Conversion – steps: Select Input file :(1) Select Input format: CSV (2) Select Output file (3) Select Output format: CROSS_SDMX (4) Select DSD: downloaded from the Euro Registry "USE_PESTICI+ESTAT+1.1.xml" (5) Select Header file (6) or create the header (7) 18

Eurostat Conversion – steps: Select Default Mapping (8) Select CSV delimiter: normally ";" (9) Select CSV Header Row value: USE_COLUMN_HEADER (10) Convert (11) 19

Eurostat Phase 3: Implementation Received the test files from Portugal and UK via eDAMIS; Validation of the test files by MDT Team; 20

Eurostat Issues on the testing phase: 1. the header unmarked in the Converter application 21

Eurostat Issues on the testing phase: 1. The solution 22

Eurostat Issues on the testing phase: 2. aggregation problem 23

Eurostat Issues on the testing phase: 3. the wrong extension of the file sent (unk instead of xml); 24

Eurostat Phase 4: Production SDMX compliant data exchange used in production; Send the package to all countries; 25

Eurostat 26 Thank you for your attention. I am happy to take questions. For further information please contact: