Data Transmission Tools & Services EDAMIS, SDMX, Validation Daniel Suranyi Hartmut Schrör Eurostat B5 Eurostat E3 Statistical Information Systems Waste Statistics

Data Transmission Tools & Services EDAMIS Single Entry Point for Data SDMX Information Model for Statistical Data and Metadata Exchange Validation Graphical: EDAMIS Web Forms module File based: EDAMIS Validation Engine Before / during data exchange using SDMX concepts EDAMIS, SDMX, Validation Waste Workshop 15/16 November 2011

EDAMIS - Overview Implements the concept of the Single Entry Point (SEP) Facilitates fully automated data transmissions From Member States to Organisations Between Organisations Mainly Eurostat, but also other international organisations Offers value added services such as Traffic monitoring & reporting Acknowledgements Automatic reminders Data Validation using SDMX EDAMIS, SDMX, Validation Waste Workshop 15/16 November 2011

Components of EDAMIS EDAMIS Web Portal (EWP) Browser based portal Transmission of data Management of the inventory (users, datasets) eDAMIS Validation Engine (EVE) for SDMX-ML EDAMIS Web Forms (EWF) Graphical user interface for SDMX (spreadsheet) For small data transmissions Real-time validation on the screen Embedded in EWP EDAMIS Web Application (EWA) “Mini” eDAMIS satellite server Installed in organisations with high volume Allows full automatic transmission EDAMIS, SDMX, Validation Waste Workshop 15/16 November 2011

EDAMIS Inventory Domain Manager Central Coordinator Administrator Expected to send? Sending Receiving Rights Domain Manager Central Coordinator Administrator Sender, Receiver Data Preparer (Web Forms) Informed Guest EDAMIS, SDMX, Validation Waste Workshop 15/16 November 2011

eDAMIS Dataset Management Periodicity of data, periodicity of transmission Transmission mode (send data, web forms, both) Link to… Country (expected to send) Organisations (sending and receiving) Users (rights for dataset) Enable confidentiality (only encrypted files possible) Send automatic reminders to data senders Link to SDMX DSD from Euro SDMX Registry for Validation EDAMIS, SDMX, Validation Waste Workshop 15/16 November 2011

eDAMIS in Waste Statistics Already in use for waste statistics Transmission of WStatR files Xml files exported from Excel data entry tool “Edawaste” Excel files themselves (not intended like this) Quality reports Transmission of quality reports on WEEE Packaging waste End-of-life vehicles (ELV) Continued use for submission of all reports (incl. WStatR quality reports) EDAMIS, SDMX, Validation Waste Workshop 15/16 November 2011

Dataset names in eDAMIS WASTE_GENER_A2 WASTE_TREATM_A2 WASTE_REGIO1_A2 (incineration and recovery) WASTE_REGIO2_A2 (landfills) EDAMIS, SDMX, Validation Waste Workshop 15/16 November 2011

Statistical Data and Metadata EXchange What is SDMX? Standard for statistical (meta)data exchange Information model Technical specifications Main file formats SDMX-ML (XML based) SDMX-EDI (a.k.a. GESMES-TS) The initiative – from 2001: 7 “sponsors”: UN, OECD, EC, ECB, BIS, IMF, WB ISO Standard 17369 – SDMX 2.0 SDMX 2.1 published April 2011 EDAMIS, SDMX, Validation Waste Workshop 15/16 November 2011 9

Statistical Data and Metadata EXchange Why SDMX? Harmonisation of (meta)data exchange process Reduction of development and maintenance costs Faster, more reliable, simpler data exchange Technical Aspects SDMX Artefacts Data Structure Definition (DSD) Code Lists, Concept Schemes, … SDMX Registry Metadata Repository Hosting and Sharing of SDMX Artefacts EDAMIS, SDMX, Validation Waste Workshop 15/16 November 2011 10

What is a DSD? Describe statistics in a standard way DIMENSIONS TIME COUNTRY FREQUENCY TOURISM TOPIC OBS_VALUE OBS_STATUS E P DIMENSIONS ATTRIBUTES MEASURES Describe statistics in a standard way EDAMIS, SDMX, Validation Waste Workshop 15/16 November 2011

How does a DSD help? Statisticians Dimensions of a table / data cube Attributes of observations (flags, footnotes, …) Code lists used Information Technology Allows IT tools to validate Structure of data Codes used Facilitates automated information exchange Across different statistical domains Standardised tools Web services EDAMIS, SDMX, Validation Waste Workshop 15/16 November 2011

SDMX Implementation in Member States Tools offered Action Web Forms No costs for organisation Manual work for senders (type, copy/paste) SDMX Converter (sender’s PC) Installation on sender’s PC Manual work for senders (convert) (batch mode) Installation on server SDMX Reference Infrastructure Mapping of database to DSDs Own SDMX Implementation Local development in organisations EDAMIS, SDMX, Validation Waste Workshop 15/16 November 2011

How do EDAMIS and SDMX come together? Validation at the Entry Point Based on SDMX standard EDAMIS Web Forms (EWF) Graphical SDMX editor with on-screen data validation Spreadsheet interface, no SDMX knowledge needed Collection of small volumes EDAMIS Validation Engine (EVE) Same experience for batch validation EDAMIS, SDMX, Validation Waste Workshop 15/16 November 2011

EDAMIS Web Forms DSD EDAMIS, SDMX, Validation Waste Workshop 15/16 November 2011

eDAMIS webforms in waste Already in use for WEEE Packaging Waste ELV Municipal waste Four more webforms for WStatR Waste generation Waste treatment Two tables on waste treatment facilities Recovery, incineration Landfills EDAMIS, SDMX, Validation Waste Workshop 15/16 November 2011

Advantage of webforms for Waste Data entry and data transmission in one tool Data entry in Excel replaced with webform Separate data transmission via EWP or EWA no longer necessary Only quality report to be sent separately Minimal maintenance effort; same tool for all data suppliers. EDAMIS, SDMX, Validation Waste Workshop 15/16 November 2011

Data Validation Before/During Transmission (EDAMIS – “First Level”) Performed in Member States Format Check (SDMX-ML) Code Check (SDMX DSD) Basic Value Check After Transmission (MDT – “Second Level”) Performed in Eurostat EDAMIS, SDMX, Validation Waste Workshop 15/16 November 2011

How does it all come together? Validation Services Member State Eurostat Prod Eurobase EDAMIS SDMX-RI MDT DSDs SDMX Registry EDAMIS, SDMX, Validation Waste Workshop 15/16 November 2011

Workflow eDAMIS Validation Engine SDMX Registry DSDs Web Service Browser SDMX Converter CSV Settings MS Database Eurostat Production Unit EDAMIS Server Validation SDMX-ML eWP eWA Report Report EDAMIS, SDMX, Validation Waste Workshop 15/16 November 2011

eDAMIS Validation Engine – Validation Report EDAMIS, SDMX, Validation Waste Workshop 15/16 November 2011

EVE – Data Validation Features Same Validation Rule Syntax as Web Forms Within one file and reference period Different rule sets per reference period possible Country specific rules Mandatory values, Range checks Basic expressions, comparison (+ - * / < > =) Mathematical expressions (SUM, AVG, MIN, MAX, …) Logical expressions (AND, OR, NOT) EDAMIS, SDMX, Validation Waste Workshop 15/16 November 2011

WStatR validations in EVE Valid combinations of “waste” and “hazard” No negative values Warning if cell is empty Valid combinations of “status flags” and “confidentiality flags” REGIO: ‘wrong’ NUTS totals only lead to warnings Inconsistencies of national / regional data may be acceptable EDAMIS, SDMX, Validation Waste Workshop 15/16 November 2011

Workshop - Demonstration of the Workflow Data sender using EDAMIS Web Portal CSV  SDMX-ML  Pre-Validation EDAMIS, SDMX, Validation Waste Workshop 15/16 November 2011

Questions? EDAMIS Support Team SDMX Support Team ESTAT-Support-EDAMIS@ec.europa.eu SDMX Support Team ESTAT-Support-SDMX@ec.europa.eu EDAMIS, SDMX, Validation Waste Workshop 15/16 November 2011