1. Data Transmission Tools & Services EDAMIS, SDMX, Validation
Daniel Suranyi Hartmut Schrör
Eurostat B5 Eurostat E3
Statistical Information Systems Waste Statistics

2. Data Transmission Tools & Services
EDAMIS
Single Entry Point for Data
SDMX
Information Model for Statistical Data and Metadata Exchange
Validation
Graphical: EDAMIS Web Forms module
File based: EDAMIS Validation Engine
Before / during data exchange using SDMX concepts
EDAMIS, SDMX, Validation
Waste Workshop 15/16 November 2011

3. EDAMIS - Overview
Implements the concept of the Single Entry Point (SEP)
Facilitates fully automated data transmissions
From Member States to Organisations
Between Organisations
Mainly Eurostat, but also other international organisations
Offers value added services such as
Traffic monitoring & reporting
Acknowledgements
Automatic reminders
Data Validation using SDMX
EDAMIS, SDMX, Validation
Waste Workshop 15/16 November 2011

4. Components of EDAMIS EDAMIS Web Portal (EWP)
Browser based portal
Transmission of data
Management of the inventory (users, datasets)
eDAMIS Validation Engine (EVE) for SDMX-ML
EDAMIS Web Forms (EWF)
Graphical user interface for SDMX (spreadsheet)
For small data transmissions
Real-time validation on the screen
Embedded in EWP
EDAMIS Web Application (EWA)
“Mini” eDAMIS satellite server
Installed in organisations with high volume
Allows full automatic transmission
EDAMIS, SDMX, Validation
Waste Workshop 15/16 November 2011

5. EDAMIS Inventory Domain Manager Central Coordinator Administrator
Expected to send?
Sending Receiving
Rights
Domain Manager
Central Coordinator
Administrator
Sender, Receiver
Data Preparer (Web Forms)
Informed
Guest
EDAMIS, SDMX, Validation
Waste Workshop 15/16 November 2011

6. eDAMIS Dataset Management
Periodicity of data, periodicity of transmission
Transmission mode (send data, web forms, both)
Link to…
Country (expected to send)
Organisations (sending and receiving)
Users (rights for dataset)
Enable confidentiality (only encrypted files possible)
Send automatic reminders to data senders
Link to SDMX DSD from Euro SDMX Registry for Validation
EDAMIS, SDMX, Validation
Waste Workshop 15/16 November 2011

7. eDAMIS in Waste Statistics
Already in use for waste statistics
Transmission of WStatR files
Xml files exported from Excel data entry tool “Edawaste”
Excel files themselves (not intended like this)
Quality reports
Transmission of quality reports on
WEEE
Packaging waste
End-of-life vehicles (ELV)
Continued use for submission of all reports (incl. WStatR quality reports)
EDAMIS, SDMX, Validation
Waste Workshop 15/16 November 2011

8. Dataset names in eDAMIS
WASTE_GENER_A2
WASTE_TREATM_A2
WASTE_REGIO1_A2 (incineration and recovery)
WASTE_REGIO2_A2 (landfills)
EDAMIS, SDMX, Validation
Waste Workshop 15/16 November 2011

9. Statistical Data and Metadata EXchange
What is SDMX?
Standard for statistical (meta)data exchange
Information model
Technical specifications
Main file formats
SDMX-ML (XML based)
SDMX-EDI (a.k.a. GESMES-TS)
The initiative – from 2001:
7 “sponsors”: UN, OECD, EC, ECB, BIS, IMF, WB
ISO Standard – SDMX 2.0
SDMX 2.1 published April 2011
EDAMIS, SDMX, Validation
Waste Workshop 15/16 November 2011 9

10. Statistical Data and Metadata EXchange
Why SDMX?
Harmonisation of (meta)data exchange process
Reduction of development and maintenance costs
Faster, more reliable, simpler data exchange
Technical Aspects
SDMX Artefacts
Data Structure Definition (DSD)
Code Lists, Concept Schemes, …
SDMX Registry
Metadata Repository
Hosting and Sharing of SDMX Artefacts
EDAMIS, SDMX, Validation
Waste Workshop 15/16 November 2011 10

11. What is a DSD? Describe statistics in a standard way DIMENSIONS
TIME
COUNTRY
FREQUENCY
TOURISM TOPIC
OBS_VALUE
OBS_STATUS E P
DIMENSIONS
ATTRIBUTES
MEASURES
Describe statistics in a standard way
EDAMIS, SDMX, Validation
Waste Workshop 15/16 November 2011

12. How does a DSD help? Statisticians Dimensions of a table / data cube
Attributes of observations (flags, footnotes, …)
Code lists used
Information Technology
Allows IT tools to validate
Structure of data
Codes used
Facilitates automated information exchange
Across different statistical domains
Standardised tools
Web services
EDAMIS, SDMX, Validation
Waste Workshop 15/16 November 2011

13. SDMX Implementation in Member States
Tools offered
Action
Web Forms
No costs for organisation
Manual work for senders (type, copy/paste)
SDMX Converter
(sender’s PC)
Installation on sender’s PC
Manual work for senders (convert)
(batch mode)
Installation on server
SDMX Reference Infrastructure
Mapping of database to DSDs
Own SDMX Implementation
Local development in organisations
EDAMIS, SDMX, Validation
Waste Workshop 15/16 November 2011

14. How do EDAMIS and SDMX come together?
Validation at the Entry Point
Based on SDMX standard
EDAMIS Web Forms (EWF)
Graphical SDMX editor with on-screen data validation
Spreadsheet interface, no SDMX knowledge needed
Collection of small volumes
EDAMIS Validation Engine (EVE)
Same experience for batch validation
EDAMIS, SDMX, Validation
Waste Workshop 15/16 November 2011

15. EDAMIS Web Forms DSD EDAMIS, SDMX, Validation
Waste Workshop 15/16 November 2011

16. eDAMIS webforms in waste
Already in use for
WEEE
Packaging Waste
ELV
Municipal waste
Four more webforms for WStatR
Waste generation
Waste treatment
Two tables on waste treatment facilities
Recovery, incineration
Landfills
EDAMIS, SDMX, Validation
Waste Workshop 15/16 November 2011

17. Advantage of webforms for Waste
Data entry and data transmission in one tool
Data entry in Excel replaced with webform
Separate data transmission via EWP or EWA no longer necessary
Only quality report to be sent separately
Minimal maintenance effort; same tool for all data suppliers.
EDAMIS, SDMX, Validation
Waste Workshop 15/16 November 2011

18. Data Validation Before/During Transmission (EDAMIS – “First Level”)
Performed in Member States
Format Check (SDMX-ML)
Code Check (SDMX DSD)
Basic Value Check
After Transmission (MDT – “Second Level”)
Performed in Eurostat
EDAMIS, SDMX, Validation
Waste Workshop 15/16 November 2011

19. How does it all come together?
Validation Services
Member State
Eurostat Prod
Eurobase
EDAMIS
SDMX-RI
MDT
DSDs
SDMX Registry
EDAMIS, SDMX, Validation
Waste Workshop 15/16 November 2011

20. Workflow eDAMIS Validation Engine
SDMX Registry
DSDs
Web Service
Browser
SDMX Converter
CSV
Settings
MS Database
Eurostat Production Unit
EDAMIS Server
Validation
SDMX-ML
eWP
eWA
Report
Report
EDAMIS, SDMX, Validation
Waste Workshop 15/16 November 2011

21. eDAMIS Validation Engine – Validation Report
EDAMIS, SDMX, Validation
Waste Workshop 15/16 November 2011

22. EVE – Data Validation Features
Same Validation Rule Syntax as Web Forms
Within one file and reference period
Different rule sets per reference period possible
Country specific rules
Mandatory values, Range checks
Basic expressions, comparison (+ - * / < > =)
Mathematical expressions (SUM, AVG, MIN, MAX, …)
Logical expressions (AND, OR, NOT)
EDAMIS, SDMX, Validation
Waste Workshop 15/16 November 2011

23. WStatR validations in EVE
Valid combinations of “waste” and “hazard”
No negative values
Warning if cell is empty
Valid combinations of “status flags” and “confidentiality flags”
REGIO: ‘wrong’ NUTS totals only lead to warnings
Inconsistencies of national / regional data may be acceptable
EDAMIS, SDMX, Validation
Waste Workshop 15/16 November 2011

24. Workshop - Demonstration of the Workflow
Data sender using EDAMIS Web Portal
CSV  SDMX-ML  Pre-Validation
EDAMIS, SDMX, Validation
Waste Workshop 15/16 November 2011

25. Questions? EDAMIS Support Team SDMX Support Team
SDMX Support Team
EDAMIS, SDMX, Validation
Waste Workshop 15/16 November 2011