SDMX as basis for water data reporting Working Group on Education and Training Statistics 25 April 2013 Daniel Suranyi Eurostat B5 Management of statistical data and metadata

Questions What is SDMX? Why use it? How do we get there? Where are we today? What can data providers do?

Statistical Data and Metadata EXchange World Bank UNSD SDMX ISO TS 17369 3 3 3 3

The SDMX Components SDMX Information Model Content Oriented Guidelines IT Infrastructure for exchange and sharing Content Oriented Guidelines SDMX Information Model 4 4

The SDMX Components Describe statistics in a standard way Objects and their relationships Data Structure Definition (DSD), Concepts, Code List Central management and standard access SDMX Registry, SDMX Web Services Cross Domain Concepts Cross Domain Code Lists Statistical Domains Metadata Common Vocabulary Push Provider generates and sends file to receiver Pull Provider opens web service to data Receiver downloads regularly Hub Special case of pull: receiver downloads on end user request 5 5

...not just a transmission format ! The SDMX Components IT Infrastructure for exchange and sharing Content Oriented Guidelines SDMX Information Model ...not just a transmission format ! 7 7

What is a DSD? DIMENSIONS ATTRIBUTES MEASURES FREQUENCY COUNTRY TIME COUNTRY FREQUENCY TOURISM TOPIC OBS_VALUE OBS_STATUS P E DIMENSIONS ATTRIBUTES MEASURES

What is a DSD? Describe statistics in a standard way DIMENSIONS TIME COUNTRY FREQUENCY TOURISM TOPIC OBS_VALUE OBS_STATUS P E Describe statistics in a standard way DIMENSIONS ATTRIBUTES MEASURES

Questions What is SDMX? Why use it? How do we get there? Where are we today? What can data providers do?

Eurostat SDMX Implementation Process

Questions What is SDMX? Why use it? How do we get there? Where are we today? What can data providers do?

Eurostat project status (data exchange) JVS/LCI (Job Vacancy Statistics/ Labour Cost Index) Water International Trade* Education? STS (Short Term Statistics) TEC (Trade by Enterprise Characteristics) Maritime R&D* (Research & Development) National Accounts* Balance of Payments* EGR IS (EuroGroups Register Identification Service) Aviation Fisheries* Orchards Pesticides TIC (Trade by Invoicing Currency) Waste * Other international organisations involved

Questions What is SDMX? Why use it? How do we get there? Where are we today? What can data providers do?

SDMX Implementation in Member States Tools offered Action Web Forms (only selected domains) No costs for organisation Manual work for senders (type, copy/paste) SDMX Converter (sender’s PC) Installation on sender’s PC Manual work for senders (convert) (batch mode) Installation on server SDMX Reference Infrastructure Mapping of database to DSDs Own SDMX Implementation Local development in organisations

SDMX Implementation in Member States Example SDMX Converter Generate local data file Take DSD and data file Produce SDMX-ML All steps each time Example SDMX-RI (installed in ~20 NSIs for Census Project) Take DSD Link to database Produce SDMX-ML Step 1 & 2 only once Only step 3 each time

Data Validation Process Before/During Transmission (“First Level”) - Implicit through SDMX - Format Check (SDMX-ML) - Code Check (SDMX DSD) After Transmission ( “Second Level”) - Currently not in SDMX - Detailed value check - Mirror check - …

Support for data providers SDMX Reference Infrastructure Installed in ~20 countries (NSIs) for Census Hub Can be extended to other domains 1st contact point - Local Transmission Coordinators In National Statistical Institutes Experience with EDAMIS and SDMX 2nd contact point - Eurostat support teams

Questions? SDMX Support Team SDMX Website Eurostat SDMX Info Space ESTAT-Support-SDMX@ec.europa.eu SDMX Website http://www.sdmx.org Eurostat SDMX Info Space https://webgate.ec.europa.eu/fpfis/mwikis/sdmx