SDMX : General introduction H. Linden, Eurostat, Unit B5 STATISTICAL DATA AND METADATA EXCHANGE SDMX : General introduction H. Linden, Eurostat, Unit B5 this is meant to be a presentation of about 15 – 20 minutes, giving you a down-to-earth summary of the project within the context of SDMX

What is SDMX and why is it so important? Current situation :  Lacking harmonisation causes extra costs and inconvenience !!

What is SDMX and why is it so important? Solution: A common and harmonised phone charger Benefits : For producers One charger  cheaper prod. costs One package cheaper transport costs For clients One charger serves all  convenience Lower prod. Costs  lower phone prices Possible charging from computers For everyone Less chargers needed  environmental benefits

What is SDMX and why is it so important? Current situation in the ESS : Different types of data and metadata files exchanged  Conversion of formats needed No standardisation of the statistical contents  Renaming, mappings needed Problem of correspondence between variables, codes, etc. The way forward: creation of technical, IT and statistical standards to be used for official data and metadata

What is SDMX and why is it so important? SDMX is carried out by seven international organizations: BIS, ECB, Eurostat, IMF, OECD, UN, World Bank; These organisations have joined their forces with the results of creating the SDMX technical and statistical standards and guidelines together with an IT service infrastructure and IT tools to be used for the efficient exchange and sharing of statistical data and metadata. These technical and statistical standards and guidelines are used and implemented more and more by statistical organisations around the world. 5

SDMX within the European Statistical System In 03/2009, the Eurostat top management confirmed SDMX and its implementation within the ESS in stating: “As a first step before making SDMX compulsory for all domains in Eurostat, the use of SDMX would be made compulsory for all new or considerably changed datasets and reference metadata sets.” The SDMX technical and statistical standards and guidelines can be regarded as one of the main enablers for implementing the new Eurostat vision based on the Commission Communication (COM 404/2009).

SDMX – the main components The SDMX technical standards (v. 2.1 revised in August 2011) SDMX information model Data and metadata messages and query formats Registry services The SDMX guidelines to harmonise the statistical contents the SDMX Content-oriented Guidelines The SDMX IT architecture the “push”, “pull” and “hub” approaches The SDMX IT tools IT tools produced by sponsoring organisations or others which are openly available The SDMX IT tools are accessible through the SDMX IT tool database on the SDMX website IM and syntax specifications form what is named “technical standards” Version 1.0 specification of the technical standards has been approved by the International Organization for Standardization (ISO) as a Technical Specification (ISO/TS 17369: 2005 SDMX). The Version 2.0 specification (November 2005) broadens the framework to support wider coverage of metadata exchange as well as a more fully articulated architecture for data and metadata exchange. Steps are also being taken to bring this work forward within the context of ISO, assuring that SDMX technical standards build on other recognized standards and providing the basis for interoperability with them. The development of the content-oriented guidelines is being undertaken outside the ISO framework. This should facilitate steps by the SDMX sponsoring institutions to broadly involve content-oriented efforts of a wide range of experts that are already working in many subject-matter domains within the global statistical community. 7

SDMX components The SDMX Content-oriented Guidelines (released in 01/2009): Annex 1: The SDMX cross-domain concepts: List of statistical concepts relevant to statistical domains to be used within the SDMX technical standards Annex 2: The SDMX cross-domain code lists: Statistical code lists relevant to statistical domains to be used within the SDMX technical standards Annex 3: The statistical subject matter domains List of subject matter domains (e.g. demography statistics, national accounts…) Annex 4: The Metadata Common Vocabulary Metadata cross-domain statistical terminology used above SDMX is not only IT oriented, but also contains statistical standards; it is not only a data transmission format. 8

ESS SDMX implementation: process Specs Codes B5 Draft DSD Metadata & Codes B6 Pilot Project MS & B Process SDMX DSD Define B3 [B5 / B4] Parallel Stage MS Working Group MS [B] Preparation Compliance Implementation Production B4 [B3 / B2] 9

Benefits of SDMX SDMX is crucial for rendering the production method of EU statistics more efficient → it implements the new Eurostat Vision The use of the SDMX standards and guidelines facilitates the exchange of data and metadata between organisations (international and national) and their comparability and consistency The use of SDMX compliant European standards for reference and quality related metadata decrease production costs within the ESS; SDMX provides standard formats for the exchange of data/metadata from end-to-end of the statistical business process; SDMX provides common IT tools created by the different organisations which are freely available.

Costs of SDMX Creation and maintenance of the Data Structure Definitions used in each statistical domain for data exchange Adaptation to the standard terminology for structural metadata (code lists) Adaptation to the European Standards for reference and quality related metadata reports Discussions/agreements between Member States and international organisations are needed Possible re-adaptation of the data production and/or dissemination systems needed

SDMX – latest progress More emphasis on the implementation of SDMX within the ESS Accelerated implementation in statistical domains (more and more Data Structure Definitions created) Harmonisation of structural metadata (more and more harmonised code lists available) and reference metadata (full use of the ESMS; implementation of the ESQRS; possible future standard on process related metadata; …) SDMX is also being implemented into IT applications (eDamis, EBB, ESS-MH, etc.) Many training and other capacity building actions organised (for IT staff, statisticians…)

Summarising SDMX is global; it provides technical and statistical standards, IT tools and an IT infrastructure. SDMX implements the Eurostat Vision for statistical business process harmonisation. SDMX implementation in statistical domains is progressing (e.g. the ESMS is fully implemented at Eurostat). The SDMX implementation has some costs for production units, but the benefits should be larger over time as data and metadata exchange with Member States and other organisations will be facilitated. For more info, please visit: www.sdmx.org or the SDMX info space Single contact point : ESTAT-SUPPORT-SDMX@ec.europa.eu

QUESTIONS ?