Marco Pellegrino, Bengt-Åke Lindblad 26th Meeting of the Working Group of Local Coordinators for Data Transmission 25-26 October 2011 Agenda Item 13 SDMX Progress Report Marco Pellegrino, Bengt-Åke Lindblad The purpose of this document: - Inform local coordinators of the progress done in the last year on SDMX at international level - Report on the effort for improving the set of software tools which compose the SDMX Reference Infrastructure The report does not deal with the implementation work in statistical domains, which is the object of another report in this GLC meeting

SDMX progress Standards Development: April 2011, SDMX 2.1 Technical Standards released @ sdmx.org Outreach: May 2011, SDMX Global Conference Self-learning tutorials comprising video, textbook and self-test. Eight e-learning tutorials can be retrieved from the CIRCA platform Governance: Creation of two SDMX Working Groups (Technical Working Group and Statistical Working Group) In the last year, SDMX has been progressing considerably. Standards Development: SDMX 2.1 Technical Standards released in April 2011, after a year-long open consultation of experts and two workshops organised by Eurostat. SDMX Global Conference in May 2011: confirmed that SDMX is broadly used by statistical organisations along the statistical business processes. Documentation available. Eight e-learning tutorials released by Eurostat, each comprising a videos, a textbook and a self-test, on the following subjects: Introduction to SDMX, Information model, Data Structure Definitions, Metadata Structure Definitions, XML techniques, SDMX-ML messages, Pull architecture for data sharing 1 and 2. Available on CIRCA in a public folder. Being updated. Governance: Creation of two SDMX Working Groups (the Technical Working Group and the Statistical Working Group) in 2011, reporting to the SDMX Secretariat. These working groups have 20 members coming from SDMX organisations, statistical institutes and central banks, with a global geographical representation. Eurostat Unit B5 – Statistical Information Technologies GLC 26th Meeting – 25/26 October 2011

SDMX Action Plan 2011 To 2015 SDMX Sponsors released the Action Plan 2011 to 2015 Priority actions on: SDMX implementation Outreach and training Content-oriented Guidelines Technical standards Governance Input coming from the SDMX Statistical Working Group, the SDMX Technical Working Group and the SDMX Secretariat Based on the Global Conference 2011 and on the priorities identified by the Secretariat together with Technical Working Group (TWG) and Statistical Working Group (SWG), the SDMX Sponsors defined the SDMX Action plan 2011-2015. The priority actions to be carried out in the next years refer to SDMX implementation, outreach and training, content-oriented Guidelines, technical standards and SDMX Governance. The SWG and the TWG have held their first meeting this month. The work progress for the various actions will be monitored regularly by the SDMX Secretariat. Corrective actions will be undertaken if needed. The work progress will also be regularly communicated on the SDMX website www.sdmx.org. Eurostat Unit B5 – Statistical Information Technologies GLC 26th Meeting – 25/26 October 2011

The SDMX Reference Infrastructure Allows exposure of data from existing databases using SDMX standards and guidelines Provides re-usable building blocks that enable production of SDMX data from existing repositories, minimizing installation efforts .NET and Java classes Well-defined API Open Source package under the EUPL licence Stimulate the “SDMX community” of IT developers Support Census Hub and similar implementation projects SDMX-RI is a generalized service infrastructure used in SDMX projects. Experiences have highlighted some important points: Building the SDMX infrastructure requires the analysis of several factors, and consequently the development of complex software modules; National data, in many cases, can be retrieved from Member States' repositories, if data can be successfully mapped to the SDMX Data Structure Definitions defined at international level; The start-up phase is crucial, because the expert knowledge of SDMX standards, XML and WS technologies is not easily available; SDMX-RI = a set of tools that allows to connect your IT systems to the SDMX world. It is platform independent and composed of re-usable components. An organization can decide to use it as a whole, extend the infrastructure by adding new modules, modify some modules, or integrate some modules within its existing dissemination environment. Countries that installed the SDMX-RI are: Bulgaria, Belgium, Cyprus, France, Italy, Ireland, Latvia, Luxembourg, Malta, Mexico, Netherlands, Norway, Poland, Slovenia, Spain and Sweden. Expressing interest: Finland and Hungary (assistance in 2011) + Albania, Iceland, Romania, Serbia, Slovakia and Turkey (assistance in 2012). Russia, ECLAC, Korea, Taiwan. Technical advice to those which are interested. Main Goals: Support Census Hub and other Eurostat projects; Facilitate SDMX implementation within international organisations (also PC-Axis community); Conform to SDMX Web Service guidelines; Stimulate a “SDMX community of developer”; Limited investment by re-usable building blocks; .NET and Java implementations; Well defined API; Open Source package under the EUPL licence. Eurostat Unit B5 – Statistical Information Technologies GLC 26th Meeting – 25/26 October 2011 4

The modules use an open source architecture developed by Eurostat. This Reference infrastructure accesses the dissemination environment of a data provider, retrieves and converts the local data to SDMX format and provides web services to pull the SDMX compatible data by a data collector. The workflow starts from a dissemination database and ends with making SDMX compatible data available by using a Web Service. The building blocks are: Web Service Provider: The web service interface, which receives an SDMX Query Message as input and provides an SDMX Data File as response. SDMX Query Parser: The module that parses the SDMX Query Message (XML Document) into an internal SDMX Data Model . Data Retriever: Ensures access to the dissemination database and gets the data related to the SDMX Query Message. It applies the mapping to SDMX data structure stored in the Mapping store. Structure Retriever is similar in operation to the module mentioned above, which retrieves the SDMX Data Structures based on an SDMX Structure Query. It translates the query to an SQL statement and takes the SDMX Structural Metadata from the Mapping Store, delivering at the end an SDMX-ML structure message. SDMX Data Generator: Transforms data stored in the internal SDMX Data model into an SDMX Data File (XML Document). The Mapping Assistant component establishes the Mapping Store, which contains the Mapping and Transcoding information to translate the structures of local data in “lecgacy” systems to SDMX-compliant data. Not shown as a building block is the SDMX Model Library since it is a class representation of the SDMX Information Model and used for storing/handling SDMX messages within the building blocks of the SDMX Reference infrastructure. It is a re-usable component, which simplifies the interfaces for the SDMX Reference service infrastructure modules and represents all SDMX related information in Java or .NET objects. Eurostat Unit B5 – Statistical Information Technologies GLC 26th Meeting – 25/26 October 2011

SDMX Software Tools The SDMX Reference Infrastructure SDMX Registry Mapping Assistant SODI pull requestor SDMX Converter Data Structure Wizard SDMX tools are not part of the standard. They have been developed and made freely available to the SDMX community to help developing SDMX-based functionalities for implementation in local IT systems. The open source tools offer functionalities such as: Data structure definition Computer readable formats preparation Registries and web services Database loading and validation Tools were developed as portable packages or components which can be installed in other organisations. All software is licensed under the terms of the European Union Public License V.1.1 to ensure the long-term sustainability of the results and availability The SDMX Reference Metadata Editor XSD Generator More information: https://webgate.ec.europa.eu/fpfis/mwikis/sdmx/ Eurostat Unit B5 – Statistical Information Technologies GLC 26th Meeting – 25/26 October 2011 6 6

Next steps New project for re-use of SDMX-RI and extension of the Hub model Enhancement of IT tools and SDMX-RI Feasibility study on migration from SDMX 2.0 to SDMX 2.1 EGR (Euro-Groups Register): analyse how ISTAT Framework modules, currently used for the Euro-Groups Register, can be integrated within the SDMX-RI SDMX-RI User Group Missions, technical meetings Eurostat envisages the following activities: New project on re-use of SDMX-RI and experience from Census Hub in a regular data exchange activity. To adapt/enhance SDMX IT tools and a generalized service infrastructure in accordance to the needs arising from its user community and by its client systems. Work with EGR (EuroGroup Registers) team on use of SDMX-RI. Analyse how the following components, Manager and Client/Loader, of the ISTAT SDMX Framework can be integrated in the SDMX RI. Conduct a feasibility study on what is required in order to migrate the SDMX 2.0 implementation to SDMX 2.1. Identification of a group of Member States which are interested to participate actively in the SDMX Reference Infrastructure user group taking account of recent IT developments and their experience in the Census Hub pilot project. Eurostat may organise missions or a technical meeting to provide further explanation of the technical aspects of the work. Eurostat Unit B5 – Statistical Information Technologies GLC 26th Meeting – 25/26 October 2011

Eurostat Unit B5 – Statistical Information Technologies GLC 26th Meeting – 25/26 October 2011

The NSI Web Service The NSI Web Service has the following modules: Web Service Wrapper (light brown colour): this module implements the Web Service SOAP interface, and is also responsible for the control of the order to which the other modules are invoked. Structure Retriever (orange colour): this module is responsible for retrieving structural metadata from the Mapping Store database. It can also return partial code lists based on the available data from the Dissemination Database. Query Parser (red colour): this module is responsible for the parsing of the SDMX Query into the Information Model Data Retriever (green colour): this module is responsible for the generation of an SQL query which is equivalent to the SDMX Query. It also executes the query on the Dissemination Database. . Data Generator (blue colour): this module is responsible for the generation of SDMX-ML dataset. Eurostat Unit B5 – Statistical Information Technologies GLC 26th Meeting – 25/26 October 2011