SISAI STATISTICAL INFORMATION SYSTEMS ARCHITECTURE AND INTEGRATION WORKING GROUP 3rdMEETING 13-14 MAY 2013 ITEM 1.3.3 SDMX Reference Infrastructure Evolution

SDMX Reference Infrastructure Evolution Eurostat Unit B3

What is SDMX Reference Infrastructure Universal framework for modern data provision and exchange Set of pick-and-choose reusable building blocks allowing a statistical office to expose data to the external world based on access rights Designed to provide data and structural metadata based on mappings to each organization's dissemination data warehouse Uses SDMX standards incl. one for Web Services Eurostat Unit B3 3

Why use SDMX Reference Infrastructure Developed to simplify the exchange of data Provides standard software and components, allowing individual statistical organisations to interact and exchange their data using the same software and methodology Modular approach, use part or the entire infrastructure, extend it by adding new modules or modify it in any other way to suit their own purposes Developed in both Java and .NET Eurostat Unit B3 4

Where is the SDMX Reference Infrastructure used As of May 2013 deployed in 22 EU countries Tested across EU Member States (2011 population census) Running in Mexico Expression of interest: Latin America, the Caribbean, OECD and Russia Autumn 2013: Expected to run in EU27 member states In Eurostat dissemination Web Service upgrade Eurostat Unit B3 5

Development strategy 2012 – 2013 “Systems must continually be adapted to the changing environment; otherwise their utility will progressively decline”. Architectural changes (two step approach) Intermediate solution to solve performance & out of memory errors "Ultimate" solution to provide a common API and implement SDMX 2.1 Implement new user requests and correct defects Widen the scope and usage of SDMX-RI among data providers ESS.VIP-programme such as the ICT project Reuse for other statistical data collections DSWS (Eurostat dissemination web service) Eurostat Unit B3 6

Intermediate solution: What changed Streaming of data in the service Usage of JAX-WS in Java (Axis 1.0 could not support streaming) SDMX Model/IO (5) revised with Streaming Writers QueryParser (2) is now part of SDMX Model/IO (5) library DataGenerator (4) functionality is now included in the SDMX Model/IO (5) library DataRetriever API (3) changes due to streaming Eurostat Unit B3 7

Intermediate solution: Benefits Better performance Improvement of approximately 75% in concurrent users scenarios Solution to “Out Of Memory” problems for large datasets No memory constraints Eurostat Unit B3 8

Why a “Ultimate” solution Eurostat’s decision for a Common SDMX API Ensure the reusability of common building blocks and allow different implementation of the API Create "clearer code" in the client programs by hiding the complexity of the SDMX messages by providing reader and writer interfaces Support for SDMX 2.1 New messages (data representation, queries) New Web Service interfaces (SOAP/REST) 9 Eurostat Unit B3

“Ultimate” solution: What changes (1) All modules will be modified to use the SDMX Common API (6) The SDMX Model/IO (5) will no longer be used For Java the MT* API implementation (7) will be used For .NET the API implementation (7) development is ongoing SDMX-RI Components APIs will be changed Due to SDMX Common API *Metadata Technology Eurostat Unit B3 10

“Utlimate” solution: What changes (2) SDMX 2.1 messages and new query features will be supported Web Service will be extended to support SDMX 2.1 standardized SOAP and RESTful APIs New Web Service endpoints will be added above the Controller and they will co-exist with SDMX 2.0 endpoint Will support SDMX 2.1 error handling Eurostat Unit B3 11

WS extension to SDMX 2.1 new interfaces SOAP Request 2.0 SOAP Request 2.1 REST Request 2.1 NSI_Service_2.0 NSI_Service_2.1 NsiRestService Web Service Provider Controller (1)Structure Retriever (6)Common SDMX API (3)Data Retriever (streaming) (7)SDMX 2.0 Implementation (7)SDMX API Implementation (7)SDMX 2.1 Implementation Eurostat Unit B3 12

“Utlimate” solution: Benefits Usage of Common SDMX API Interchangeable implementation Foster component reusability Support of data streaming Support of SDMX 2.1 New query capabilities New message formats Support of RESTful API Eurostat Unit B3 13

SDMX-RI Development Timeline summary Java SDMX-RI integrated Java SDMX-RI aligned Java SDMX-RI packaged Feb 2013 Mar 2013 April 2013 May 2013 June 2013 July 2013 Oct 2013 .NET SDMX-RI packaged .NET/Java SDMX-RI enhancements .NET SDMX-RI integrated .NET SDMX-RI Aligned MA/MS enhancements MA/MS integrated MA/MS aligned 14