1. SISAI STATISTICAL INFORMATION SYSTEMS ARCHITECTURE AND INTEGRATION
WORKING GROUP 3rdMEETING MAY 2013 ITEM SDMX Reference Infrastructure Evolution

2. SDMX Reference Infrastructure Evolution
Eurostat Unit B3

3. What is SDMX Reference Infrastructure
Universal framework for modern data provision and exchange
Set of pick-and-choose reusable building blocks allowing a statistical office to expose data to the external world based on access rights
Designed to provide data and structural metadata based on mappings to each organization's dissemination data warehouse
Uses SDMX standards incl. one for Web Services
Eurostat Unit B3 3

4. Why use SDMX Reference Infrastructure
Developed to simplify the exchange of data
Provides standard software and components, allowing individual statistical organisations to interact and exchange their data using the same software and methodology
Modular approach, use part or the entire infrastructure, extend it by adding new modules or modify it in any other way to suit their own purposes
Developed in both Java and .NET
Eurostat Unit B3 4

5. Where is the SDMX Reference Infrastructure used
As of May 2013 deployed in 22 EU countries
Tested across EU Member States (2011 population census)
Running in Mexico
Expression of interest:
Latin America, the Caribbean, OECD and Russia
Autumn 2013:
Expected to run in EU27 member states
In Eurostat dissemination Web Service upgrade
Eurostat Unit B3 5

6. Development strategy 2012 – 2013
“Systems must continually be adapted to the changing environment; otherwise their utility will progressively decline”.
Architectural changes (two step approach)
Intermediate solution to solve performance & out of memory errors
"Ultimate" solution to provide a common API and implement SDMX 2.1
Implement new user requests and correct defects
Widen the scope and usage of SDMX-RI among data providers
ESS.VIP-programme such as the ICT project
Reuse for other statistical data collections
DSWS (Eurostat dissemination web service)
Eurostat Unit B3 6

7. Intermediate solution: What changed
Streaming of data in the service
Usage of JAX-WS in Java (Axis 1.0 could not support streaming)
SDMX Model/IO (5) revised with Streaming Writers
QueryParser (2) is now part of SDMX Model/IO (5) library
DataGenerator (4) functionality is now included in the SDMX Model/IO (5) library
DataRetriever API (3) changes due to streaming
Eurostat Unit B3 7

8. Intermediate solution: Benefits
Better performance
Improvement of approximately 75% in concurrent users scenarios
Solution to “Out Of Memory” problems for large datasets
No memory constraints
Eurostat Unit B3 8

9. Why a “Ultimate” solution
Eurostat’s decision for a Common SDMX API
Ensure the reusability of common building blocks and allow different implementation of the API
Create "clearer code" in the client programs by hiding the complexity of the SDMX messages by providing reader and writer interfaces
Support for SDMX 2.1
New messages (data representation, queries)
New Web Service interfaces (SOAP/REST) 9
Eurostat Unit B3

10. “Ultimate” solution: What changes (1)
All modules will be modified to use the SDMX Common API (6)
The SDMX Model/IO (5) will no longer be used
For Java the MT* API implementation (7) will be used
For .NET the API implementation (7) development is ongoing
SDMX-RI Components APIs will be changed
Due to SDMX Common API
*Metadata Technology
Eurostat Unit B3 10

11. “Utlimate” solution: What changes (2)
SDMX 2.1 messages and new query features will be supported
Web Service will be extended to support SDMX 2.1 standardized SOAP and RESTful APIs
New Web Service endpoints will be added above the Controller and they will co-exist with SDMX 2.0 endpoint
Will support SDMX 2.1 error handling
Eurostat Unit B3 11

12. WS extension to SDMX 2.1 new interfaces
SOAP Request 2.0
SOAP Request 2.1
REST Request 2.1
NSI_Service_2.0
NSI_Service_2.1
NsiRestService
Web Service Provider
Controller
(1)Structure Retriever
(6)Common SDMX API
(3)Data Retriever (streaming)
(7)SDMX 2.0 Implementation
(7)SDMX API Implementation
(7)SDMX 2.1 Implementation
Eurostat Unit B3 12

13. “Utlimate” solution: Benefits
Usage of Common SDMX API
Interchangeable implementation
Foster component reusability
Support of data streaming
Support of SDMX 2.1
New query capabilities
New message formats
Support of RESTful API
Eurostat Unit B3 13

14. SDMX-RI Development Timeline summary
Java SDMX-RI
integrated
Java SDMX-RI
aligned
Java SDMX-RI
packaged
Feb 2013
Mar 2013
April 2013
May 2013
June 2013
July 2013
Oct 2013
.NET SDMX-RI
packaged
.NET/Java SDMX-RI
enhancements
.NET SDMX-RI
integrated
.NET SDMX-RI
Aligned
MA/MS
enhancements
MA/MS
integrated
MA/MS
aligned 14