1. 1 SDMX Reference Infrastructure (SDMX-RI) Work in progress, status and plans Bengt-Åke Lindblad, Adam Wroński Eurostat Eurostat Unit B3 – IT and standards for data and metadata exchange

2. Content  What is SDMX-RI  Why use SDMX-RI  Usage status  Development strategy  Release calendar Eurostat Unit B3 – IT and standards for data and metadata exchange 2

3. What is SDMX-RI  Universal framework for modern data provision  Set of pick-and-choose reusable building blocks allowing a statistical office to expose data to the external world based on access rights  Designed to provide data and structural metadata based on mappings to each organization's dissemination data warehouse  Uses SDMX standards incl. one for Web Services Eurostat Unit B3 – IT and standards for data and metadata exchange 3

4. 4

5. Why SDMX-RI  Developed to simplify the exchange of data  Provides standard software and components, allowing individual statistical organisations to interact and exchange their data using the same software and methodology  Modular approach, use part or the entire infrastructure, extend it by adding new modules or modify it in any other way to suit their own purposes  Developed in both Java and.NET Eurostat Unit B3 – IT and standards for data and metadata exchange 5

6. Usage  As of Sept 2012 deployed in 22 EU countries  Tested across EU Member States (2011 population census)  Running in Mexico  Expression of interest  Latin America, the Caribbean, OECD and Russia  July 2013:  expected to run in EU27 member states  in Eurostat dissemination Web Service upgrade Eurostat Unit B3 – IT and standards for data and metadata exchange 6

7. SDMX-RI versions Streaming SDMX-RI Initial Solution Common API SDMX v2.1 SDMX-RI Intermediate Solution SDMX-RI Ultimate Solution Eurostat Unit B3 – IT and standards for data and metadata exchange 7

8. Development strategy 2012-2013  Architectural changes are needed (two step approach)  Intermediate solution to solve performance & out of memory errors  Ultimate solution to provide a common API and implement SDMX 2.1  Implement new user requests and correct defects  Widen the scope and usage of SDMX-RI among data providers  ESSVIP project such as Web INfrastructure for Statistics  Reuse for other statistical data collections  DSWS (Eurostat dissemination web service) Eurostat Unit B3 – IT and standards for data and metadata exchange 8

9. Why an intermediate solution  Problems needed to be solved within short time for existing users  Performance  High memory consumption and response times constrained organizations from putting it in production  Out Of Memory errors  Impossible to serve large data requests. Eurostat Unit B3 – IT and standards for data and metadata exchange 9

10. What changed?  Streaming of data in the service  Migration to JAX-WS in Java  IO revised with Streaming Writers  Data Retriever API changed (streaming) Eurostat Unit B3 – IT and standards for data and metadata exchange 10

11. Benefits  Better performance  Improvement up to 4 times faster for.Net  Improvement up to 7 times faster for Java  Solution to Out Of Memory problems for large datasets  No memory constrains  Data are streamed to the client Eurostat Unit B3 – IT and standards for data and metadata exchange 11

12. Impact  Organizations that have installed initial solution  Only re-install the Web Service Package  Existing users of Web Service package not affected  Same v2.0 interface remain  Organizations using the components API  API has changed due to streaming support  Migration required  Organizations has done modifications to the source code  Have to make the changes again if they want to use the intermediate version. Eurostat Unit B3 – IT and standards for data and metadata exchange 12

13. Why an ultimate solution?  Use of common API (that supports various versions and flavors of SDMX)  Interfaces which allow implementation using the components and also the interfaces for component-to- component communication  Better inter-organization component reusability  Ability to support new versions of SDMX without changing our applications  Support of SDMX 2.1  New messages (data representation, queries)  New Interfaces (SOAP/REST) Eurostat Unit B3 – IT and standards for data and metadata exchange 13

14. What will be changed?  All modules migrated to the common API  To use of new SDMX Model/IO  Components APIs will be changed  Support of SDMX 2.1 messages and new query features  Extend to support 2.1 standardized SOAP and RESTful APIs  New Web Service end points  Co-existence with v2.0 service  Support of v2.1 error codes Eurostat Unit B3 – IT and standards for data and metadata exchange 14

15. Impact  Mapping Assistant users will be able to reuse their Mapping Stores  Mapping Store upgrade will be supported  Will be required to re-install MA only  Organizations that have installed initial solution  Only to re-install the Web Service Package  Existing users of Web Service package will not be affected  Same v2.0 interface will remain  Organizations using components API  API has changed due to streaming support  Migration will be required  Migration guidelines will be provided  Organizations done modifications to the source code  Will have to make the changes again if they want to use the new version. Eurostat Unit B3 – IT and standards for data and metadata exchange 15

16. Status - release  Common API development (jar/dll expected end 2012)  Java: ongoing .NET: not started yet (API should be ported to C# and then implemented)  SDMX 2.1 support (expected Q1 2013)  Not yet started (dependent on API)  Enhancements and bug corrections (independent)  Mapping Assistant (Sept 2012)  Improved transcoding, error handling, QueryEditor, etc  Mapping Assistant (end 2012)  Implementation of 11 request mainly linked to the user interface and the usage  Test Client  using encrypted user id and password Eurostat Unit B3 – IT and standards for data and metadata exchange 16

17. Bengt-Ake.Lindblad@ec.europa.eu Adam.Wronski@ec.europa.eu SDMX-RI contacts Eurostat Unit B3 – IT and standards for data and metadata exchange 17

18. Web Service Package extension to SDMX 2.1 and new interfaces Controller Data Retriever (streaming) Web Service Provider Common API Structure Retriever NSI_Service_2.0 NSI_Service_2.1 SOAP Request v2.0 SOAP Request v2.1 NsiRestService REST Request v2.1 API Implementation Eurostat Unit B3 – IT and standards for data and metadata exchange 18