1. SDMX & Validation
Item 4.1
Data Transmission Coordinators Group meeting
17-18 June 2019
Luca Gramaglia
Eurostat, Unit B5

2. Agenda News on SDMX & Validation Tools Implementation News
Global Conference and SDMX 3.0

3. News on SDMX & Validation Tools

4. Eurostat validation architecture
SDMX Converter
SDMX-RI
Let's go through what is new in the different components!

5. SDMX Converter & STRUVAL
All public releases of the SDMX Converter and STRUVAL are available on CIRCABC:
Last official public release of the SDMX Converter was version
Version 6.14 published as a test release last year, but because of issues with it it was not widely endorsed for use.
Eurostat is working on a new public release, version 7.x

6. SDMX Converter & STRUVAL
4 flavours: GUI, Web Service, API, CLI
STRUVAL can only validate SDMX-ML
GUI has a "one-screen" interface
SDMX Converter 7.x
5 flavours: GUI, Web Service, API, CLI, WebApp
STRUVAL can validate SDMX-ML, (SDMX-)CSV, FLR and Excel
GUI and WebApp built as a multiple screen wizard
+ bug fixes, other features…

7. SDMX Converter & STRUVAL
SDMX Converter GUI
SDMX Converter 7.x Web App

8. SDMX Converter & STRUVAL
Microsoft's (former) Vision Statement "A computer on every desk" Eurostat's SDMX Support Vision Statement "Let's please try not to have a Converter on every computer"
Eurostat does not plan to make any new Desktop GUI releases after 7.x
Web App will be offered as a public service
STRUVAL Web Service already exposed online (username and password must be requested)

9. Other SDMX tools news
SDMX Tools Task Force and SERV Task Force merged into permanent entity – Shared Tools Expert Group
SDMX-RI release cycle
Public release by end June
Test release of the web Mapping Assistant and Mapping Assistant WS/API -> 2nd half of the year
Test releases are made available under

10. VTL version history VTL 1.0: March 2015 VTL 1.1: November 2016
Collection of comments (public review)
VTL 1.1: November 2016
VTL 2.0: Published 16 April 2018 on SDMX Website (at
July 2018: eBNF (technical notation for implementation)

11. VTL version 2.0 Tools and services under development
ECB VTL parser and BIRD portal
ESS-NSI:
Norway Java API based on JSON-stat format
Poland VTL to SQL translator (UNECE paper)
Istat VTL Editor
Eurostat:
Sandbox: simple GUI + VTL translator to SQL
Compiler (part of CONVAL Service – under implementation)
Validation Rule Manager (planned)

12. VTL implementation in CONVAL
Timeline
Operators per Iteration => usable product after Iteration 2
Start Fri 8/06/18
Finish Mon 3/08/20
July
October
January
April
Iteration 1 Fri 8/06/18 - Tue 4/09/18
Iteration 2 Tue 4/09/18 - Mon 15/04/19
Iteration 3 Mon 3/06/19 - Tue 22/10/19
Iteration 4 Wed 23/10/19 - Thu 12/03/20
Iteration 5 Fri 13/03/20 - Mon 3/08/20
Today

13. Validation Rule Manager
Other plans
CONVAL deployment
Standard enterprise stack
WebLogic application server
Oracle database
Open source stack
Apache application server
PostgreSQL database
Predefined Docker images
Standalone installer (to be phased out)
Validation Rule Manager
VTL editor
SDMX support
Validation rule/script management functionality
Versioning, grouping
Validation rule templates
Access control and permissions
Web services

14. Implementation news

15. SDMX-based validation for everyone The light implementation
The light implementation focuses on making available SDMX-based validation tools to all statistical domains
Does not foresee changes in structure/codes and does not prescribe switching to an SDMX standard format. Only seeks to describe the current structure of the data via a DSD to enable automated SDMX-based validation
Effort is foreseen to be small for domains that currently receive data in CSV or FLR, larger for domains using Excel.
Applicable to both aggregate data and micro-data.

16. SDMX-based validation for everyone The light implementation
Light implementation gives access to the validation architecture developed by the ESS Vision 2020 Validation project.
Should represent the minimum SDMX implementation level for all domains

17. Light implementation: example
Example below taken from the monthly aggregate data exchange in trade statistics.
Exchange occurs using CSV, simple file structure (see screenshot below)
5 dimensions (Time Period, Reference Area, Flow, Product code, Trade Partner Area) and one measure (total trade value)

18. Light implementation: example
The data structure does not comply to the SDMX statistical guidelines:
Time format does not follow SDMX recommendations
Standard recommnded concepts such as Frequency, Observation status and Confidentiality status are not present
Coding for Product code and Partner area do not follow recommended code lists
The data structure also does not follow the technical standards: the transmission is not in an SDMX-compliant format.
However, the domain wanted access to the SDMX pre-validation services now available

19. Light implementation: example
Solution: create a DSD that represents the current structure and that can be used to implement SDMX-based validation services.
The DSD for this dataset is in the Euro-SDMX Registry (acceptance) under the name ESTAT+AGG_ITGS+1.1
No format / structure changes required for data providers
Validation services can be made available to both Eurostat and data providers
Structure is defined in the Euro-SDMX Registry

21. Highlights New global DSDs now or soon available (SEEA, Education)
Pre-validation services available for around 14% of datasets, representing around 25% of transmissions
Important steps forward in Demography and Energy domains

22. GESMES: end of life?
Over the past 12 months, the percentage of GESMES transmissions in the STS and PRODCOM domains (red line in graph above) has fallen from 63% to 19%. Full phase-out is expected by Q3/2019

23. GESMES: end of life?
After the completion of the phase-out in STS and PRODCOM, the overall yearly GESMES usage figures are expected to be substantially reduced.

24. GESMES: end of life?
ECB is expected to become the source of 51% of the remaining GESMES transmissions.
The importance of GESMES for NSIs would be minimal. GESMES would be used for only 2% of transmissions from NSIs.

25. GESMES: end of life?
Given the current low usage of GESMES and the availability of alternative formats, Eurostat will likely not introduce additional GESMES support in the Eurostat corporate tools.
Eurostat is also in discussion with the ECB to completely phase-out GESMES for the ECB-Eurostat exchange, which would further dimisnish the importance of GESMES.

26. SDMX Global Conference & SDMX 3.0

27. SDMX Roadmap 2020: Where we are
Strengthening the implementation of SDMX
Making data usage easier
Implement SDMX in more statistical domains
Develop additional SDMX Data Structure Definitions
Encourage more data sharing agreements
Improve data sharing processes with a better coverage of reference metadata
Encourage the use of SDMX for data dissemination
Encourage compatibility with micro-data standards
Develop closer collaboration with LOD community
Provide more SDMX-compatible exchange formats
Modernising statistical processes and improving standards
Better communication and capacity building
Further develop the SDMX IT infrastructure and software tools
Improve technical standards and statistical guidelines for a better coverage of user needs
Integrate data validation within the SDMX standard, based on VTL
Production of targeted communication material for users and decision-makers
Review format of Global SDMX Conferences
Increased coordination in capacity-building
Mapping of SDMX readiness and maturity
Developing regional strategies

28. The final stretch: Needed changes to the standard
The realisation of the 2020 roadmap requires changes to the current version of the standard. This is especially true in the following areas:
1.4: Merging DSDs and MSDs into a single object would greatly facilitate the support of reference metadata exchange in SDMX
2.1: An increase in multi-language support for the RESTful API will increase the usability of SDMX data for data dissemination and visualisation
2.2: Improvements to the standard are needed to better support microdata and geospatial data
2.4: The promotion of user-defined formats will facilitate the implementation of SDMX at national level
3.1: Improving the notification / subscription mechanism in SDMX will increase its capacity to support data sharing use cases
3.2: Several important requests from users must be implemented (e.g. more versatile Codelist maintenance, introducing semantic versioning etc.)
3.3: VTL must be implemented in the SDMX Information Model for SDMX to better support validation

29. SDMX 3.0: prioritisation exercise
A public consultation on the proposed features for the "SDMX 3.0" package has been launched on SDMX.org. The public consultation will last until the end of June. All ESS organisations are invited to participate. Link to the public consultation:
Based on the outcome of the public consultation, a final proposal for the scope and implementation timeline for SDMX 3.0 will be drafted.

30. 2019 SDMX Global Conference
Biannual key event for the community to share information on recent and upcoming SDMX developments
Upcoming conference: 7th SDMX Global Conference
„SDMX for you”
Organised by the SDMX Sponsors (this year in collaboration with the HCSO)
Date: September 2019
Location: Budapest, Hungary

31. Register! Registration is open! Conference language is English
Participation to the SDMX Global Conference is free of charge
Registration is open!
Please use the online registration form

32. Questions & Discussion
Would any TCOs be interested in testing the new Converter WebApp / GUI prioir to the public release?
Any questions / feedback on the further development and implementation directions? Any requests for continuing to develop GESMES support for new features?
TCOs are invited to participate in the public consultation on SDMX 3.0
Register for the Global Conference and spread the word!

33. Thank you for your attention!