1. SDMX IT Tools SDMX use in practice in NA
National Account use case
Jean-Francois LEBLANC
Christian SEBASTIAN
11-13 May 2016
Eurostat, Unit B3 – IT solutions for statistical production

2. Table of contents Objectives National Accounts (NA) NA business needs
NA Domains
NA Technical Integration
NA Data Validation Process
NA Project Cycle

3. Table of contents SDMX Tools in practice SDMX Tools for Data Providers
Data Structure Wizard
SDMX Converter
SDMX Reference Infrastructure
Tools relationship
SDMX Tools – Summary
SDMX Tools for Data Providers
Possibilities for Data Providers
IT Architectures for Data Providers
How to use the tools
Conclusion

4. 1 Objectives To reflect how National Accounts work
To present the SMDX IT tools
To expose how to work with the data and SDMX in real situations
To give a base understanding of the tools and their relationship that will be of help for further explanations

5. 2 National Accounts (NA)
BOP
National Accounts
Main Aggregates
Sector Accounts
Supply & Use
Input / Output
GFS
FDI
Excel questionnaires
GESMES files
Excel templates will be supported for a 2 years period. From then on, the NSI have to create their info in SDMX format because there won’t be any more support for excel templates.
The Converter could be a solution but not relying in new templates whatsoever.

6. 2 Why SDMX for NA?
Within Eurostat
Within the ESS
Internationally

7. 2.1 NA business needs Main trigger: new methodology SNA 93  SNA 2008
ESA 95  ESA 2010
Business process reengineering
Corporate production systems
Integration of production chains
Harmonised coding
Within Eurostat
Within the ESS
Internationally

8. 2.1 NA business needs Cross-domain integration Balance of Payments
Foreign Direct Investment
ESS Business process reengineering
Standardization
Improved validation
Reuse of tools
Within Eurostat
Within the ESS
Internationally

9. 2.1 NA business needs Data quality Lowering reporting burden
1 country  1 year  1 GDP
Shared validation rules
Across all organisations
Lowering reporting burden
1 data flow from national to international
International cooperation
Inter-Agency Group on Economic and Financial Statistics = SDMX Sponsors
Example: PGI Website
Within Eurostat
Within the ESS
Internationally

10. 2.2 NA Domains Shared Concepts and Code Lists
National Accounts
SNA 2008
ESA 210
Other related domains
FDI statistics
Other domains
Balance of Payment
Statistics
Shared Concepts and Code Lists
Cross Domain Concepts and Code Lists
National accounts have 5 domains with one DSD each (MAIN aggregates, SECtor accounts and government finance, Supply/use and input/output statistics, REGional statistics, and PENSion schemes.
FDI and BoP (Balance of Payment) are not using the NA DSDs, these domains have their own DSD. They only share some codelists with NA DSDs.

11. 2.3 NA Technical Integration
National statistical organizations
International organizations
National DBs
Mapping
Source
SDMX DSDs & Data Flows
National data sources
Metadata driven process based on SDMX standards and tools Full automation of the data exchange possible

12. 2.4 NA Data Validation Process
Before/during transmission “First Level”
Currently covered by SDMX
Format Check (SDMX-ML)
Code Check (SDMX DSD)
Code used correctly
(SDMX Dataflow & Constraint)
After Transmission “Second Level”
Not covered by SDMX
Detailed value check
Validation expressions …

13. 2.5 NA Project Cycle
All projects go through the same process which is shown in this cycle

14. Registry DSW Converter SDMX-RI 3. SDMX Tools in practice Find DSDs
Edit/Create DSDs
Data files -> SDMX files
Database -> SDMX WS/files
The DSDs, where can they find them? In the registry! How can they edit/create the DSD? The DSW is the solution.
The first conversions can be realised with the converter. It is an intermediate solution for the dissemination. The converter also implements limited validation functionalities.
Remember that the converter is limited when it comes to the dissemination phase.
How can the user export the data directly from the database to the client? The RI is the solution.
The conclusion is that Eurostat tools cover each need and it is illustrated by the NA example.

15. Any oranization can create their EURO SDMX registry
The Data Structure Wizard (DSW) application provides an improved solution for the management of Data Structure Definitions (DSDs) within the SDMX-ML v2.0 and 2.1 standards. DSW covers and completes the following functionalities for handling structural metadata artefacts from the KeyFamilyDB Access tool, SDMX Registry, Eurostat GESMES Structural Metadatabase and ECB Structural Metadatabase (the functionality of the two GESMES tools is under development at the time being).
Specifically, DSW supports:
Maintenance of the following SDMX v2.0 and v2.1 artefacts: Code Lists, Concept Schemes, Data Structure Definitions, Data Flows, Category Schemes, Hierarchical Code lists and Organization Schemes Dynamic creation of a template for a specified DSD in SDMX v2.0 and v2.1 format
Import and export of artefacts from/to SDMX-ML v2.0 and v2.1 format
Import and export of DSDs from/to GESMES/TS structure files
Interaction with any standard-compliant SDMX Registry (using SDMX RegistryInterface messages)
Connection to SDMX Registry with "query" privilege and update of the local repository of artefacts
Update of the SDMX Registry, using the "maintain" privilege, with an artefact that was edited or created locally
Print of summary reports for Codelists, Concepts from ConceptSchemes, DSDs, Dataflows
Print reports for the DSDs components and for the Codes from Codelists
Advanced search (search Dataflow for a specified DSD, search DSD for a specified Dataflow, search all DSDs using a code list and search all the DSDs using a given concept)
SDMX-ML Data Authoring feature (creating sample SDMX-ML datasets from DSDs)
3.1 Data Structure Wizard
Java tool (open source and platform independent)
Supports
Maintenance of SDMX v2.0 and v2.1 structures
Import and export of artefacts from/to SDMX-ML v2.0, v2.1 and GESMES format
Interaction with Euro SDMX Registry (retrieval, modification and search of artefacts)
Sample dataset creation

16. 3.2 SDMX Converter Java tool (open source and platform independent)
File based conversion
Different ways of using it:
Graphical user interface
Command Line interface
Web Service interface
Common API interface
Formats*: SDMX-ML 2.0, SDMX-ML 2.1, SDMX-EDI (Gesmes), FLR, CSV, Google DSPL, predefined Excel templates
*Exceptions may apply

17. 3.3 SDMX Reference Infrastructure
Data Consumer (external)
Web Service
NSI Client
Dissemination DB
Mapping
Store
Mapping
DSD
Is my dataflow working?
Is my WS working?
Local
environment
Test Client
(Desktop Software)

18. 3.4 Tools relationship produces stores uses DSD DSW SDMX Registry
ESS-MH
MSD
MSD
produces
DSD
SDMX MetaDataset
uses
produces
uses
uses
SDMX Converter
SDMX-RI
Mapping Assistant
SDMX Dataset
produces
exposes
produces
Mapping
Census Hub
CSV or other datafile
National DB

19. 3.4.1 Tools relationship – Example
Eurostat produces NA DSDs
Eurostat stores NA DSDs
DSW
SDMX Registry
DSD
DSD
NSI uses
NA DSDs
SDMX-RI
NSI maps
NA DSDs
National DB
Mapping Assistant
Mapping
The XSD Generator is responsible for producing XML Schema Definitions (XSD) based on a DSD received. The present version of the tool produces XSDs for Compact, Cross-Sectional or Utility formats of SDMX standard v2.0 and can be used
• As a re-usable building block, through its API.
• Through a standalone Graphic User Interface (GUI)
• Through Command Line Interface (CLI)
• Through a web GUI
The tool is licenced under the terms of the European Union Public Licence V.1.1
A complete Euro SDMX Registry package is stored on CIRCA repository and can be downloaded here: XSD Generator_app_v1.0.0_ (ZIP) The package contains an application, source code and also documentation such as a User Manual, Installation Guide, Analysis and Design as well as Testing Documentation.
All versions of the tool can be downloaded from CIRCA repository.
SDMX-ML Dataset
NSI produces
NSI uses
NA DSDs
SDMX
Converter
NSI exports NA data (CSV, XLS)

20. 3.5 SDMX Tools – Summary Eurostat tools  Eurostat SDMX Info Space
SDMX Converter
SDMX Reference Infrastructure & Mapping Assistant
SDMX Registry & Data Structure Wizard
Eurostat SDMX Info Space is a wiki with
Information about the tools
Trainings
Videos
References to CIRCABC
Eurostat SDMX Info Space is a wiki used mainly by ESTAT B3 and B5.

21. 4. SDMX Tools for Data Providers
Tools offered
Use
Action
Web Forms (EU)
Excel-like templates
Transmission of low volumes of data
No costs for EU organisations
Manual work for senders (type, copy/paste)
SDMX Converter
Converts data files from different file formats to SDMX format
Local installation on sender’s PC
Installation on server
SDMX Reference Infrastructure
A set of tools that allows to transform directly your data from a database to SDMX
Mapping of database to DSDs
Common API
Supports the development of local SDMX systems
Local development in organisations

22. 4.1 Possibilities for Data Providers
Webforms
EDAMIS
Single Entry Point
SDMX-ML file
Database export
SDMX Converter
Database
SDMX Reference Infrastructure
SDMX-ML file SDMX Web Service
Same DSD 
Same Format
Excel sheets
SDMX Converter
SDMX-ML file

23. 4.1.1 Possibilities for Data Providers. NA.
Not applicable in National Accounts
Webforms
EDAMIS
Single Entry Point
SDMX-ML file
Database export
SDMX Converter
Database
SDMX Reference Infrastructure
SDMX-ML file SDMX Web Service
Will be phased out in National Accounts
Same DSD 
Same Format
Excel sheets
SDMX Converter
SDMX-ML file

24. 4.2 IT Architectures for Data Providers
Webforms
Web-based
Push mode (EDAMIS)
Excel sheets
SDMX Converter
Local conversion
Push mode (EDAMIS)
Database export
All domains in Eurostat use push mode except Census HUB
SDMX Converter
Local conversion
Push mode (EDAMIS)
Database
SDMX Reference Infrastructure
Pull mode

25. 4.3 How to use the tools Design your DSD using the DSW
If you are a Data Collector
Design your DSD using the DSW
Make the DSD available in the SDMX Registry
Inform your data providers for the available DSD

26. 4.3 How to use the tools Download/pull the DSD from the SDMX Registry
If you are a Data Provider
Download/pull the DSD from the SDMX Registry
If you have a DDB
Data stores as files
DDB
Install and use the SDMX-RI
Install and use the Converter
Expose the data to be pulled (WS, HUB)
Push via EDAMIS

27. 4.4 Conclusion SDMX statistical and technical standards
& SDMX IT tools and IT infrastructure
provide the "toolkit" for
the harmonisation and integration of data structures
the rationalisation of national/international data flows
finally improving the data quality of official statistics.

28. SDMX in practice in NA