1. SDMX Tools Overview and architecture
ESTP Workshop on SDMX in National Accounts
Luxembourg, March 2016

2. Technical Integration
National statistical organisations
International organisations
National databases
(Mapping)
Source
SDMX Data Structure Defintions & Data Flows
National data sources
SDMX
Implementation Challenge
Metadata driven process based on SDMX standards and tools Full automation of the data exchange possible

3. The SDMX Components SDMX Information Model Content Oriented Guidelines
IT Infrastructure for exchange and sharing
Content Oriented Guidelines
SDMX Information Model 3

4. The SDMX Components SDMX Information Model Content Oriented Guidelines
IT Infrastructure for exchange and sharing
Content Oriented Guidelines
SDMX Information Model 4

5. SDMX Tools for National Accounts – decision tree
Do you have a database for ESA?
Yes Good start 
1) RI + Mapping
2) CSV export with Converter
3) Own development
No Do you plan to create one?
Yes  RI+Loader No
Really not?
 SDMX Converter

6. Getting there: One goal – different possibilities
Excel sheets
Phase out in National Accounts
SDMX Converter
SDMX-ML file
Database & Export
SDMX Converter
SDMX-ML file
Database & Mapping
SDMX Reference Infrastructure
SDMX-ML file SDMX Web Service
Same DSD 
Same Format
Database & Loader
SDMX Reference Infrastructure
SDMX-ML file SDMX Web Service

7. SDMX Converter https://webgate. ec. europa. eu/fpfis/mwikis/sdmx/index
File based conversion
Open source and platform independend (Java)
Different ways of using it: (will be demonstrated)
Graphical user interface
Batch file (client side or server side)
Web Service interface
Reusing source code in your own Java application
Formats: SDMX-ML 2.0/2.1, SDMX-EDI (Gesmes), FLR, CSV, Google DSPL XLS, XLSX (read only with additional preparation)

8. SDMX Dataset convert DSD
The data structure definition being defined, let’s see how the statistical data are organised in the file to be exchanged, what is called Dataset in SDMX.
In the related SDMX-ML structure Specific Time series data file we can highlight the different structural constructs to which attributes can be related:
The Data set, where for this example Attribute UNIT is related
The Series, where for this example no Attribute is related
And finally the Observation level, where the Attribute Observation status is linked to.
Each information in the table is placed into the dataset with its corresponding concept name:
UNIT
FREQ
COUNTRY
TOURISM INDICATOR
TOURISM ACTIVITY
TIME,
OBSERVATION VALUE and OBSERVATION STATUS 8

9. Demonstration SDMX Converter & ESA Excel template
Open the SDMX-NA maintenance page on SDMX.org (
Download ESA table 0101 template and the correct DSD
Convert it to SDMX-ML with the SDMX Converter

10. Getting there: One goal – different possibilities
Excel sheets
Phase out in National Accounts
SDMX Converter
SDMX-ML file
Database & Export
SDMX Converter
SDMX-ML file
Database & Mapping
SDMX Reference Infrastructure
SDMX-ML file SDMX Web Service
Same DSD 
Same Format
Database & Loader
SDMX Reference Infrastructure
SDMX-ML file SDMX Web Service

11. SDMX Reference Infrastructure https://webgate. ec. europa
Data Provider
Data Collector
SDMX Registry
Mapping
Assistant
DSD
Web Svc
Web Client
Test Client
Non-SDMX local database
SDMX data set
SDMX-RI

12. Getting there: One goal – different possibilities
Excel sheets
Phase out in National Accounts
SDMX Converter
SDMX-ML file
Database & Export
SDMX Converter
SDMX-ML file
Database & Mapping
SDMX Reference Infrastructure
SDMX-ML file SDMX Web Service
Same DSD 
Same Format
Database & Loader
SDMX Reference Infrastructure
SDMX-ML file SDMX Web Service

13. National Source
(e.g. CSV)
ISTAT Loader & SDMX-RI
Data Provider
Data Collector
SDMX Registry
ISTAT Loader
Mapping
Assistant
DSD
SDMX compliant local database
Web Svc
Web Client
Test Client
SDMX data set
SDMX-RI

14. Functional characteristics
Allows the creation of a dissemination database based on the SDMX-IM
It is integrated with the SDMX-RI.
Loads data files having CSV, FLR, GESMES-TS, SDMX formats
Allows the user managing at three different levels: Administrator, DSDs and dataflows user.
Uses three DBSM types (Oracle, SQL Server, MySql)

15. Architecture Loader SDMX, flr, gesmesTS,csv Loader database
Web Service
Structure files
Mapping assistant
Mapping Store database
Structural tables
Data tables
Mapping set tables

16. Import dataflows from MS Creation of Data Tables
Flow of the application
Import dataflows from MS
Creation of Data Tables
Definition of the mapping
Import Data files
Publish dataflows
User manager

17. SDMX Tools for National Accounts – decision tree
Do you have a database for ESA?
Yes Good start 
1) RI + Mapping
2) CSV export with Converter
3) Own development
No Do you plan to create one?
Yes  RI+Loader No
Really not?
 SDMX Converter