SDMX Tools Overview and architecture ESTP Workshop on SDMX in National Accounts Luxembourg, 16-18 March 2016

Technical Integration National statistical organisations International organisations National databases (Mapping) Source SDMX Data Structure Defintions & Data Flows National data sources SDMX Implementation Challenge Metadata driven process based on SDMX standards and tools Full automation of the data exchange possible

The SDMX Components SDMX Information Model Content Oriented Guidelines IT Infrastructure for exchange and sharing Content Oriented Guidelines SDMX Information Model 3

The SDMX Components SDMX Information Model Content Oriented Guidelines IT Infrastructure for exchange and sharing Content Oriented Guidelines SDMX Information Model 4

SDMX Tools for National Accounts – decision tree Do you have a database for ESA? Yes Good start  1) RI + Mapping 2) CSV export with Converter 3) Own development No Do you plan to create one? Yes  RI+Loader No Really not?  SDMX Converter

Getting there: One goal – different possibilities Excel sheets Phase out in National Accounts SDMX Converter SDMX-ML file Database & Export SDMX Converter SDMX-ML file Database & Mapping SDMX Reference Infrastructure SDMX-ML file SDMX Web Service Same DSD  Same Format Database & Loader SDMX Reference Infrastructure SDMX-ML file SDMX Web Service

SDMX Converter https://webgate. ec. europa. eu/fpfis/mwikis/sdmx/index File based conversion Open source and platform independend (Java) Different ways of using it: (will be demonstrated) Graphical user interface Batch file (client side or server side) Web Service interface Reusing source code in your own Java application Formats: SDMX-ML 2.0/2.1, SDMX-EDI (Gesmes), FLR, CSV, Google DSPL XLS, XLSX (read only with additional preparation)

SDMX Dataset convert DSD The data structure definition being defined, let’s see how the statistical data are organised in the file to be exchanged, what is called Dataset in SDMX. In the related SDMX-ML structure Specific Time series data file we can highlight the different structural constructs to which attributes can be related: The Data set, where for this example Attribute UNIT is related The Series, where for this example no Attribute is related And finally the Observation level, where the Attribute Observation status is linked to. Each information in the table is placed into the dataset with its corresponding concept name: UNIT FREQ COUNTRY TOURISM INDICATOR TOURISM ACTIVITY TIME, OBSERVATION VALUE and OBSERVATION STATUS 8

Demonstration SDMX Converter & ESA Excel template Open the SDMX-NA maintenance page on SDMX.org (https://sdmx.org/?page_id=1498) Download ESA table 0101 template and the correct DSD Convert it to SDMX-ML with the SDMX Converter

Getting there: One goal – different possibilities Excel sheets Phase out in National Accounts SDMX Converter SDMX-ML file Database & Export SDMX Converter SDMX-ML file Database & Mapping SDMX Reference Infrastructure SDMX-ML file SDMX Web Service Same DSD  Same Format Database & Loader SDMX Reference Infrastructure SDMX-ML file SDMX Web Service

SDMX Reference Infrastructure https://webgate. ec. europa Data Provider Data Collector SDMX Registry Mapping Assistant DSD Web Svc Web Client Test Client Non-SDMX local database SDMX data set SDMX-RI

Getting there: One goal – different possibilities Excel sheets Phase out in National Accounts SDMX Converter SDMX-ML file Database & Export SDMX Converter SDMX-ML file Database & Mapping SDMX Reference Infrastructure SDMX-ML file SDMX Web Service Same DSD  Same Format Database & Loader SDMX Reference Infrastructure SDMX-ML file SDMX Web Service

National Source (e.g. CSV) ISTAT Loader & SDMX-RI http://sdmx2.istat.it/sodidownload/Download.aspx Data Provider Data Collector SDMX Registry ISTAT Loader Mapping Assistant DSD SDMX compliant local database Web Svc Web Client Test Client SDMX data set SDMX-RI

Functional characteristics Allows the creation of a dissemination database based on the SDMX-IM It is integrated with the SDMX-RI. Loads data files having CSV, FLR, GESMES-TS, SDMX formats Allows the user managing at three different levels: Administrator, DSDs and dataflows user. Uses three DBSM types (Oracle, SQL Server, MySql)

Architecture Loader SDMX, flr, gesmesTS,csv Loader database Web Service Structure files Mapping assistant Mapping Store database Structural tables Data tables Mapping set tables

Import dataflows from MS Creation of Data Tables Flow of the application Import dataflows from MS Creation of Data Tables Definition of the mapping Import Data files Publish dataflows User manager

SDMX Tools for National Accounts – decision tree Do you have a database for ESA? Yes Good start  1) RI + Mapping 2) CSV export with Converter 3) Own development No Do you plan to create one? Yes  RI+Loader No Really not?  SDMX Converter