1. Eurostat Unit B3 – IT and standards for data and metadata exchange
SDMX for IT Experts
Mapping Assistant
Jorge Nunes Raynald Palmieri
18-20 February 2014
Eurostat Unit B3 – IT and standards for data and metadata exchange

2. Table of Contents Mapping Assistant
Introduction
Mapping Store DB
Importing SDMX structure files
Configuring mappings
Dissemination Databases
Databases in practice.
Storage Schemes: A, B, C, D
Hands-on use of Mapping Assistant 2

3. Dissemination Databases
What does a DDB contain?
Data
Observations
Attribute values
Dimension values
Structural Metadata
Codelists
Dataflows
Reference metadata
Creation metadata
Processing metadata

4. Dissemination Databases Extracting data in SDMX
Metadata
Headers
DSD Components
SSTSRTD_PROD_M
Local Tables
Codelists
SSTSRTD_IND_M
Annotations
Local Columns
DataFlows
SSTSRTD_TOVT_Q
SQL
SDMX Infrastructure
SDMX Queries
SDMX Output 4

5. Databases in practice Relational DDB issues
Structural
Many inherently different storage schemes
Content
Transcoding
Date and time formats (text, datetime)
Fragmented information in multiple columns
Other data in the same tables not related to requested transmission.

6. Databases in practice Requirements
Advanced Mapping
Capable of handling many relational DDB storage schemes
Data retrieval
Using the performed mappings
SDMX-ML generation

7. Storage Scheme A (simple)
DSD
Id: CENSUS_HUB
Version:
Agency: Eurostat
Dimensions:
AGE
CAS
GEO
SEX
OBS_VALUE
AGE
CAS
GEO
SEX
OBS_VALUE
001 GR
100
002
200
TOT
300 IT
150
250
400 7

8. Storage Scheme B DSD Id: CENSUS_HUB Version: 1.1 Agency: Eurostat
Dimensions:
AGE
CAS
GEO
SEX
OBS_VALUE
AGE
CAS
GEO
001
002
TOT GR
100
200
300 IT
150
250
400 8

9. Storage Scheme C “Master” Table “Slave” Tables FR_006_TABLE
DSD
Id: CENSUS_HUB
Version:
Agency: Eurostat
Dimensions:
AGE
CAS
GEO
SEX
OBS_VALUE
“Master” Table
“Slave” Tables
AGE
SEX
OBS_VALUE …
CAS
GEO
TABLE
001 IT
IT_001_TABLE GR
GR_001_TABLE …
AGE
SEX
OBS_VALUE …
AGE
SEX
OBS_VALUE
001
100
002
200
TOT
300
AGE
SEX
OBS_VALUE
001
150
002
250
TOT
400
FR_006_TABLE
GB_002_TABLE
IT_001_TABLE
GR_001_TABLE 9

10. Storage Scheme D “Secondary” Table “Primary” Table 10 DSD
Id: CENSUS_HUB
Version:
Agency: Eurostat
Dimensions:
AGE
CAS
GEO
SEX
OBS_VALUE
“Secondary” Table
“Primary” Table
AGE
CAS
GEO Id
001 GR
00001 IT
00002 … Id
SEX
OBS_VALUE
00001
001
100
002
200
TOT
300
00002
150
250
400 … 10

11. Storage Scheme UNION Data for Italy Data for Spain
CAS
GEO
SEX
OBS_VALUE
001 GR
100
002
200
TOT
300 IT
150
250
400
Data for Spain
DSD
Id: CENSUS_HUB
Version:
Agency: Eurostat
Dimensions:
AGE
CAS
GEO
SEX
OBS_VALUE
AGE
CAS
GEO
SEX
OBS_VALUE
001 GR
100
002
200
TOT
300 IT
150
250
400
Area dimension is missing
in both tables 11

12. Combination of Storages Schemes
“Slave” Tables
“Primary” Table
“Secondary”, “Master” Table
SEX
OBS_VALUE …
AGE Id
001
00001
002
00002 … Id
CAS
GEO
TABLE
00001
001 GR
GR_001_001_TABLE
00002 IT
IT_001_001_TABLE …
SEX
OBS_VALUE …
SEX
OBS_VALUE
001
100
002
200
TOT
300
SEX
OBS_VALUE
001
150
002
250
TOT
400
FR_001_003_TABLE
GB_002_001_TABLE
IT_001_001_TABLE
GR_001_001_TABLE 12

13. Problems Infinite storage schemes (chaotic) – custom queries
Transcoding issues
Date-time issues
Level of attributes – replication required
Performance issues
External programs: Limited permissions on DDBs

14. Mapping Assistant application
Facilitate mapping between the structural metadata of the DSD and the one of the Dissemination Database.
MSDB is used by others tools.
NSI Web Service
Test Client
i.e. Data Retriever API
The Mapping Assistant is meant to facilitate the mapping between the structural metadata provided by the DSD and the respective ones that reside in the Dissemination database of the NSI operational environment.
The Mapping Store DB created and managed by the Mapping Assistant can then be used by other tools, such as the Data Retriever.

15. SDMX-RI Components MSDB Store DDB Firewall Firewall Firewall DB Server
DB Server
MSDB Store
Workstation
App Server
Mapping
Assistant
NSI-WS
External Client
Test Client
NSI-Client
DB Server
DDB
Internal Network
Secure DMZ
DMZ
Internet

16. Mapping Store DB Used only by the SDMX-RI software. Contains:
Accessed and managed only by SDMX-RI software.
Contains:
SDMX artifacts.
Data flow mappings, transcodings.

17. MSDB initial setup Creation of Mapping Store DB objects.
Tables, sequences, stored-procedures.
Performed by the Mapping Assistant.
Performed only once.

18. MSDB Initial Setup Connection Parameters
One database has to be selected to store the mapping store.
The first thing we have to do is to create the data base connexion to this database.

19. MSDB Initial Setup Success

20. MSDB Initial Setup Tables Created

21. Import SDMX structure files
SDMX artifacts are imported from SDMX-ML files.
Category schemes and categories.
Code lists.
DSDs.
Data flows.
Information on SDMX artifacts is stored in the MSDB.

22. Import SDMX Structure Files

23. After Importing SDMX Structure Files

24. Configuring mappings Create DDB connection. Create data set.
One DDB connection can be used for multiple data sets and mappings.
Create data set.
A data set is similar to a “view” on a DDB table or tables.
Create mapping set.
Maps the fields of a data set to the components of a DSD.
Bridge between the relational DB world and the DSDs of the SDMX world.

25. DDB Connections

26. Creating a Data Set (1)

27. Creating a Data Set (2)

28. Creating a Data Set (3) Preview Query

29. Creating a Mapping Set (1)

30. Creating a Mapping Set (2)

31. Creating a Mapping Set (3) Component with Fixed Value

32. Creating a Mapping Set (4) Check Mapped Data

33. Creating a Mapping Set (6) Check Mapped Data again

34. 09 - Mapping Assistant