1. Developing SDMX artefacts for data exchange, sharing and dissemination
Francesco Rizzo
Istat
ENP-South Training Course
“Information standards for describing, processing and disseminating data and metadata”
Rome, June 2018

2. Data warehousing cases
SDMX use cases
Data warehousing cases
SDMX as a model for the structure of a data warehouse or metadata repository
for extraction, transformation, and load of data
Reporting (to the International Organisations) cases
“push” reporting format for data and metadata (reporter pushes data to collector)
“pull” reporting format for data and metadata (collector pulls data from reporter)
Discovery and visualization (GUI browsing and machine-actionable)
to drive website presentation of data and metadata
as a queryable data source
for standardized file downloads
Validate data using XML schemes and VTL

3. Major SDMX constructs for data modelling
Data Structure Definition (DSD)
Concept Scheme
Code List
Data Flow
Category Scheme

4. Modelling a statistical cube for data warehousing

5. Turnover index base 2010=100. Monthly data. Italy
Source: National Statistical Institute of Italy

6. Statistical concepts and Codelists
Concept id
Concept name
Codelist id
Codelist name
FREQ
Frequency
CL_FREQ
MARKET
Reference Market
CL_MARKET
Market
REF_AREA
Reference Area
CL_REF_AREA
ADJUSTEMENT
Adjustment
CL_ADJUSTEMENT
INDICATOR
Indicator
CL_ INDICATOR
ACTIVITY
Economic Activity
CL_ ACTIVITY
BASE_YEAR
Base year
CL_ BASE_YEAR
TIME_PERIOD
Time period
OBS_VALUE
Observed value
SOURCE
Source
CL_SOURCE

7. SDMX MetaManager: a tool for data modelling

8. SDMX Registry GUI – Main characteristics
Free and Open source tool
Based on the SdmxSource.NET API (release by Eurostat)
Interaction with any standard-compliant SDMX Web Service
Maintenance of SDMX v2.1 artefacts
Authentication and authorization mechanism is required
Dissemination of SDMX v2.1 artefact
Import and export of artefacts from/to SDMX-ML v2.0 and SDMX v2.1 format
Download of structural metadata
SDMX v2.1, SDMX v2.0, CSV and .STAT
Import Codelists from CSV files
Order the items of the Codelists (if is final=“false”)
Create MIGs for Data Structure Definition artefacts

9. SDMX Istat Registry– first steps

10. Add a Code List
General information
Information for each item

11. Create a new Codelist with the following general information:
Add CL_FREQ Code List
Create a new Codelist with the following general information:
ID = CL_FREQ
Agency ID= IT1
Version=1.0
Name [en] = Frequency
Add the following items:

12. Add CL_INDICATOR Code List
Create a new Codelist with the following general information:
ID = CL_INDICATOR
Agency ID= IT1
Version=1.0
Name [en] = Indicator
Add the following items:

13. Add CL_MARKET Code List
Create a new Codelist with the following general information:
ID = CL_MARKET
Agency ID= IT1
Version=1.0
Name [en] = Market
Add the following items:

14. Import a Code Lists from an SDMX-ML file
Select the file CL_ACTIVITY+IT1+1.0.xml
Repeat the previous step with the following files:
CL_REF_AREA+IT1+1.0.xml
CL_SOURCE+IT1+1.0.xml

15. Import a Code List from a CSV file
Create a new Codelist with the following general information:
ID = CL_ADJUSTMENT; Agency ID= IT1; Version=1.0
Name [en] = Adjustment
Import the file [en] CL_ADJUSTMENT +IT1+1.0.csv

16. Import a Code List from a CSV file
Create a new Codelist with the following general information:
ID = CL_BASE_YEAR; Agency ID= IT1; Version=1.0
Name [en] = Adjustment
Import the file [en]CL_BASE_YEAR+IT1+1.0.csv

17. Add STS_SCHEME Conceptscheme
Create a new Conceptscheme with the following general information:
ID = STS_SCHEME
Agency ID= IT1
Version=1.0
Name [en] = STS Concepts
Add the following items:

18. Data Structure Definition – Message Implementation Guide (MIG)

19. Add the general part of the Data Structure Definitionid
STS
Name (en)
Short Term Statistics
version
1.0
agencyID
IT1
Before starting creating a new DSD, there must be at least one concept scheme with concepts defined in the application.
An existing DSD can be downloaded as MIG.

20. Add a primary measure
(1)
(2)
(3)

21. Add Dimensions (1/2)
(1)

22. Add Dimensions (2/2)
(2)
(3)
(4)
(5)

23. Add Attributes

24. Export a DSD and the related artefacts
(1)
(2)
(3)

25. APPENDIX

26. General Design Principles - Reuse of existing DSDs and Code Lists (1/4)
Identify existing DSDs and code lists (look inside the available registries and/or metadata repositories)
Global DSDs with maintenance agency “SDMX”
Global DSDs with SDMX sponsors organization(s) as maintenance agency
Other internationally agreed DSDs
DSDs used by other organizations
DSDs used by departments
If none of the available DSDs is appropriate, it still possible that existing concepts and/or code lists may be reused

27. General Design Principles - Reuse of existing DSDs and Code Lists (2/4)
Only if the required concepts and code lists do not exist at all, a completely new DSD has to be developed with new concepts and new code lists
SDMX COG code lists
ISO code lists
Code lists used by many SDMX sponsors organizations
Other internationally agreed code lists
Nationally agreed code lists
Organization-wide code lists
Departmental code lists

28. Contain irrelevant concepts
General Design Principles - Reuse of existing DSDs and Code Lists (3/4)
In case an existing DSD is close to but differs from what is needed, it may:
Contain irrelevant concepts
Define a new, reduced concept schema that includes only the relevant concepts and related code lists, and define a new DSD that uses the reduced concept scheme
Reuse concept scheme, code lists, and DSD, but add constraints to the data flow
Lack some required concepts
Create a new DSD by adapting the existing DSD (adding the concepts, dimensions/attributes, and code lists)
Use the concepts in different roles than required
Create a ned DSD. It can reuse the concept schema and the code lists, but specifies the concepts in the new DSD as dimensions or attribute as required

29. Deviate with respect to some of the code lists
General Design Principles - Reuse of existing DSDs and Code Lists (4/4)
….. In case an existing DSD is close to but differs from what is needed, it may:
Deviate with respect to some of the code lists
Only a subset of codes being relevant; deviating hierarchical structure; necessitating additional codes
Contain “pure” dimensions when “mixed” dimensions would make more sense or vice versa
If an existing DSD does not have the desired degree of dimension purity, it is necessary to further decompose and/or combine dimensions of the DSD