1. 4. SDMX: Main objects for data exchange
Raynald Palmieri Eurostat Unit B5: “Central data and metadata services”
SDMX Basics course, March 2016

2. The SDMX Components Describe statistics in a standard way
Objects and their relationships
Data Structure Definition (DSD), Concepts, Code List
Central management and standard access
SDMX Registry, SDMX Web Services
Cross Domain Concepts
Cross Domain Code Lists
Statistical Domains
Metadata Common Vocabulary
Push
Provider generates and sends file to receiver
Pull
Provider opens web service to data
Receiver downloads regularly
Hub
Special case of pull: receiver downloads on end user request 2 2

3. Describing the data exchange
Who?
Who?
When?
How?
Where?
What?
What?
To exchange statistical data in a standard way, we need to collect information and to define a model that will convey this information.
We will analyse the different type of information and we will see how the SDMX information model is built to handle this information.
Imagine, that Italy wishes to send a statistical table and its corresponding quality report.
Take a simplified table: three tourism topics over four periods. 3 2

4. Dataflows - classification
Category Tourism
Statistical Tables = data flows
Sub categories

5. SDMX Implementation steps
Dataflows
Concepts & Code lists
DSD sharing
SDMX Data Structure Definition

6. Dataflows - classification
Categories
Dataflows
Tourism
Capacity
Occupancy
Night_Spent
Arrival_of_
residents
Occupancy_
rate

7. Concepts & Codelists : Tourism Example
What do we want to exchange?
Statistical tables

8. Preparation phase SDMX Implementation steps
Dataflows
Concepts & Code lists
DSD sharing
SDMX Data Structure Definition

9. Model of the statistical table
Number
Tourism establishments
Italy
Annual data
Any statistical table is a set of observations or measures.
But the observation itself means nothing.
We need metadata to identify the figure such as the Tourism establishments, the unit, the country, the frequency, the time…
This information will be called structural metadata.
We can now introduce our fist SDMX object: the Concept.
The concept is a label describing the data.
It has an ID, a name and description in several languages.
2529

10. Model of the statistical table: What do we need to do first?
Identify the Concepts
A concept is a unit of knowledge created by a unique combination of characteristics (SDMX Information Model)
Sources
Existing data set tables
From website
From applications
Data Collection Instruments
Questionnaires/Excel spreadsheets
Handbooks, User Guides
Database Tables
Existing Data Structure Definitions
From other organisations
Legislation/Regulation

11. Identifying the concepts
FREQUENCY
COUNTRY
TOURISM_ACTIVITY
TOURISM_INDICATOR
UNIT
OBS_VALUE
TIME E
OBS_STATUS P
In our example, we can identify the following statistical concepts to describe the information in the statistical table:
Frequency (in this case annual)
Country
Tourism topic
Time
The observation value, which is the actual single figure
The observation status, defined by footnotes like “estimated” or “provisional”
At the same time we identify the statistical concepts, we also define the format of the information (Alphanumeric, numeric etc.)
Then we regroup all the statistical concepts into an SDMX container called Concept scheme.

12. Concept Scheme
As all SDMX object the Concept Scheme has an ID, a name and a description in several languages.
We will see later that the Concept Scheme is a maintainable object which can have many versions.
The ID and the name are mandatory in the SMDX information model, the description is optional.
The name should be provided at least in one language, generally in English.
In the Concept Scheme, we can also define a format for the statistical concepts to be used later for validation purpose,
But this is also optional.
Finally, when statistical concepts correspond to coded values, we define code lists objects.
For example, concept FREQUENCY identifies fixed values such as A, M, Q etc.. We define a SDMX object Code lists .

13. Identify/Define Code Lists
Purpose of a Code List
Constrains the value domain of concepts when used in a structure like a data structure definition
Defines a shortened language independent representation of the values
Gives semantic meaning to the values, possibly in multiple languages
Agreeing on harmonised code lists is an important aspect of defining a data structure definition
Ask the question
Reveals first set of minor bullets on mouse click
Then reveals the second major bullet

14. Concepts & Codelists : Tourism Example
SDMX Code List
Code list is maintainable SDMX container.
Each code is defined uniquely by an ID, a maintenance agency, and a version. The name can be provided in several languages.
Partial code lists can also be exchanged (v2.1).
The content of the partial code list is specified in a Constraint.

15. Exercice Exercise: Deriving a concept scheme from a table
On this slide is represented a statistical table from WASTE
Identify the statistical concepts that desrcibe the structure of the table
Imagine the possible format and code list for each concept
Fill up the Concept Scheme table
Launch the DSW
Create the code lists
Create the concept scheme

16. Proposed solution Deriving a concept scheme from a table
This is a proposal of Concept Scheme.

17. Data Set Structure
Computers need to know the structure of data in terms of:
Dimensionality
Additional metadata
Measures (Observation)
Concepts
Valid content
Code Lists
Non coded format (integer, date, text)

18. Concepts play roles in a Data Structure
Comprises
Concepts that identify the observation value
Concepts that add additional metadata about the observation value (as a value or the context of the value)
Concept that is the observation value
Any of these may be
coded
text
date/time
number
etc.
Dimensions
Attributes
Measure
Representation

19. DERIVING A DATA STRUCTURE FROM A TABLE
FREQUENCY
COUNTRY
TOURISM_ACTIVITY
TOURISM_INDICATOR
UNIT
OBS_VALUE
TIME E
OBS_STATUS P
DIMENSIONS
ATTRIBUTES
MEASURES
Until now we have defined the statistical concepts that identify the information contained in our statistical table.
But to have a full picture of the table we need to assign roles to our concepts.
Indeed, if we think this table as a slice of a statistical cube, each observation has coordinates that identify it uniquely.
In our exemple: Frequency, Time Period, Country, Tourism Indicator and Tourism Activity are required to identify the statistics: they act as dimensions.
The observation value, the actual figure, is what we call measure.
The ‘observation status’ and "Unit" further explain the figures as an attribute. We can see that they are displayed at different levels of the table. This depends on whether the attribute explains only a single figure, a group of figures or the whole table. In our example the "unit" explains the whole table: it is attached at dataset level.
The "Observation status" explains only single figures: it is attached at observation level.
In SDMX it is also possible to attach attributes to a group of observations. This is not the case in our example.
The result of our small exercise is a formal definition of the corresponding Data Structure Definition, also called DSD, with all the structural metadata of the table.

20. DATA STRUCTURE DEFINITION
As all SDMX objects, the DSD has an ID, a name and a description in several languages.
We will see later that the DSD is a maintainable object which can have many versions.
The ID and the name are mandatory in the SMDX information model, the description is optional.
The name should be provided at least in one language, generally in English.
In the DSD, we can also define locally a format and a code list for the statistical concepts to be used later for validation purpose,
But this is also optional.
The SDMX information model is flexible enough to define local representations to be used only in a specific DSD that shares a common Concept scheme with other DSDs.

21. DATA STRUCTURE DEFINITION - Summary
Reference
DSD
Reference
Concept Scheme
Reference
Code lists

22. DATA STRUCTURE DEFINITION - Design
Java desktop application
Graphical Interface
For DSD designers
Maintenance of SDMX v2.0/2.1 data and meta data structures
Web service to query/submit SDMX registries
Data Structure Wizard

23. SDMX Registry: Designing & Publishing DSDs
Graphical
User
Interface
Web service
Now we'll turn to another important module of the standard: the IT architecture.
The IT architecture involves:
The standard formats for the data exchange.
Different architectures for data exchange.
And the SDMX registry. 23

24. Exercise: Consult a DSD
URL Registry ( Test purpose):
DSD: WASTE_GENER
Now we have defined our DSD we will create the corresponding SDMX file with the Data Strucure Wizard tool.
Launch the DSW
Select DSD in the left tree, click on +
Fill the DSD information as mentionned in the slide (ID, description…)
Select the content tab and choose « Observation value »
Select the subtab « dimensions » and add the dimensions
Note: Dimension can be assigned roles: isFrequency, IsTime, IsMeasure, simple dimension.
6) Select subtab “Attributes” and add the Unit attribute.

25. Exercise: Browse the different objects of the DSD
Codelists:
CL_FREQ
CL_GEO_EUCCEFTA
CL_WASTE
CL_HAZARD
CL_NACE_R2_WASTE
Concept Scheme:
CS_WASTE
Now we have defined our DSD we will create the corresponding SDMX file with the Data Strucure Wizard tool.
Launch the DSW
Select DSD in the left tree, click on +
Fill the DSD information as mentionned in the slide (ID, description…)
Select the content tab and choose « Observation value »
Select the subtab « dimensions » and add the dimensions
Note: Dimension can be assigned roles: isFrequency, IsTime, IsMeasure, simple dimension.
6) Select subtab “Attributes” and add the Unit attribute.
DSD:
WASTE_GENER

26. SDMX Implementation steps
Dataflows
Concepts & Code lists
DSD sharing
SDMX Data Structure Definition

27. DSD Sharing: Tourism Example

28. How to achieve DSD sharing? Use of Constraints
The Constraint can define one or both of:
• the Codes in a Code List that are applicable
Ex: (A, M, W, Q)  (A)
• the list of series keys that are applicable
Can be used to constrain the DSD for which a sub set of the DSD content is meaningful. Constraints are usually linked to the dataflows or the provision agreements.
FREQ
COUNTRY
TOURISM
_INDICATOR
_ACTIVITY A IT
A003
B100 28 28

29. Constraints – Example
DSD_TOUR_CAP_XS
DSD_TOUR_DEM_XS 29 29

30. SDMX Dataset
Define the structure
DSD
Dataset = XML file describing the table content according to the DSD. E P
The data structure definition being defined, let’s see how the statistical data are organised in the file to be exchanged, what is called Dataset in SDMX.
In the related SDMX-ML structure Specific Time series data file we can highlight the different structural constructs to which attributes can be related:
The Data set, where for this example Attribute UNIT is related
The Series, where for this example no Attribute is related
And finally the Observation level, where the Attribute Observation status is linked to.
Each information in the table is placed into the dataset with its corresponding concept name:
UNIT
FREQ
COUNTRY
TOURISM INDICATOR
TOURISM ACTIVITY
TIME,
OBSERVATION VALUE and OBSERVATION STATUS 30

31. Syntaxes for SDMX datasets
Based on a common Information Model
SDMX-EDI (GESMES/TS)
EDIFACT syntax
Time-series oriented – One format for Data Sets
SDMX-ML
XML syntax
Different formats for Data Sets
Easier validation (XML based)
From the same information model two syntaxes can be used:
under some conditions EDIFACT (SDMX-EDI or GESMES/TS)
XML (SDMX-ML)
The EDIFACT syntax, SDMX-EDI which is also known as GESMES/TS can be used only with Time series data sets and if the Data structure definition contains the dimensions required by the GESMES/TS standard. There is only one format of SDMX-EDI.
The SDMX-ML uses the XML syntax making it much easier to generate and to validate. SDMX-ML has four different formats for data sets.
Last there are tools that can convert between formats and under some conditions between syntaxes in order to obtain the desired format. 31

32. SDMX-ML formats Conversions
Equivalent formats
Generic SDMX-ML
Cross-sectional SDMX-ML
Compact SDMX-ML
Based on the same IM
Equivalent SDMX-ML formats can be converted between them. Equivalent SDMX-ML formats are those that are based on the same Information Model. Exceptions are the Cross-Sectional DSD that do not contain a time dimension which is required by the other formats.
Conversion is possible also between SDMX-EDI and SDMX-ML formats, provided it is a time series data set and all dimensions and attributes required by SDMX-EDI are present in the Information model. Besides, it is possible as well to convert between other formats such as CSV.
Can be expanded to other formats (e.g. CSV, GESMES) 32

33. SDMX data common header
A common part is the <Header>, which has the same basic structure for all message types.
An example is shown below providing typical information on the dataset:
The information to be included in the header is inserted in the SDMX-ML message as a text file provided as input of the SDMX Converter too.
An example of the content of this file “header.prop” is shown in the next figure. In the left column are listed the different elements, their corresponding values are presented in the right column. Some elements may be let blank except ID, Test, Prepared, and Sender. 33

34. SDMX 2.0 vs 2.1

35. Equivalent representations for reporting Datasets
SDMX-ML formats
Equivalent representations for reporting Datasets
Version 2.0
Version 2.1
4 data messages, each with a distinct format.
Therefore, there are now 4 data messages which are based on two general formats:
GenericData
• GenericData
GenericTimeSeriesData
CrossSectional Data
Compact Data
• StructureSpecificData
StructureSpecificTimeSeriesData
UtilityData
SDMX offers four different representations for reporting datasets according to the final use.
These are equivalent in a way, but has different objectives.
In most cases conversion are possible from one representation to others.
Generic message:
Conveys data in a form independent of a data structure definition.
It is designed for data provision on websites and in any scenario
where applications receiving the data may not have detailed understanding of
the data set's structure before they obtain the data set itself.
Generic messages and offers no validation.
Compact message:
Exchange of large data sets in a data structure definition-dependent form.
Requires an XML schema specific to the DSD.
Utility message:
For schema-based functions, such as advanced validation,
in a data structure definition-dependent form.
Cross-Sectional message:
Exchange of many observation types in a data structure definition-dependent form.
Used for non time-series data.
SDMX messages are deeply described in a devoted student book available by the end of this year.
Phased out 35

36. Data structure Definition (DSD)
Version 2.0
Version 2.1
Support for non-time-series data structures
Measure Dimension
DSD
DSD
Concepts
Concepts
Measures
Code lists
Primary Measure
Attributes
Code lists
Attributes
Code lists
Dimensions
And
Measure dimension
Dimensions
Code lists
Code lists
Measure Dimension
Concept Scheme
Concept role
explicit element

37. Constraint Constraint is only available for use in a Registry context
Version 2.0
Version 2.1
Dataflow
Dataflow
Registry
Constraint
Constraint
Provision
agreement
Furthermore, the Constraint is structural metadata in version 2.1 and is contained in the Structure schemas.
Note that a Constraint always constrains the allowable or actual content of structures
or data sets that relate to a DSD.
Rules have been specified to define how
Constraints are inherited or “cascaded”
when constraints are specified for one or
more objects that are related to the same DSD (e.g. DSD and Dataflow).
Provision agreement
DSD
Constraint
Constraint is embedded in the object it constrains
The same Constraint can be “used” to constrain multiple objects
Constraint is independently maintained

38. Thank you for your attention!
Questions