Eurostat 4. SDMX: Main objects for data exchange 1 Raynald Palmieri Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, October 2015

Eurostat The SDMX Components 2  Describe statistics in a standard way  Objects and their relationships  Data Structure Definition (DSD), Concepts, Code List  Central management and standard access  SDMX Registry, SDMX Web Services  Cross Domain Concepts  Cross Domain Code Lists  Statistical Domains  Metadata Common Vocabulary  Push  Provider generates and sends file to receiver  Pull  Provider opens web service to data  Receiver downloads regularly  Hub  Special case of pull: receiver downloads on end user request

Eurostat Who? What? When? Who? Where? How? What? Describing the data exchange

Eurostat Dataflows - classification 4 Sub categories Statistical Tables = data flows Category Tourism

Eurostat SDMX Implementation steps 5 DSD sharing Dataflows Concepts & Code lists SDMX Data Structure Definition

Eurostat SDMX Implementation steps 6 Provision agreement Dataflows Data Structure Data Provider? Dataflows Table 1 Table 2 Table 3 Definition of flows Definition of table structures Data Structure

Eurostat Dataflows - classification 7 Tourism Capacity Occupancy Night_Spent Arrival_of_ residents Arrival_of_ residents Occupancy_ rate Occupancy_ rate DataflowsCategories

Eurostat Concepts & Codelists : Tourism Example What do we want to exchange? Statistical tables 8

Eurostat Preparation phase SDMX Implementation steps 9 DSD sharing Dataflows Concepts & Code lists SDMX Data Structure Definition

Eurostat 2529 Tourism establishments Italy Annual data Number Model of the statistical table

Eurostat Sources Existing data set tables From website From applications Data Collection Instruments Questionnaires/Excel spreadsheets Handbooks, User Guides Database Tables Existing Data Structure Definitions From other organisations Legislation/Regulation Identify the Concepts A concept is a unit of knowledge created by a unique combination of characteristics (SDMX Information Model) Model of the statistical table: What do we need to do first?

Eurostat TIME COUNTRY FREQUENCY TOURISM_ACTIVITY OBS_VALUE E UNIT TOURISM_INDICATOR P Identifying the concepts OBS_STATUS

Eurostat Concept Scheme

Eurostat Identify/Define Code Lists Purpose of a Code List Constrains the value domain of concepts when used in a structure like a data structure definition Defines a shortened language independent representation of the values Gives semantic meaning to the values, possibly in multiple languages Agreeing on harmonised code lists is an important aspect of defining a data structure definition

Eurostat Concepts & Codelists : Tourism Example Partial code lists can also be exchanged (v2.1). The content of the partial code list is specified in a Constraint. Code list is maintainable SDMX container. Each code is defined uniquely by an ID, a maintenance agency, and a version. The name can be provided in several languages. 15 SDMX Code List

Eurostat Exercise: Deriving a concept scheme from a table Exercice

Eurostat Deriving a concept scheme from a table Proposed solution

Eurostat Data Set Structure Computers need to know the structure of data in terms of: Dimensionality Additional metadata Measures (Observation) Concepts Valid content Code Lists Non coded format (integer, date, text)

Eurostat Concepts play roles in a Data Structure Comprises –Concepts that identify the observation value –Concepts that add additional metadata about the observation value (as a value or the context of the value) –Concept that is the observation value –Any of these may be coded text date/time number etc. Dimensions Attributes Measure Representation

Eurostat TIME COUNTRY FREQUENCY TOURISM_ACTIVITY OBS_VALUE P E OBS_STATUS DIMENSIONSATTRIBUTESMEASURES UNIT TOURISM_INDICATOR DERIVING A DATA STRUCTURE FROM A TABLE

Eurostat DATA STRUCTURE DEFINITION

Eurostat DATA STRUCTURE DEFINITION - Summary DSD Concept Scheme Code lists Reference

Eurostat DATA STRUCTURE DEFINITION - Design Data Structure Wizard Java desktop application Graphical Interface For DSD designers Maintenance of SDMX v2.0/2.1 data and meta data structures Web service to query/submit SDMX registries

Eurostat Publishing DSDs: SDMX Registry Graphical User Interface Graphical User Interface Web service

Eurostat Exercise: Consult a DSD URL Registry ( Test purpose): DSD: WASTE_GENER

Eurostat Exercise: Browse the different objects of the DSD Codelists: CL_FREQ CL_GEO_EUCCEFTA CL_WASTE CL_HAZARD CL_NACE_R2_WASTE Concept Scheme: CS_WASTE DSD: WASTE_GENER

Eurostat SDMX Implementation steps 27 DSD sharing Dataflows Concepts & Code lists SDMX Data Structure Definition

Eurostat DSD Sharing: Tourism Example 28

Eurostat How to achieve DSD sharing? Use of Constraints The Constraint can define one or both of: the Codes in a Code List that are applicable Ex: (A, M, W, Q) -> (A) the list of series keys that are applicable Can be used to constrain the DSD for which a sub set of the DSD content is meaningful. Constraints are usually linked to the dataflows or the provision agreements. 29 FREQCOUNTRYTOURISM _INDICATOR TOURISM _ACTIVITY AITA003B100

Eurostat Constraints – Example DSD_TOUR_CAP_XS DSD_TOUR_DEM_XS 30

Eurostat SDMX Dataset P E DSD Define the structure Dataset = XML file describing the table content according to the DSD.

Eurostat Syntaxes for SDMX datasets Based on a common Information Model SDMX-EDI (GESMES/TS) EDIFACT syntax Time-series oriented – One format for Data Sets SDMX-ML XML syntax Different formats for Data Sets Easier validation (XML based)

Eurostat Equivalent formats Generic SDMX-ML Cross-sectional SDMX-ML Compact SDMX-ML Can be expanded to other formats (e.g. CSV, GESMES) Based on the same IM SDMX-ML formats Conversions

Eurostat SDMX data common header

Eurostat SDMX 2.0 vs 2.1

Eurostat Equivalent representations for reporting Datasets Version 2.0Version data messages, each with a distinct format. GenericData CrossSectional Data Compact Data UtilityData Therefore, there are now 4 data messages which are based on two general formats: GenericData GenericTimeSeriesData StructureSpecificData StructureSpecificTimeSeriesData Phased out SDMX-ML formats

Eurostat Data structure Definition (DSD) Support for non-time-series data structures Measure Dimension DSD Code lists Dimensions And Measure dimension Dimensions And Measure dimension Attributes Measures Concepts DSD Version 2.0Version 2.1 Measure Dimension Dimensions Attributes Primary Measure Concepts Concept Scheme Code lists Concept role explicit element

Eurostat The same Constraint can be “used” to constrain multiple objects Constraint Version 2.0 Version 2.1 Dataflow Provision agreement Constraint Registry Constraint Dataflow Provision agreement DSD Constraint is embedded in the object it constrains Constraint is only available for use in a Registry context Constraint is independently maintained

Eurostat Code List Common Code list Common Code list Constraint 1 Partial DSD Constraint 2 Version 2.1

Eurostat Questions