1. Eurostat 4. SDMX: Main objects for data exchange 1 Raynald Palmieri Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015

2. Eurostat The SDMX Components 2  Describe statistics in a standard way  Objects and their relationships  Data Structure Definition (DSD), Concepts, Code List  Central management and standard access  SDMX Registry, SDMX Web Services  Cross Domain Concepts  Cross Domain Code Lists  Statistical Domains  Metadata Common Vocabulary  Push  Provider generates and sends file to receiver  Pull  Provider opens web service to data  Receiver downloads regularly  Hub  Special case of pull: receiver downloads on end user request

3. Eurostat Who? What? When? Who? Where? How? What? Describing the data exchange

4. Eurostat Dataflows - classification 4 Sub categories Statistical Tables = data flows Category Tourism

5. Eurostat SDMX Implementation steps 5 DSD sharing Dataflows Concepts & Code lists SDMX Data Structure Definition

6. Eurostat SDMX Implementation steps 6 Provision agreement Dataflows Data Structure Data Provider? Dataflows Table 1 Table 2 Table 3 Definition of flows Definition of table structures Data Structure

7. Eurostat Dataflows - classification 7 Tourism Capacity Occupancy Night_Spent Arrival_of_ residents Arrival_of_ residents Occupancy_ rate Occupancy_ rate DataflowsCategories

8. Eurostat Concepts & Codelists : Tourism Example What do we want to exchange? Statistical tables 8

9. Eurostat Preparation phase SDMX Implementation steps 9 DSD sharing Dataflows Concepts & Code lists SDMX Data Structure Definition

10. Eurostat 2529 Tourism establishments Italy Annual data Number Model of the statistical table

11. Eurostat Sources Existing data set tables From website From applications Data Collection Instruments Questionnaires/Excel spreadsheets Handbooks, User Guides Database Tables Existing Data Structure Definitions From other organisations Legislation/Regulation Identify the Concepts A concept is a unit of knowledge created by a unique combination of characteristics (SDMX Information Model) Model of the statistical table: What do we need to do first?

12. Eurostat TIME COUNTRY FREQUENCY TOURISM_ACTIVITY OBS_VALUE E UNIT TOURISM_INDICATOR P Identifying the concepts OBS_STATUS

13. Eurostat Concept Scheme

14. Eurostat Identify/Define Code Lists Purpose of a Code List Constrains the value domain of concepts when used in a structure like a data structure definition Defines a shortened language independent representation of the values Gives semantic meaning to the values, possibly in multiple languages Agreeing on harmonised code lists is an important aspect of defining a data structure definition

15. Eurostat Concepts & Codelists : Tourism Example Partial code lists can also be exchanged (v2.1). The content of the partial code list is specified in a Constraint. Code list is maintainable SDMX container. Each code is defined uniquely by an ID, a maintenance agency, and a version. The name can be provided in several languages. 15 SDMX Code List

16. Eurostat Exercise: Deriving a concept scheme from a table Exercice

17. Eurostat Deriving a concept scheme from a table Proposed solution

18. Eurostat Data Set Structure Computers need to know the structure of data in terms of: Dimensionality Additional metadata Measures (Observation) Concepts Valid content Code Lists Non coded format (integer, date, text)

19. Eurostat Concepts play roles in a Data Structure Comprises –Concepts that identify the observation value –Concepts that add additional metadata about the observation value (as a value or the context of the value) –Concept that is the observation value –Any of these may be coded text date/time number etc. Dimensions Attributes Measure Representation

20. Eurostat TIME COUNTRY FREQUENCY TOURISM_ACTIVITY OBS_VALUE P E OBS_STATUS DIMENSIONSATTRIBUTESMEASURES UNIT TOURISM_INDICATOR DERIVING A DATA STRUCTURE FROM A TABLE

21. Eurostat DATA STRUCTURE DEFINITION

22. Eurostat DATA STRUCTURE DEFINITION - Summary DSD Concept Scheme Code lists Reference

23. Eurostat DATA STRUCTURE DEFINITION - Design Data Structure Wizard Java desktop application Graphical Interface For DSD designers Maintenance of SDMX v2.0/2.1 data and meta data structures Web service to query/submit SDMX registries

24. Eurostat Publishing DSDs: SDMX Registry Graphical User Interface Graphical User Interface Web service

25. Eurostat Exercise: Consult a DSD URL Registry ( Test purpose): https://webgate.acceptance.ec.europa.eu/sdmxregistry/ DSD: WASTE_GENER

26. Eurostat Exercise: Browse the different objects of the DSD Codelists: CL_FREQ CL_GEO_EUCCEFTA CL_WASTE CL_HAZARD CL_NACE_R2_WASTE Concept Scheme: CS_WASTE DSD: WASTE_GENER

27. Eurostat SDMX Implementation steps 27 DSD sharing Dataflows Concepts & Code lists SDMX Data Structure Definition

28. Eurostat DSD Sharing: Tourism Example 28

29. Eurostat How to achieve DSD sharing? Use of Constraints The Constraint can define one or both of: the Codes in a Code List that are applicable Ex: (A, M, W, Q) -> (A) the list of series keys that are applicable Can be used to constrain the DSD for which a sub set of the DSD content is meaningful. Constraints are usually linked to the dataflows or the provision agreements. 29 FREQCOUNTRYTOURISM _INDICATOR TOURISM _ACTIVITY AITA003B100

30. Eurostat Constraints – Example DSD_TOUR_CAP_XS DSD_TOUR_DEM_XS 30

31. Eurostat SDMX Dataset P E DSD Define the structure Dataset = XML file describing the table content according to the DSD.

32. Eurostat Syntaxes for SDMX datasets Based on a common Information Model SDMX-EDI (GESMES/TS) EDIFACT syntax Time-series oriented – One format for Data Sets SDMX-ML XML syntax Different formats for Data Sets Easier validation (XML based)

33. Eurostat Equivalent formats Generic SDMX-ML Cross-sectional SDMX-ML Compact SDMX-ML Can be expanded to other formats (e.g. CSV, GESMES) Based on the same IM SDMX-ML formats Conversions

34. Eurostat SDMX data common header

35. Eurostat SDMX 2.0 vs 2.1

36. Eurostat Equivalent representations for reporting Datasets Version 2.0Version 2.1 4 data messages, each with a distinct format. GenericData CrossSectional Data Compact Data UtilityData Therefore, there are now 4 data messages which are based on two general formats: GenericData GenericTimeSeriesData StructureSpecificData StructureSpecificTimeSeriesData Phased out SDMX-ML formats

37. Eurostat Data structure Definition (DSD) Support for non-time-series data structures Measure Dimension DSD Code lists Dimensions And Measure dimension Dimensions And Measure dimension Attributes Measures Concepts DSD Version 2.0Version 2.1 Measure Dimension Dimensions Attributes Primary Measure Concepts Concept Scheme Code lists Concept role explicit element

38. Eurostat The same Constraint can be “used” to constrain multiple objects Constraint Version 2.0 Version 2.1 Dataflow Provision agreement Constraint Registry Constraint Dataflow Provision agreement DSD Constraint is embedded in the object it constrains Constraint is only available for use in a Registry context Constraint is independently maintained

39. Eurostat Code List Common Code list Common Code list Constraint 1 Partial DSD Constraint 2 Version 2.1

40. Eurostat Questions