Developing SDMX artefacts for data exchange, sharing and dissemination Francesco Rizzo Istat ENP-South Training Course “Information standards for describing, processing and disseminating data and metadata” Rome, 19-22 June 2018

Data warehousing cases SDMX use cases Data warehousing cases SDMX as a model for the structure of a data warehouse or metadata repository for extraction, transformation, and load of data Reporting (to the International Organisations) cases “push” reporting format for data and metadata (reporter pushes data to collector) “pull” reporting format for data and metadata (collector pulls data from reporter) Discovery and visualization (GUI browsing and machine-actionable) to drive website presentation of data and metadata as a queryable data source for standardized file downloads Validate data using XML schemes and VTL

Major SDMX constructs for data modelling Data Structure Definition (DSD) Concept Scheme Code List Data Flow Category Scheme

Modelling a statistical cube for data warehousing

Turnover index base 2010=100. Monthly data. Italy Source: National Statistical Institute of Italy

Statistical concepts and Codelists Concept id Concept name Codelist id Codelist name FREQ Frequency CL_FREQ MARKET Reference Market CL_MARKET Market REF_AREA Reference Area CL_REF_AREA ADJUSTEMENT Adjustment CL_ADJUSTEMENT INDICATOR Indicator CL_ INDICATOR ACTIVITY Economic Activity CL_ ACTIVITY BASE_YEAR Base year CL_ BASE_YEAR TIME_PERIOD Time period   OBS_VALUE Observed value SOURCE Source CL_SOURCE

SDMX MetaManager: a tool for data modelling

SDMX Registry GUI – Main characteristics Free and Open source tool Based on the SdmxSource.NET API (release by Eurostat) Interaction with any standard-compliant SDMX Web Service Maintenance of SDMX v2.1 artefacts Authentication and authorization mechanism is required Dissemination of SDMX v2.1 artefact Import and export of artefacts from/to SDMX-ML v2.0 and SDMX v2.1 format Download of structural metadata SDMX v2.1, SDMX v2.0, CSV and .STAT Import Codelists from CSV files Order the items of the Codelists (if is final=“false”) Create MIGs for Data Structure Definition artefacts

SDMX Istat Registry– first steps http://localhost/Registry

Add a Code List General information Information for each item

Create a new Codelist with the following general information: Add CL_FREQ Code List Create a new Codelist with the following general information: ID = CL_FREQ Agency ID= IT1 Version=1.0 Name [en] = Frequency Add the following items:

Add CL_INDICATOR Code List Create a new Codelist with the following general information: ID = CL_INDICATOR Agency ID= IT1 Version=1.0 Name [en] = Indicator Add the following items:

Add CL_MARKET Code List Create a new Codelist with the following general information: ID = CL_MARKET Agency ID= IT1 Version=1.0 Name [en] = Market Add the following items:

Import a Code Lists from an SDMX-ML file Select the file CL_ACTIVITY+IT1+1.0.xml Repeat the previous step with the following files: CL_REF_AREA+IT1+1.0.xml CL_SOURCE+IT1+1.0.xml

Import a Code List from a CSV file Create a new Codelist with the following general information: ID = CL_ADJUSTMENT; Agency ID= IT1; Version=1.0 Name [en] = Adjustment Import the file [en] CL_ADJUSTMENT +IT1+1.0.csv

Import a Code List from a CSV file Create a new Codelist with the following general information: ID = CL_BASE_YEAR; Agency ID= IT1; Version=1.0 Name [en] = Adjustment Import the file [en]CL_BASE_YEAR+IT1+1.0.csv

Add STS_SCHEME Conceptscheme Create a new Conceptscheme with the following general information: ID = STS_SCHEME Agency ID= IT1 Version=1.0 Name [en] = STS Concepts Add the following items:

Data Structure Definition – Message Implementation Guide (MIG)

Add the general part of the Data Structure Definition id STS Name (en) Short Term Statistics version 1.0 agencyID IT1 Before starting creating a new DSD, there must be at least one concept scheme with concepts defined in the application. An existing DSD can be downloaded as MIG.

Add a primary measure (1) (2) (3)

Add Dimensions (1/2) (1)

Add Dimensions (2/2) (2) (3) (4) (5)

Add Attributes

Export a DSD and the related artefacts (1) (2) (3)

APPENDIX

General Design Principles - Reuse of existing DSDs and Code Lists (1/4) Identify existing DSDs and code lists (look inside the available registries and/or metadata repositories) Global DSDs with maintenance agency “SDMX” Global DSDs with SDMX sponsors organization(s) as maintenance agency Other internationally agreed DSDs DSDs used by other organizations DSDs used by departments If none of the available DSDs is appropriate, it still possible that existing concepts and/or code lists may be reused

General Design Principles - Reuse of existing DSDs and Code Lists (2/4) Only if the required concepts and code lists do not exist at all, a completely new DSD has to be developed with new concepts and new code lists SDMX COG code lists ISO code lists Code lists used by many SDMX sponsors organizations Other internationally agreed code lists Nationally agreed code lists Organization-wide code lists Departmental code lists

Contain irrelevant concepts General Design Principles - Reuse of existing DSDs and Code Lists (3/4) In case an existing DSD is close to but differs from what is needed, it may: Contain irrelevant concepts Define a new, reduced concept schema that includes only the relevant concepts and related code lists, and define a new DSD that uses the reduced concept scheme Reuse concept scheme, code lists, and DSD, but add constraints to the data flow Lack some required concepts Create a new DSD by adapting the existing DSD (adding the concepts, dimensions/attributes, and code lists) Use the concepts in different roles than required Create a ned DSD. It can reuse the concept schema and the code lists, but specifies the concepts in the new DSD as dimensions or attribute as required

Deviate with respect to some of the code lists General Design Principles - Reuse of existing DSDs and Code Lists (4/4) ….. In case an existing DSD is close to but differs from what is needed, it may: Deviate with respect to some of the code lists Only a subset of codes being relevant; deviating hierarchical structure; necessitating additional codes Contain “pure” dimensions when “mixed” dimensions would make more sense or vice versa If an existing DSD does not have the desired degree of dimension purity, it is necessary to further decompose and/or combine dimensions of the DSD