Eurostat November 2015 Eurostat Unit B3 – IT and standards for data and metadata exchange Jean-Francois LEBLANC Christian SEBASTIAN SDMX IT Tools SDMX Registry

Eurostat Table of Contents 1.Objectives 2.In a nutshell 1.Content of the SDMX registries 2.Availability 3.Basic Principles 4.Relations 5.In which registry? 3.SDMX Global Registry 1.Availability 2.How to use it? 2

Eurostat Table of Contents 4.SDMX Eurostat Registry 1.Availability 2.How to use it? 3.Architecture 4.Entity-Relationship model 5.Interface 5.Hands-on exercise 3

Eurostat 1. Objectives What are the different SDMX registries What can I find in them What are their principles Who should use them How to use them What is their architecture 4

Eurostat 2. In a nutshell General features of SDMX registries Central repositories Containing all artefacts needed for exchanging statistical information using SDMX Publicly accessible to anyone without constraints Advanced features of SDMX registries Subscription to specified elements Display of linked structures Versioning of artefacts implying strong dependencies 5

Eurostat 2.1 Content of the SDMX registries DSDs and their related artefacts Concept schemes Code lists Data flows etc. Metadata Structure Definitions Various other structures Link to related data 6

Eurostat 2.2 Availability Accessible to Production side: any authorized person or organization involved in SDMX data exchanges Dissemination side: any end-user of statistical data Publicly available information (exceptions may apply) No need to log in to view or to download information Web services 7

Eurostat 2.3 Basic Principles Each registry stores its own material Global DSDs in SDMX Global Registry ESS DSDs in Euro-SDMX Registry UN DSDs in UN SDMX Browser, etc. Each registry links to the other registries Information stored once (easier to manage) Users are informed of all potential information sources 8

Eurostat 2.4 Relations Global Registry Euro SDMX Registry UN SDMX Browser ECB SDMX Registry Etc. National SDMX Registry 1 National SDMX Registry 2 National SDMX Registry 3 9

Eurostat 2.5 In which registry? 10

Eurostat 3. SDMX Global Registry Launched in 2011 by the SDMX Sponsoring institutions To make SDMX artefacts public and centrally available, which are used and maintained by national and international organisations Designed to be developed into a global portal for SDMX structural data and metadata, providing: how statistical information is structured where the related data can be accessed 11

Eurostat 3.1 Availability Any national or international agency that wants to use SDMX standards for data management or for exchanging data with other organisations using SDMX standard Accessible to any end-user of statistical data 12

Eurostat 3.2 How to use it? All information published in the SGR is publicly available and freely viewable and downloadable. No need to log in to view or to download information - SDMX Global Registry Content.SDMX Global Registry Content Users can view the full content of the Global Registry, see how the various objects are interlinked and use the SDMX Global Registry Technical viewer.SDMX Global Registry Technical viewer Help is available in the Global Registry WikiGlobal Registry Wiki 13

Eurostat Home page 14

Eurostat Navigate the content of the registry Search for an item that contains the text entered Technical view of SDMX objects 15 Registry Content

Eurostat 4. SDMX Eurostat Registry Eurostat's implementation of the SDMX Registry specifications as published by the SDMX initiative sdmx.org. sdmx.org Based on SDMX 2.1 (as published on April 2011) Also capable of importing and exporting SDMX 2.0 artefacts. Allows browsing, searching, editing and subscribing to artefacts Advanced access control mechanism for distributed maintenance of artefacts controlling also their visibility. 16

Eurostat 4.1 Availability Any Domain manager in Eurostat Any end-user of statistical data can access the PUBLIC data structures posted on the SDMX Eurostat Registry (SER) Access to artefacts can be controlled per domain to limit the visibility of those which are private or under construction, thus not to be disclosed 17

Eurostat 4.2 How to use it? Some information published in the SER is publicly available and freely viewable and downloadable Some information published in the SER is private No need to log in to view or to download information - SDMX Eurostat Registry Content Users can view the public content of the Registry, see how the various objects are interlinked Contextual help is available online 18

Eurostat Most recent items Access to the content of the Registry: text search 19 Home page Access to the content of the Registry by type

Eurostat 20 Actions

Eurostat Quick view 2. Detailed view 3. Download

Eurostat Quick view 2. Detailed view 3. Download

Eurostat Quick view 2. Detailed view 3. Download

Eurostat Quick view 2. Detailed view 3. Download

Eurostat 4.3 Architecture Web Service All requests to the database are handled by the WS Request-Response Interface pattern Used by the GUI using SOAP Used by other applications, e.g. o SODI-WS o SDMX Converter Database Schema according to SDMX – IM 25

Eurostat 4.4 Entity-Relationship model SDMX Information Model -> ER model ER does not support all relationships of class diagram Inheritance Many to many relationship No one-to-one mapping of classes and tables Query performance (e.g. number of joins) Query simplicity for application development Maintainability and table size Enforcing data constraints 26

Eurostat Request/Response pattern J2EE application Model-View-Controller (MVC approach) Struts framework Interface SubmitStructureRequest - SubmitStructureResponse Insert, update, delete DSDs, Code Lists, Concept Schemes, Dataflows, etc. QueryStructureRequest - QueryStructureResponse Query DSDs, Code Lists, Concept Schemes SubmitProvisioningRequest - SubmitProvisioningResponse Insert, update, delete provision agreements QueryProvisioningRequest - QueryProvisioningResponse SubmitSubscriptionRequest - SubmitSubscriptionResponse SubmitRegistrationRequest - SubmitRegistrationResponse QueryRegistrationRequest - QueryRegistrationResponse NotifyRegistryEvent

Eurostat 5. SDMX Registry Hands-on exercise How to download DSDs 28

Eurostat 29 SDMX Registry