SDMX IT Tools SDMX Registry Jean-Francois LEBLANC Christian SEBASTIAN 29-31 May 2017 Eurostat, Unit B3 – IT solutions for statistical production

Table of Contents Objectives In a nutshell SDMX Global Registry Content of the SDMX registries Availability Basic Principles Relations In which registry? SDMX Global Registry How to use it?

Table of Contents SDMX Eurostat Registry Hands-on exercise Availability How to use it? Architecture Entity-Relationship model Interface Hands-on exercise

1. Objectives What are the different SDMX registries What can I find in them What are their principles Who should use them How to use them What is their architecture

2. In a nutshell General features of SDMX registries Central repositories Containing all artefacts needed for exchanging statistical information using SDMX Publicly accessible to anyone without constraints Advanced features of SDMX registries Subscription to specified elements Display of linked structures Versioning of artefacts implying strong dependencies

2.1 Content of the SDMX registries DSDs and their related artefacts Concept schemes Code lists Data flows etc. Metadata Structure Definitions Various other structures Link to related data

2.2 Availability Accessible to Production side: any authorized person or organization involved in SDMX data exchanges Dissemination side: any end-user of statistical data Publicly available information (exceptions may apply) No need to log in to view or to download information Web services

2.3 Basic Principles Each registry stores its own material Global DSDs in SDMX Global Registry ESS DSDs in Euro-SDMX Registry UN DSDs in UN SDMX Browser, etc. Each registry links to the other registries Information stored once (easier to manage) Users are informed of all potential information sources

2.4 Relations Global Registry Euro SDMX Registry UN SDMX Browser ECB SDMX Registry Etc. National SDMX Registry 1 National SDMX Registry 2 National SDMX Registry 3

2.5 In which registry?

3. SDMX Global Registry Launched in 2011 by the SDMX Sponsoring institutions To make SDMX artefacts public and centrally available, which are used and maintained by national and international organisations Designed to be developed into a global portal for SDMX structural data and metadata, providing: how statistical information is structured where the related data can be accessed

3.1 Availability Any national or international agency that wants to use SDMX standards for data management or for exchanging data with other organisations using SDMX standard Accessible to any end-user of statistical data

3.2 How to use it? All information published in the SGR is publicly available and freely viewable and downloadable. No need to log in to view or to download information - SDMX Global Registry Content. Users can view the full content of the Global Registry , see how the various objects are interlinked and use the SDMX Global Registry Technical viewer. Help is available in the Global Registry Wiki

Home page

Navigate the content of the registry Registry Content Navigate the content of the registry Search for an item that contains the text entered Technical view of SDMX objects

4. SDMX Eurostat Registry Eurostat's implementation of the SDMX Registry specifications as published by the SDMX initiative sdmx.org. Based on SDMX 2.1 (as published on April 2011) Also capable of importing and exporting SDMX 2.0 artefacts. Allows browsing, searching, editing and subscribing to artefacts Advanced access control mechanism for distributed maintenance of artefacts controlling also their visibility.

4.1 Availability Any Domain manager in Eurostat Any end-user of statistical data can access the PUBLIC data structures posted on the SDMX Eurostat Registry (SER) Access to artefacts can be controlled per domain to limit the visibility of those which are private or under construction, thus not to be disclosed

4.2 How to use it? Some information published in the SER is publicly available and freely viewable and downloadable Some information published in the SER is private No need to log in to view or to download information - SDMX Eurostat Registry Content Users can view the public content of the Registry, see how the various objects are interlinked Contextual help is available online

Access to the content of the Registry: text search Home page Access to the content of the Registry: text search Access to the content of the Registry by type Most recent items

Actions

2. Detailed view 3. Download 1. Quick view

2. Detailed view 3. Download 1. Quick view

2. Detailed view 3. Download 1. Quick view

2. Detailed view 3. Download 1. Quick view

4.3 Architecture Web Service Database All requests to the database are handled by the WS Request-Response Interface pattern Used by the GUI using SOAP Used by other applications, e.g. SODI-WS SDMX Converter Database Schema according to SDMX – IM The Graphical User Interface (GUI) is actually a web interface that allows users (with a valid ECAS login) access the contents of Eurostat’s SDMX Registry. The GUI accesses the Repository via the Registry Web Service using the SDMX Registry Interface messages. Thus, it is capable of accessing any SDMX compliant Registry Web Service as long as it implements the SDMX Registry Interface messages. The Web Service (WS) is the core of Eurostat’s SDMX Registry. All requests to the database are handled by the WS (even those from the GUI). This means that the database is only exposed via the WS to any potential client (human or machine). It follows the Request-Response Interface pattern described by the SDMX standard, since it implements a large part of the SDMX Registry Interface specification. Other potential users of the WS are the SDMX Converter, the DSW. The Repository (DB) is the actual data-storage of Eurostat’s SDMX Registry. To be more precise, it is a metadata storage, since it stores structural ad provisioning metadata. The database schema has been built according to the SDMX-IM.

4.4 Entity-Relationship model SDMX Information Model -> ER model ER does not support all relationships of class diagram Inheritance Many to many relationship No one-to-one mapping of classes and tables Query performance (e.g. number of joins) Query simplicity for application development Maintainability and table size Enforcing data constraints

4.5 Interface Request/Response pattern J2EE application Model-View-Controller (MVC approach) Struts framework The SDMX Registry Graphical User Interface (GUI) is a J2EE application based on a Model-View-Controller (MVC) approach. It has been built using Struts framework (http://struts.apache.org/), which is a free open-source framework for creating Java web applications. Apache Struts incorporates an MVC architecture, thus separating software applications in manageable parts. Specifically in an MVC architecture, the Model represents the business or database code, the View represents the page design code, and the Controller represents the navigational code. The Struts framework is designed to help developers create web applications that utilize a MVC architecture. The key components of the Struts framework are: A "request" handler provided by the application developer that is mapped to a standard URI. A "response" handler that transfers control to another resource which completes the response. A tag library that helps developers create interactive form-based applications with server pages. Moreover, the GUI application incorporates: JSP custom tag libraries and associated support in the controller Servlet Utility classes that support XML parsing, automatic population of Java Beans properties based on the Java reflection APIs, and internationalisation of prompts and messages SubmitStructureRequest - SubmitStructureResponse Insert, update, delete DSDs, Code Lists, Concept Schemes, Dataflows, etc. QueryStructureRequest - QueryStructureResponse Query DSDs, Code Lists, Concept Schemes SubmitProvisioningRequest - SubmitProvisioningResponse Insert, update, delete provision agreements QueryProvisioningRequest - QueryProvisioningResponse SubmitSubscriptionRequest - SubmitSubscriptionResponse SubmitRegistrationRequest - SubmitRegistrationResponse QueryRegistrationRequest - QueryRegistrationResponse NotifyRegistryEvent

5. SDMX Registry Hands-on exercise How to download DSDs

SDMX Registry