The Euro SDMX Registry & SDMX Global Registry Bengt-Åke Lindblad (Eurostat B3) Emmanuel Clement (Eurostat B5) SISAI Meeting – Luxembourg – 12-13 June 2012

Presentation overview What is the Euro SDMX Registry Role of the Euro SDMX Registry Euro SDMX Registry in figures Evolution and Roadmap 2012 What is the SDMX Global Registry Early version of the SDMX Global Registry Tentative roadmap for the SDMX Global Registry 2

What is the Euro SDMX Registry? This metadata registry implements the SDMX registry specifications It is a repository of SDMX artefacts (DSDs, standard code list…) used for SDMX based data/metadata exchange by Eurostat and Member States It has a web-based user interface as well as web services for interacting with SDMX structural metadata objects in use within Eurostat, the ESS and beyond It is accessible to other programs over the Internet (or an Intranet or Extranet) to provide information needed to facilitate the reporting, collection and dissemination of European statistics It is important to stress that registry services are not concerned with the storage of data or reference metadata sets. Data and reference metadata sets are stored elsewhere. The registry is only concerned with providing information needed to access the data and reference metadata, including information on the structure of the data and reference metadata. In addition, it should be understood that metadata registries are intended to interact with other applications, rather than directly with human users. An application which wants a particular data or reference metadata set should query the registry to know where those data and metadata sets reside, and how they are structured. The application can then go and retrieve the data directly from another database. The Euro-SDMX registry is a metadata registry which implements the SDMX registry specifications. The SDMX Registry is tasked with providing structure, organisation, and maintenance and query interfaces for most of the SDMX components required to support the data sharing vision. Provides the specification for the logical registry interfaces, including subscription/notification, registration of data and metadata, submission of structural metadata, and querying The SDMX Registry plays an important role in this architecture, in fact it can be seen as a central application which is accessible to other programs over the Internet (or an Intranet or Extranet) to provide information needed to facilitate the reporting, collection and dissemination of statistics. In its broad terms, the SDMX Registry – as understood in web services terminology – is an application which stores metadata for querying, and which can be used by any other application in the network with sufficient access privileges. 3 3 3

Role of the Euro SDMX Registry To enable IT applications, organisations (NSIs, and individuals To share data and metadata structures and other SDMX artefacts To allow applications to sign up (or subscribe) for notifications To act as central repository of SDMX objects used in (international) data and metadata exchange processes (DSDs, MSDs, Code Lists, etc.) To interact with other IT applications such as: ESS Metadata-Handler Data transmission (eDAMIS) Global Registry To allow organisations to publish statistical data and reference metadata in known formats such that interested third parties can discover these data and interpret them accurately and correctly To enable National Statistics Institutes and other external organisations to obtain Data Structure Definitions (DSD) and other structural metadata, such as Metadata Structure Definitions (MSD) for the Euro-SDMX Metadata Structures (ESMS) and European Standard Quality Reports (ESQR) To provide information about the structure of datasets and reference metadata sets, answering questions like: what code lists do they use? What concepts are involved? To provide information about what datasets and reference metadata sets are available, and where they are located To allow applications to sign up (or subscribe) for notifications, so that when information in the registry is updated (for example, a dataset or reference metadata set of interest becomes available, or structural metadata are changed), the application will be automatically alerted. To interact with other IT applications: Central place in the ESS Metadata-Handler (e.g. interaction with the NRME) Central place in data transmission (interaction with eDAMIS) Global Registry 4 4 4

as central repository for structural data The Euro SDMX Registry as central repository for structural data ESS Metadata Handler Other tools Euro SDMX Registry Storage of MSDs and report structures used to create Reference Metadata files (NRME) Like Data Structure Wizard or SDMX Converter The registry is the corner stone Storage of DSDs used for validation and creation of WebForms EDAMIS Web Interface for the maintenance of the data

Euro SDMX Registry in figures In 2008 the Euro SDMX Registry was fed with the content of two Access DB (ESTAT+ECB) GESMES-TS objects used in data/metadata exchange at that time (SDMX compliant) From 2008 and onwards: All newly created SDMX Artefacts used for data/metadata exchange are stored in the Euro SDMX Registry 6 6 6

Euro SDMX Registry in figures Situation as of May 2012: 7 7 7

Evolution and Roadmap 2012 Extension of functionalities Re-design of the Graphical User Interface Revision of the Access Control Mechanism Improvement of performance to meet service level objectives and limit expensive design rework Support SDMX 2.1 Technical Specification 8 8 8

Evolution and Roadmap 2012 (1) Extension of functionalities Import & Export of artefact from other file formats Locking & editing of artefacts Be able to identify in which other artefacts a given artefact is referenced Ensure that no objecct can be deleted if it is referenced by another object Update warning mechanism Enchancement of error messages/codes Audit trail 9 9 9

Evolution and Roadmap 2012 (2) New Graphical User Interface Different philosphy with tabular approach, resizable screens/windows and new features such as: Anonymous access Login via ECAS/registered mail Quick/Advanced Search What’s new feature Simple/detailed view of Artefacts Wizard guided creation of artefacts Storing of favourite artefacts/queries Compare artefacts Table sorting etc. 10 10 10

Collapsable and resizable elements to improve user experience GUI re-design (1) Tabular layout Easy browsing on results, content of artefact selected displayed under results Collapsable and resizable elements to improve user experience 11

GUI re-design (2) New Quick search functionality What's new containing recently modified artefacts 12

GUI re-design (3) New filtering and sorting functionalities New Referenced by functionality 13 13

GUI re-design (4) Advanced Search using more attributes Customizable Search depending on the type of artefact 14

Evolution and Roadmap 2012 (3) Revision of the Access Control Mechanism ECAS authentication system Revision of the login mechanism Role-Based Access Control (RBAC) Sub-agencies allowed Assignment of artefacts to subgroup of users within an agency Public/private artefacts Administration Module 15 15 15

Evolution and Roadmap 2012 (4) Resolve performance issues Improvement of performance to meet service level objectives and limit expensive design rework Improve SQL statements used to retrieve artefacts Improve insertions of artefacts in the database Resolve caching mechanism issue 16 16 16

Evolution and Roadmap 2012 (5) Support SDMX 2.1 Specifications Support for new web services interfaces Explicit WSDL for SOAP services New RESTful API Different End Points for SDMX 2.1 and 2.0 messages Unique dataset model for time series and non-time series Data model adaption to deal with both versions of the SDMX standard 17 17 17

What is the SDMX Global Registry? The Global Registry is a term which has been discussed on a community level since SDMX Global Conference in Washington 2011. The following are deemed as core functions: Provision of mechanism for SDMX as a maintenance agency to maintain and make available SDMX global resources; Offering a facility for registered Agencies to publish their structures for dissemination, their location of their Registry, acting as a “federated node”; Support for hosting all types of SDMX structure. core functions The following are deemed as core functions of the Global Registry: Provision of a mechanism for SDMX as a maintenance agency to maintain and make available SDMX global resources such as the top level Agency Scheme of Agency Ids, the cross domain concepts and code lists, and the concept scheme of dimension/attribute roles.. This is seen as a high priority requirement for SDMX as without these structures being maintained and globally accessible, at best there will be no guarantee of interoperability, and at worst SDMX simply will not work . Offering a facility for registered Agencies to publish their structures for dissemination This will be necessary for the wide dissemination of “global DSDs” or any metadata that is to be disseminated to a wide audience. Offering a facility for registered Agencies to register the location of their Registry, acting as a ‘federated node’ If there is no registry federation then the Global Registry cannot know about structures maintained in an organisation-controlled registry and it cannot offer a service to retrieve such structures. This would force all DSDs that are to be shared or made available publicly to be uploaded in full to the Global Registry. Support for hosting all types of SDMX structure (i.e. not restricted to a sub set of the structures contained in the SDMX standards) Different organisation will require different structures to be maintained. If these are not supported there is a danger that (2) and (3) above cannot be offered. As a minimum, an SDMX registry supports machine-to-machine interaction via a web service API. In addition, it should also offer a basic GUI to support user query for registry structures. 18 18 18

The SDMX Global Registry The SDMX Sponsors will put it in place for storing SDMX global artefacts; The SDMX Global Registry will be linked to other SDMX registries; SDMX artefacts should only be stored once and be accessible through different registries; The governance of the SDMX artefacts is discussed by the SDMX Secretariat (e.g. maintaining agency, versioning etc.); guidelines will be available in autumn 2012. core functions The following are deemed as core functions of the Global Registry: Provision of a mechanism for SDMX as a maintenance agency to maintain and make available SDMX global resources such as the top level Agency Scheme of Agency Ids, the cross domain concepts and code lists, and the concept scheme of dimension/attribute roles.. This is seen as a high priority requirement for SDMX as without these structures being maintained and globally accessible, at best there will be no guarantee of interoperability, and at worst SDMX simply will not work . Offering a facility for registered Agencies to publish their structures for dissemination This will be necessary for the wide dissemination of “global DSDs” or any metadata that is to be disseminated to a wide audience. Offering a facility for registered Agencies to register the location of their Registry, acting as a ‘federated node’ If there is no registry federation then the Global Registry cannot know about structures maintained in an organisation-controlled registry and it cannot offer a service to retrieve such structures. This would force all DSDs that are to be shared or made available publicly to be uploaded in full to the Global Registry. Support for hosting all types of SDMX structure (i.e. not restricted to a sub set of the structures contained in the SDMX standards) Different organisation will require different structures to be maintained. If these are not supported there is a danger that (2) and (3) above cannot be offered. As a minimum, an SDMX registry supports machine-to-machine interaction via a web service API. In addition, it should also offer a basic GUI to support user query for registry structures. 19 19 19

The early version of the Global Registry Will be based on (and limited to) the SDMX 2.1 Registry specification The core use cases the early version must support are limited to the following ones: Store and retrieve SDMX artefacts; Store and retrieve internationally agreed DSDs; Allow Agencies (organisations) to store and retrieve SDMX artefacts they maintain in the Global Registry; Provide a GUI for maintenance of and viewing of SDMX artefacts. Overall NSIs should access the EURO SDMX registry which will also provide access to the SDMX Global Registry. We will communicate more on the use and access of the Euro SDMX Registry and the Global Registry in the ESS Metadata Handler Newsletter. The early version of the Global Registry will be based on (and limited to) the SDMX 2.1 Registry specification. By “limited to”, we want to emphasize the fact that no modification of the SDMX information model or the SDMX Registry specification will be requested for the early version of the Global Registry. The core use cases that the early version of the Global Registry must support are limited to the following ones: Store (including updates) and retrieve the SDMX artefacts (core concepts and codelists from the content-oriented guidelines, list of top-level maintenance agencies, etc.). Store (including updates) and retrieve the internationally agreed DSDs. Allow agencies which do not want to deploy their own SDMX registry to store (including updates) and retrieve the SDMX artefacts they maintain in the Global Registry. Provide a GUI for maintenance of and viewing of SDMX artefacts in the Global Registry. 20 20 20

Development of enhancements Tentative Roadmap 2012-2013 Procurement Development of enhancements Initial setup Security audit Deployment 3 months 3 months 2 weeks 2 weeks 1 month June 2012 Sep 2012 Dec 2012 Jan 2013 Feb 2013 21 21 21

Contact Bengt-Åke Lindblad Bengt-Ake.Lindblad@ec.europa.eu Emmanuel Clement Emmanuel.Clement@ext.ec.europa.eu 22 22 22