Presentation is loading. Please wait.

Presentation is loading. Please wait.

Eurostat 6. SDMX: A non-technical overview of the SDMX architecture and IT tools 1 Raynald Palmieri Eurostat Unit B5: “Central data and metadata services”

Published byHugo Bailey Modified over 8 years ago

Similar presentations

Presentation on theme: "Eurostat 6. SDMX: A non-technical overview of the SDMX architecture and IT tools 1 Raynald Palmieri Eurostat Unit B5: “Central data and metadata services”"— Presentation transcript:

1 Eurostat 6. SDMX: A non-technical overview of the SDMX architecture and IT tools 1 Raynald Palmieri Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, March 2016

2 Eurostat The SDMX Components 2 IT Infrastructure for exchange and sharing Content Oriented Guidelines SDMX Information Model

3 Eurostat Who? What? When? Who? How? Describing the data exchange What's next?

4 Eurostat EUROSTAT SDMX TOOLS FOR DESIGNERS 4

5 Eurostat Eurostat SDMX tools for designers 5 Tools offeredUse Data Structure WizardUsed to create, edit and test SDMX objects Maintenance of SDMX registry SDMX RegistryA central repository for storing and sharing SDMX objects

6 Eurostat Eurostat SDMX tools for designers 6 Data Structure Wizard SDMX Registry Create/Edit Store/Edit Web service

7 Work roles and process SDMX Registry Create DSD Publish DSD Format / structure SDMX messages Consult and get DSD Send SDMX messages COLLECTING ORGANISATION Consult and get DSD Interpret SDMX messages 7

8 Data Structure Wizard Creation / edition of DSDs Validation of DSD Local repository Artefacts are stored in a local repository Remote repository Can be used for retrieving existing artefacts 8

9 DSW: Management of SDMX artefacts 9

10 DSW: Artefact example: the codelists 10

11 Eurostat SDMX Registry 11 Metadata registry which implements the SDMX registry specificationsSDMX Tasked with providing structure, organisation, maintenance and query interfaces for the SDMX components required to support the data sharing vision Provides a web-based user interface as well as web services for interacting with the SDMX structural metadata objects in use within the collecting organisation and with statistical partners.

12 Eurostat SDMX Registry 12 Only concerned with providing information need to access the data and reference metadata sets. An application which wants a particular data or metadata set would then query the registry for the structure, and then go and retrieve the data or metadata set directly from the provider's web server. Example of existing SDMX registries: Euro SDMX Registry (2.1) SDMX Global Registry (2.1)

13 Eurostat New Euro SDMX Registry (2.1)  Eurostat's implementation of the SDMX Registry  Based on SDMX 2.1 (as published on April 2011) Also capable of importing and exporting SDMX 2.0 artefacts.  Allows browsing, searching, editing and subscribing to artefacts.  Advanced access control mechanism of artefacts controlling their visibility (Public/Private).

14 Eurostat New Euro SDMX Registry (2.1) Home page Most recent items Access to the content of the Registry by type Access to the content of the Registry text search

15 Eurostat Registry Content Number of objects of type selected Filtering list New Euro SDMX Registry (2.1)

16 SDMX Global Registry(SGR) Launched in 2011 by the SDMX Sponsoring institutions Originally to support the world-wide technical implementation of Balance of Payments and National Account implementations To make metadata material (DSDs, concept schemes, MSDs, code lists, etc.) publicly and centrally available used and maintained by national and international organisations.

17 Who should use the SDMX Global Registry (SGR)? Any national or international agency that wants to use SDMX standards for data management or for exchanging data with other organisations using SDMX standard. Any end-user of statistical data can access the data structures posted on the SGR.

18 How to use the SDMX Global Registry (SGR)? All information published in the SGR is publicly available and freely viewable and downloadable. No need to log in to view or to download information - SDMX Global Registry Content.SDMX Global Registry Content Users can view the full content of the Global Registry

19 Exploring by agency Available SDMX objects Click on a type Available SDMX objects Click on a type Enter a text to be found in the Registry Click on Export to choose a SDMX object to download Access to the web service

20 Codelist Content 2. Select an object

21 Dimensions, measure and attributes Corresponding list of codes 1. Choose a DSD in the list

22 Example of result for "NA-MAIN" 5 versions found Example of result for "NA-MAIN" 5 versions found

23 Web Service Query Select the SDMX version, the Object type and the identifiers of the SDMX object (ID, agency, version) Select the SDMX version, the Object type and the identifiers of the SDMX object (ID, agency, version) View the xml file in the browser or Download the file Select which referenced objects (e.g. Code lists) you want to include The URL generated is the SDMX query to be sent to the web service to get the xml file corresponding to the selected object

24 View the xml file in your browser (DSD NA_MAIN)

25 Eurostat SDMX TOOLS FOR DATA PROVIDERS 25

26 Eurostat SDMX Tools for Data Providers 26 Tools offeredUseAction Web Forms (EU)Excel-like templates Transmission of low volumes of data No costs for EU organisations Manual work for senders (type, copy/paste) SDMX Converter (sender’s PC) Converts data files between SDMX formats and other file formats Installation on sender’s PC Manual work for senders (convert) SDMX Converter (batch mode) Installation on server SDMX Reference Infrastructure A set of tools that allows to connect your IT systems to the SDMX world Installation on server Mapping of database to DSDs Own SDMX Implementation Local development in organisations

27 Eurostat Data provider view: One goal – different possibilities Webforms EDAMIS Single Entry Point SDMX-ML file Database export SDMX ConverterSDMX-ML file Database SDMX Reference Infrastructure SDMX-ML file SDMX Web Service Same DSD  Same Format Excel sheets SDMX ConverterSDMX-ML file

28 Eurostat Data provider view: One goal – different IT Architectures Webforms Web-based Push mode Database export SDMX Converter Local conversion Push mode Database SDMX Reference Infrastructure Push or Pull mode Excel sheets SDMX Converter Local conversion Push mode

29 Eurostat SDMX Converter https://webgate.ec.europa.eu/fpfis/mwikis/sdmx/index.php/SDMX_Converter https://webgate.ec.europa.eu/fpfis/mwikis/sdmx/index.php/SDMX_Converter File based conversion Open source and platform independent (Java) Different ways of using it: Graphical user interface Batch file (server or client side) Web Service interface Reusing source code in your own Java application Formats: SDMX-ML 2.0 & 2.1, SDMX-EDI (Gesmes), FLR, CSV, Google DSPL, predefined Excel templates

30 Eurostat GUI (Converter 5.2.1) 2.a Select the DSD in the local drive 2.b Identify a DSD to download from the SDMX Registry (configuration required) 2.c Identify a dataflow linked to the DSD to download from the SDMX Registry (configuration required) CSV parameters XML parameters for SDMX output formats 1. Selection of the input/output files and their format 6. SDMX (output) validation 2. If the local DSD includes multiple versions, we can specify the one derired 2. If the local DSD includes multiple versions, we can specify the one desired 3. SDMX header (.prop file) Only for flat and excel files 4. Mapping and Transcoding 5. CSV quotation 3. Excel parameter file

31 Eurostat 31

32 Eurostat 32

33 Eurostat Exercice: conversions 33 From CSV to SDMX Compact From XLS to SDMX Compact From SDMX Compact to CSV

34 Eurostat What is SDMX-RI 34 Framework for SDMX data provision Set of pick-and-choose reusable building blocks allowing a statistical office to disseminate SDMX data to the external world Designed to provide data and structural metadata based on mappings to each organization's dissemination data warehouse Uses SDMX standards including the one for Web Services

35 Eurostat Why SDMX-RI 35 Developed to simplify the exchange of data Provides standard software and components, allowing individual statistical organizations to interact and exchange their data using the same software and methodology Modular approach, users can o use part or the entire infrastructure, o extend it by adding new modules or modify it in any other way to suit their own purposes Developed in both Java and.NET

36 Eurostat SDMX Query (XML File) SDMX Reference Infrastructure Census-HUB architecture – Eurostat to NSI SDMX DataMessage (XML File) Eurostat Census HubNational Statistics Institute SDMX -RI Web service

37 Eurostat Supported DBs 37 Oracle SQL Server MySQL Pc Axis ODBC Supported DBs Oracle SQL Server MySQL PC Axis ODBC

38 Eurostat SDMX-RI components 38 Mapping Assistant  Graphical tool to create the mapping between the DSD and the dissemination database. Test Client  Used to test your dataflow locally NSI Web Service  Allows you to share your dataflow NSI Client  Web interface to interact with the Web Service

39 Eurostat SDMX-RI process 39 Dissemin ation DB DSD Mapping Store Mapping Store Test Client (Desktop Software) Test Client (Desktop Software) Web Service Data Consumer (external) NSI Client Local environment Is my dataflow working? Is my WS working?

40 Eurostat Dissemination Database 40 The SDMX-RI is configured to directly map the dissemination Database An SDMX output is generated by the SDMX-RI using the PULL mode (the Receiver Organisation directly requests data and receives an SDMX file with no intervention of the Sender Organisation)

41 Eurostat Dissemination Database 41

42 Eurostat Mapping Assistant 42 Facilitates the mapping between the structural metadata provided by an SDMX-ML Data Structure Definition (DSD) and those that reside in a database of a dissemination environment Maintains a Mapping Store for keeping the mappings between the SDMX and the local data storage scheme In the SDMX Reference Infrastructure, provides mapping information to the Data Retriever

43 Eurostat 43

44 Eurostat 44

45 Eurostat

46 Transcoding 46

47 Eurostat Test Client 47 Test mappings of data flows configured through the Mapping Assistant Test the NSI Web Service is correctly set up Export data sets to file in SDMX format

48 Eurostat

49 Web Client 49 Web Interface to directly request data. Graphical view of the results Generates the output in different formats

50 Eurostat NSI Web Client : Welcome screen 50

51 Eurostat

52 Support for data providers SDMX Reference Infrastructure Installed in ~32 countries for Census Hub Can be extended to other domains 1st contact point - Local Transmission Coordinators In National Statistical Institutes Experience with EDAMIS and SDMX 2nd contact point - Eurostat support teams

53 Eurostat Tools Relationships – Example 53 DSW SDMX Converter SDMX Converter SDMX Registry SDMX-RI Mapping Assistant DSD National DB CSV File SDMX-ML Dataset Mapping Produce use Store Produce use DSD

54 Eurostat SDMX Tools – Summary http://www.sdmxtools.org http://www.sdmxtools.org Eurostat tools  Eurostat SDMX Info Space SDMX Converter SDMX Reference Infrastructure & Mapping Assistant SDMX Registry & Data Structure Wizard Other tools available in the community Fusion Weaver (validation, transformation, creation) Fusion Transformer (command line conversion) Fusion Registry ECB Checker/Loader (conversion and validation) FAO OpenSDMX project

55 Eurostat Thank you for your attention! Questions

Download ppt "Eurostat 6. SDMX: A non-technical overview of the SDMX architecture and IT tools 1 Raynald Palmieri Eurostat Unit B5: “Central data and metadata services”"

Similar presentations

1 SDMX Reference Infrastructure (SDMX-RI) Work in progress, status and plans Bengt-Åke Lindblad, Adam Wroński Eurostat Eurostat Unit B3 – IT and standards.

1 SDMX Reference Infrastructure (SDMX-RI) Work in progress, status and plans Bengt-Åke Lindblad, Adam Wroński Eurostat Eurostat Unit B3 – IT and standards.
Agenda Item 3.3 SDMX reference architecture for NSIs Francesco Rizzo 24 th Meeting of the STNE Working Group “Statistics, Telematic Network & EDI” 16-17.

Agenda Item 3.3 SDMX reference architecture for NSIs Francesco Rizzo 24 th Meeting of the STNE Working Group “Statistics, Telematic Network & EDI”
Overview of SDMX: Statistical Data and Metadata eXchange Technical and Content Standards for Statistical Data Ann McPhail, Division Chief Statistics Department,

Overview of SDMX: Statistical Data and Metadata eXchange Technical and Content Standards for Statistical Data Ann McPhail, Division Chief Statistics Department,
CountryData Development Improving the collation, availability and dissemination of development indicators (including the MDGs) Nairobi, 27 November 2013.

CountryData Development Improving the collation, availability and dissemination of development indicators (including the MDGs) Nairobi, 27 November 2013.
SDMX at the IMF Progress Report Expert Group on Statistical Data and Metadata Exchange (SDMX 2007), Geneva, May 8-11, 2007 Patrick Hinderdael, Economic.

SDMX at the IMF Progress Report Expert Group on Statistical Data and Metadata Exchange (SDMX 2007), Geneva, May 8-11, 2007 Patrick Hinderdael, Economic.
Restricted Daejeon, 26-29 April 2010 1 An SDMX based unified data catalogue (UDC) MSIS – Meeting on the Management of Statistical Information Systems 1.

Restricted Daejeon, April An SDMX based unified data catalogue (UDC) MSIS – Meeting on the Management of Statistical Information Systems 1.
Implementation of SDMX for data and metadata exchange SDMX Basics Course 12 April 2013 Daniel Suranyi Eurostat B5 Management of statistical data and metadata.

Implementation of SDMX for data and metadata exchange SDMX Basics Course 12 April 2013 Daniel Suranyi Eurostat B5 Management of statistical data and metadata.
Eurostat Unit B3 – IT and standards for data and metadata exchange SDMX Basics Training – 2012 IT architectures for data exchange SDMX-RI and the Hub approach.

Eurostat Unit B3 – IT and standards for data and metadata exchange SDMX Basics Training – 2012 IT architectures for data exchange SDMX-RI and the Hub approach.
13-Jul-07 Implementation of SDMX for data and metadata exchange Balance of Payments Working Group 2-3 April 2012 Daniel Suranyi Eurostat B5 Management.

13-Jul-07 Implementation of SDMX for data and metadata exchange Balance of Payments Working Group 2-3 April 2012 Daniel Suranyi Eurostat B5 Management.
1 Meeting on the Management of Statistical Information Systems (MSIS 2010) SDMX architecture for data sharing and interoperability Francesco Rizzo, ISTAT,

1 Meeting on the Management of Statistical Information Systems (MSIS 2010) SDMX architecture for data sharing and interoperability Francesco Rizzo, ISTAT,
Basics David Barraclough OECD SDMX Coordinator

Basics David Barraclough OECD SDMX Coordinator
Model and Representations

Model and Representations
Eurostat 6. SDMX: A non-technical overview of the SDMX architecture and IT tools 1 Raynald Palmieri Eurostat Unit B5: “Central data and metadata services”

Eurostat 6. SDMX: A non-technical overview of the SDMX architecture and IT tools 1 Raynald Palmieri Eurostat Unit B5: “Central data and metadata services”
Eurostat 4. SDMX: Main objects for data exchange 1 Raynald Palmieri Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October.

Eurostat 4. SDMX: Main objects for data exchange 1 Raynald Palmieri Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, October.
SDMX IT Tools Introduction

SDMX IT Tools Introduction
2.An overview of SDMX (What is SDMX? Part I) 1 Edward Cook Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, 27-29 October 2015.

2.An overview of SDMX (What is SDMX? Part I) 1 Edward Cook Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, October 2015.
SDMX IT Tools SDMX Reference Infrastructure

SDMX IT Tools SDMX Reference Infrastructure
Eurostat SDMX Reference Infrastructure: Tools demonstration November 2015 Eurostat Unit B3 – IT and standards for data and metadata exchange Jean-Francois.

Eurostat SDMX Reference Infrastructure: Tools demonstration November 2015 Eurostat Unit B3 – IT and standards for data and metadata exchange Jean-Francois.
SDMX IT Tools SDMX use in practice in NA

SDMX IT Tools SDMX use in practice in NA
Eurostat November 2015 Eurostat Unit B3 – IT and standards for data and metadata exchange Jean-Francois LEBLANC Christian SEBASTIAN SDMX IT Tools SDMX.

Eurostat November 2015 Eurostat Unit B3 – IT and standards for data and metadata exchange Jean-Francois LEBLANC Christian SEBASTIAN SDMX IT Tools SDMX.

Similar presentations

Ads by Google