SDMX IT Tools SDMX use in practice in NA

Slides:



Advertisements
Similar presentations
SDMX training session on basic principles, data structure definitions and data file implementation 29 November
Advertisements

Eurostat Unit B3 – IT and standards for data and metadata exchange SDMX Basics Training – 2012 IT architectures for data exchange SDMX-RI and the Hub approach.
Eurostat Global Implementation of SDMX in National Accounts Committee for the Coordination of Statistical Activities 22 nd Session 4-6 September Ankara,
13-Jul-07 Implementation of SDMX for data and metadata exchange Balance of Payments Working Group 2-3 April 2012 Daniel Suranyi Eurostat B5 Management.
1 Meeting on the Management of Statistical Information Systems (MSIS 2010) SDMX architecture for data sharing and interoperability Francesco Rizzo, ISTAT,
Eurostat 6. SDMX: A non-technical overview of the SDMX architecture and IT tools 1 Raynald Palmieri Eurostat Unit B5: “Central data and metadata services”
Eurostat 4. SDMX: Main objects for data exchange 1 Raynald Palmieri Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, October.
SDMX IT Tools Introduction
2.An overview of SDMX (What is SDMX? Part I) 1 Edward Cook Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, October 2015.
SDMX IT Tools SDMX Reference Infrastructure
Eurostat SDMX Reference Infrastructure: Tools demonstration November 2015 Eurostat Unit B3 – IT and standards for data and metadata exchange Jean-Francois.
SDMX IT Tools SDMX use in practice in NA
Eurostat November 2015 Eurostat Unit B3 – IT and standards for data and metadata exchange Jean-Francois LEBLANC Christian SEBASTIAN SDMX IT Tools SDMX.
7b. SDMX practical use case: Census Hub
Implementation of SDMX for Balance of Payments Balance of Payments Working Group 9-10 April 2013 BP Daniel Suranyi Eurostat B5 Management of statistical.
Eurostat November 2015 Eurostat Unit B3 – IT and standards for data and metadata exchange SDMX IT Tools SDMX Converter Jean-Francois LEBLANC Christian.
Eurostat 6. SDMX: A non-technical overview of the SDMX architecture and IT tools 1 Raynald Palmieri Eurostat Unit B5: “Central data and metadata services”
Eurostat Mapping Assistant May 2016 Eurostat, Unit B3 – IT solutions for statistical production Jean-Francois LEBLANC Christian SEBASTIAN 1.
IAEA International Atomic Energy Agency Implementing SDMX for Energy Domain: From Discussion to Actual Implementation and Testing Andrii Gritsevskyi Oslo.
The Eurostat Metadata Handler Götzfried Eurostat (Head of Unit B6)
B.6 Roadmap 2013 – 2014 SDMX RI User Group Luxembourg, September 2013.
The evolution of the SDMX infrastructure and services
Progress Update MSIS: Bratislava, April 2005
Training course on Euro SDMX Registry
The CVD Metadata Handler
SDMX Opportunities MED Meeting 14 May 2013 Daniel Suranyi Eurostat B5
7. SDMX practical use case: National Accounts
Practical use case of SDMX (1): Short-term Statistics (STS)
SDMX Converter Raynald PALMIERI June 2015
11. The future of SDMX Introducing the SDMX Roadmap 2020
SDMX Reference Infrastructure Introduction
SDMX Introduction and practical exercises
Jean-Francois LEBLANC Christian SEBASTIAN
2. An overview of SDMX (What is SDMX? Part I)
Eurostat – Units E2, B5 Cristina BLANARU
2. An overview of SDMX (What is SDMX? Part I)
ESS technical standards and tools for quality reporting
SDMX Tools Architecture
Workshop on ESA 2010 transmission programme – What and how?
Data Transmission Tools & Services EDAMIS, SDMX, Validation
Implementation of SDMX in the ESS
SDMX in the S-DWH Layered Architecture
SDMX Tools Overview and architecture
Statistical Information Technology
SDMX IT Tools SDMX Converter
SDMX as basis for water data reporting
Practical use cases of SDMX: Census Hub
Sharing data validation activities in the ESS.
August Götzfried Eurostat unit B 4
ESS VIP ICT Project Task Force Meeting 5-6 March 2013.
X-DIS project: final report
SDMX IT Tools Data Structure Wizard
SDMX : General introduction H. Linden, Eurostat, Unit B5
Item of the Agenda Towards an integrated Eurostat metadata handler – Eurostat SDMX Registry services for Member States Francesco Rizzo Unit B3 13.
SDMX Implementation The National Accounts use case
GENEDI EUROPEAN COMMISSION - EUROSTAT GENERIC EDI TOOLBOX
European Statistical System Metadata Handler ESS MH (Super) Providers
Eurostat Unit B3 – IT and standards for data and metadata exchange
Jean-Francois LEBLANC Christian SEBASTIAN
Eurostat Unit B3 – IT and standards for data and metadata exchange
Eurostat Unit B3 – IT and standards for data and metadata exchange
Validation Activities in the ESS What you will hear today…
Developing SDMX artefacts for data exchange, sharing and dissemination
14. SDMX: Global and regional implementation projects
Standardizing and industrializing a business process – the dissemination use case Alessio Cardacino - ESTP Course “Information standards.
SDMX IT Tools SDMX Registry
SDMX IT building blocks
Presentation transcript:

SDMX IT Tools SDMX use in practice in NA National Account use case Jean-Francois LEBLANC Christian SEBASTIAN 11-13 May 2016 Eurostat, Unit B3 – IT solutions for statistical production

Table of contents Objectives National Accounts (NA) NA business needs NA Domains NA Technical Integration NA Data Validation Process NA Project Cycle

Table of contents SDMX Tools in practice SDMX Tools for Data Providers Data Structure Wizard SDMX Converter SDMX Reference Infrastructure Tools relationship SDMX Tools – Summary SDMX Tools for Data Providers Possibilities for Data Providers IT Architectures for Data Providers How to use the tools Conclusion

1 Objectives To reflect how National Accounts work To present the SMDX IT tools To expose how to work with the data and SDMX in real situations To give a base understanding of the tools and their relationship that will be of help for further explanations

2 National Accounts (NA) BOP National Accounts Main Aggregates Sector Accounts Supply & Use Input / Output GFS FDI Excel questionnaires GESMES files Excel templates will be supported for a 2 years period. From then on, the NSI have to create their info in SDMX format because there won’t be any more support for excel templates. The Converter could be a solution but not relying in new templates whatsoever.

2 Why SDMX for NA? Within Eurostat Within the ESS Internationally

2.1 NA business needs Main trigger: new methodology SNA 93  SNA 2008 ESA 95  ESA 2010 Business process reengineering Corporate production systems Integration of production chains Harmonised coding Within Eurostat Within the ESS Internationally

2.1 NA business needs Cross-domain integration Balance of Payments Foreign Direct Investment ESS Business process reengineering Standardization Improved validation Reuse of tools Within Eurostat Within the ESS Internationally

2.1 NA business needs Data quality Lowering reporting burden 1 country  1 year  1 GDP Shared validation rules Across all organisations Lowering reporting burden 1 data flow from national to international International cooperation Inter-Agency Group on Economic and Financial Statistics = SDMX Sponsors Example: PGI Website http://www.principalglobalindicators.org Within Eurostat Within the ESS Internationally

2.2 NA Domains Shared Concepts and Code Lists National Accounts SNA 2008 ESA 210 Other related domains FDI statistics Other domains Balance of Payment Statistics Shared Concepts and Code Lists Cross Domain Concepts and Code Lists National accounts have 5 domains with one DSD each (MAIN aggregates, SECtor accounts and government finance, Supply/use and input/output statistics, REGional statistics, and PENSion schemes. FDI and BoP (Balance of Payment) are not using the NA DSDs, these domains have their own DSD. They only share some codelists with NA DSDs.

2.3 NA Technical Integration National statistical organizations International organizations National DBs Mapping Source SDMX DSDs & Data Flows National data sources Metadata driven process based on SDMX standards and tools Full automation of the data exchange possible

2.4 NA Data Validation Process Before/during transmission “First Level” Currently covered by SDMX Format Check (SDMX-ML) Code Check (SDMX DSD) Code used correctly (SDMX Dataflow & Constraint) After Transmission “Second Level” Not covered by SDMX Detailed value check Validation expressions …

2.5 NA Project Cycle All projects go through the same process which is shown in this cycle

Registry DSW Converter SDMX-RI 3. SDMX Tools in practice Find DSDs Edit/Create DSDs Data files -> SDMX files Database -> SDMX WS/files The DSDs, where can they find them? In the registry! How can they edit/create the DSD? The DSW is the solution. The first conversions can be realised with the converter. It is an intermediate solution for the dissemination. The converter also implements limited validation functionalities. Remember that the converter is limited when it comes to the dissemination phase. How can the user export the data directly from the database to the client? The RI is the solution. The conclusion is that Eurostat tools cover each need and it is illustrated by the NA example.

Any oranization can create their EURO SDMX registry The Data Structure Wizard (DSW) application provides an improved solution for the management of Data Structure Definitions (DSDs) within the SDMX-ML v2.0 and 2.1 standards. DSW covers and completes the following functionalities for handling structural metadata artefacts from the KeyFamilyDB Access tool, SDMX Registry, Eurostat GESMES Structural Metadatabase and ECB Structural Metadatabase (the functionality of the two GESMES tools is under development at the time being). Specifically, DSW supports: Maintenance of the following SDMX v2.0 and v2.1 artefacts: Code Lists, Concept Schemes, Data Structure Definitions, Data Flows, Category Schemes, Hierarchical Code lists and Organization Schemes Dynamic creation of a template for a specified DSD in SDMX v2.0 and v2.1 format Import and export of artefacts from/to SDMX-ML v2.0 and v2.1 format Import and export of DSDs from/to GESMES/TS structure files Interaction with any standard-compliant SDMX Registry (using SDMX RegistryInterface messages) Connection to SDMX Registry with "query" privilege and update of the local repository of artefacts Update of the SDMX Registry, using the "maintain" privilege, with an artefact that was edited or created locally Print of summary reports for Codelists, Concepts from ConceptSchemes, DSDs, Dataflows Print reports for the DSDs components and for the Codes from Codelists Advanced search (search Dataflow for a specified DSD, search DSD for a specified Dataflow, search all DSDs using a code list and search all the DSDs using a given concept) SDMX-ML Data Authoring feature (creating sample SDMX-ML datasets from DSDs) 3.1 Data Structure Wizard Java tool (open source and platform independent) Supports Maintenance of SDMX v2.0 and v2.1 structures Import and export of artefacts from/to SDMX-ML v2.0, v2.1 and GESMES format Interaction with Euro SDMX Registry (retrieval, modification and search of artefacts) Sample dataset creation

3.2 SDMX Converter Java tool (open source and platform independent) File based conversion Different ways of using it: Graphical user interface Command Line interface Web Service interface Common API interface Formats*: SDMX-ML 2.0, SDMX-ML 2.1, SDMX-EDI (Gesmes), FLR, CSV, Google DSPL, predefined Excel templates *Exceptions may apply

3.3 SDMX Reference Infrastructure Data Consumer (external) Web Service NSI Client Dissemination DB Mapping Store Mapping DSD Is my dataflow working? Is my WS working? Local environment Test Client (Desktop Software)

3.4 Tools relationship produces stores uses DSD DSW SDMX Registry ESS-MH MSD MSD produces DSD SDMX MetaDataset uses produces uses uses SDMX Converter SDMX-RI Mapping Assistant SDMX Dataset produces exposes produces Mapping Census Hub CSV or other datafile National DB

3.4.1 Tools relationship – Example Eurostat produces NA DSDs Eurostat stores NA DSDs DSW SDMX Registry DSD DSD NSI uses NA DSDs SDMX-RI NSI maps NA DSDs National DB Mapping Assistant Mapping The XSD Generator is responsible for producing XML Schema Definitions (XSD) based on a DSD received. The present version of the tool produces XSDs for Compact, Cross-Sectional or Utility formats of SDMX standard v2.0 and can be used • As a re-usable building block, through its API. • Through a standalone Graphic User Interface (GUI) • Through Command Line Interface (CLI) • Through a web GUI The tool is licenced under the terms of the European Union Public Licence V.1.1 A complete Euro SDMX Registry package is stored on CIRCA repository and can be downloaded here: XSD Generator_app_v1.0.0_2011.06.10 (ZIP) The package contains an application, source code and also documentation such as a User Manual, Installation Guide, Analysis and Design as well as Testing Documentation. All versions of the tool can be downloaded from CIRCA repository. SDMX-ML Dataset NSI produces NSI uses NA DSDs SDMX Converter NSI exports NA data (CSV, XLS)

3.5 SDMX Tools – Summary Eurostat tools  Eurostat SDMX Info Space SDMX Converter SDMX Reference Infrastructure & Mapping Assistant SDMX Registry & Data Structure Wizard Eurostat SDMX Info Space is a wiki with Information about the tools Trainings Videos References to CIRCABC Eurostat SDMX Info Space is a wiki used mainly by ESTAT B3 and B5.

4. SDMX Tools for Data Providers Tools offered Use Action Web Forms (EU) Excel-like templates Transmission of low volumes of data No costs for EU organisations Manual work for senders (type, copy/paste) SDMX Converter Converts data files from different file formats to SDMX format Local installation on sender’s PC Installation on server SDMX Reference Infrastructure A set of tools that allows to transform directly your data from a database to SDMX Mapping of database to DSDs Common API Supports the development of local SDMX systems Local development in organisations

4.1 Possibilities for Data Providers Webforms EDAMIS Single Entry Point SDMX-ML file Database export SDMX Converter Database SDMX Reference Infrastructure SDMX-ML file SDMX Web Service Same DSD  Same Format Excel sheets SDMX Converter SDMX-ML file

4.1.1 Possibilities for Data Providers. NA. Not applicable in National Accounts Webforms EDAMIS Single Entry Point SDMX-ML file Database export SDMX Converter Database SDMX Reference Infrastructure SDMX-ML file SDMX Web Service Will be phased out in National Accounts Same DSD  Same Format Excel sheets SDMX Converter SDMX-ML file

4.2 IT Architectures for Data Providers Webforms Web-based Push mode (EDAMIS) Excel sheets SDMX Converter Local conversion Push mode (EDAMIS) Database export All domains in Eurostat use push mode except Census HUB SDMX Converter Local conversion Push mode (EDAMIS) Database SDMX Reference Infrastructure Pull mode

4.3 How to use the tools Design your DSD using the DSW If you are a Data Collector Design your DSD using the DSW Make the DSD available in the SDMX Registry Inform your data providers for the available DSD

4.3 How to use the tools Download/pull the DSD from the SDMX Registry If you are a Data Provider Download/pull the DSD from the SDMX Registry If you have a DDB Data stores as files DDB Install and use the SDMX-RI Install and use the Converter Expose the data to be pulled (WS, HUB) Push via EDAMIS

4.4 Conclusion SDMX statistical and technical standards & SDMX IT tools and IT infrastructure provide the "toolkit" for the harmonisation and integration of data structures the rationalisation of national/international data flows finally improving the data quality of official statistics.

SDMX in practice in NA