ESS technical standards and tools for quality reporting

Slides:



Advertisements
Similar presentations
SDMX training session on basic principles, data structure definitions and data file implementation 29 November
Advertisements

Implementing ESS standards for reference metadata and quality reporting at Istat Work Session on Statistical Metadata Topic (i): Metadata standards and.
Metadata management and statistical business process at Statistics Estonia Work Session on Statistical Metadata (Geneva, Switzerland 8-10 May 2013) Kaja.
13-Jul-07 Implementation of SDMX for data and metadata exchange Balance of Payments Working Group 2-3 April 2012 Daniel Suranyi Eurostat B5 Management.
1 Integration of the Eurostat and ESS Metadata Systems A. Götzfried Head of Unit B6 Eurostat.
2.An overview of SDMX (What is SDMX? Part I) 1 Edward Cook Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, October 2015.
1 5a. SDMX and reference metadata exchanges Bogdan ZDRENTU Eurostat Unit B5: “Central data and metadata services” SDMX Basics course, October 2015.
SDMX IT Tools SDMX use in practice in NA
1 Quality reporting within the Eurostat and the ESS metadata systems August Götzfried and Håkan Linden Eurostat Unit B6: Reference databases and metadata.
Eurostat November 2015 Eurostat Unit B3 – IT and standards for data and metadata exchange Jean-Francois LEBLANC Christian SEBASTIAN SDMX IT Tools SDMX.
The Eurostat Metadata Handler Götzfried Eurostat (Head of Unit B6)
Prepared by: Galya STATEVA, Chief expert
The ESS vision, ESSnets and SDMX
Structural and reference metadata in the European Statistical System
Item 6 - Introduction to ESS Metadata Handler
Exchanging Reference Metadata using SDMX
Interoperable data formats: SDMX
The CVD Metadata Handler
SDMX Information Model
Usage of National Reference Metadata Editor (NRME)
11. The future of SDMX Introducing the SDMX Roadmap 2020
ESTP programme for 2016 Živilė Aleksonytė-Cormier
2. An overview of SDMX (What is SDMX? Part I)
Time Use Survey data processing and dissemination 17 July 2014
Reporting on QUALITY issues
Eurostat – Units E2, B5 Cristina BLANARU
2. An overview of SDMX (What is SDMX? Part I)
Draft EP/Council Regulation for processes, standards and
Workshop on ESA 2010 transmission programme – What and how?
5. Detail: Main SDMX objects for. metadata exchange. (What is SDMX
Interoperability issues in the implementation of SIMS
Implementation of SDMX in the ESS
Quality assessment ESTP Training Course “Quality Management and survey Quality Measurement” Rome, 24 – 27 September 2013 Giorgia Simeoni Researcher Unit.
SDMX Tools Overview and architecture
What is next? H. Linden Eurostat, Unit B5
SDMX as basis for water data reporting
Validation services developed in the ESS
August Götzfried Eurostat unit B 4
ESS VIP ICT Project Task Force Meeting 5-6 March 2013.
The ESS reference metadata standards
Education and Training Statistics Working Group – 2-3 June 2016
August Götzfried Eurostat unit B 4
SDMX : General introduction H. Linden, Eurostat, Unit B5
Working Group "Agriculture and Environment – Pesticides Statistics" September Quality reports B. Zdrentu V. Delhaxhe-Wolmering Eurostat.
ESQRS implementation in the in Labour Force Survey
Working Group "Education and Training Statistics" April 2013
ESTP Advanced course on quality reporting
Task Force Household Budget Survey Innovative tools and sources
SDMX IT Tools SDMX use in practice in NA
SDMX Progress and implementation A. Götzfried, Unit B6
9. Practical use case 3: Pesticides Use Project
Quality reporting in the ESS - State of play and next steps
SDMX Implementation The National Accounts use case
M. Henrard, B5 N. Buysse and H. Linden, B6 Eurostat
Annegrete Wulff Statistics Denmark
Work Session on Statistical Metadata (Geneva, Switzerland May 2013)
European Statistical System Metadata Handler ESS MH (Super) Providers
ESS technical standards and tools for quality reporting
Introduction to reference metadata and quality reporting
5. SDMX: General input requirements
7. Introduction to the main SDMX objects for metadata exchange
ESS conceptual standards for quality reporting
14. SDMX: Global and regional implementation projects
SDMX: Frequently Asked Questions
Interoperability of metadata systems: Follow-up actions
Approach to SIMS implementation
Improving the interoperability of metadata exchange in the ESS
Joint meeting of the working groups Environmental accounts and Monetary environmental statistics and accounts 15 May 2019 Quality reporting-National Metadata.
Presentation transcript:

ESS technical standards and tools for quality reporting Giorgia Simeoni Istat, simeoni@istat.it ESTP Training Course “Advanced course on quality reporting” Luxembourg, 24-25 November 2016

Contents Overview on SDMX for reference metadata Introduction to ESS Metadata Handler (ESS-MH) Interoperability between national metadata systems and ESS-MH ESS-MH Demonstration

Reference metadata in SDMX SDMX: the technical standard for Statistical Data and Metadata eXchange Based on the exchange of xml messages (SDMX-ml), and including an Information Model. The implementation of the SDMX standard is supported by a series of SDMX tools. Introduction of SDMX for the transmission of reference metadata and: supported quality reporting harmonisation influenced the structure for ESS quality reports

SDMX Information Model* * SDMX self-learning package No. 2 Student book https://webgate.ec.europa.eu/fpfis/mwikis/sdmx/index.php/Self_Learning_Tutorial:_Information_Model

Metadata Structure Definition - MSD The Metadata Structure Definition is the SDMX artifact that defines the structure of the quality report. It corresponds to the template that the quality report should follow. It is made up of 2 main parts: Identifiers: describe the object which describes reference metadata. Report structure: a set of report attributes. The report attributes correspond to the statistical concepts required from the template. ESMS and ESQRS are implemented through MSDs.

Metadata Structure Definition ESMS ESMS_REPORT_FULL FULL_ESMS DATAFLOW > Domain-subdomain DATA_PROVIDER > NSI TIME_PERIO > reference period ESTAT_DATAFLOWS_SCHEME: List of Domains ESTAT_ORGANISATIONS: List of NSIs and other organisations

Classification system Metadata Structure Definition SDMX cross domain concepts ESMS … Data description Classification system Coverage - sector …. FULL_ESMS ESMS_REPORT_FULL

Metadataflow The Metadataflow is the conjuction between the template (MSD) and the statistical domain (reported as a CATEGORY of a CATEGORY SCHEME in SDMX standard). It provides also the information on the reporting frequency (annual, quarterly…) e.g. the metadataflow AEI_ESQRS_A means that for the AEI domain (Agri-environmental statistics) the ESQRS template should be used with annual (A) reporting frequency

Metadataset The Metadataset is the SDMX artifact that contains the filled in quality report It refers to a metadataflow It reports the values of the identifiers of the MSD It reports the values of the metadata attributes of the MSD

What is ESS Metadata Handler (ESS-MH)? The ESS-MH is an IT application provided by Eurostat allowing the compilation, management and transmission of reference metadata files and quality reports following ESS standards SDMX compliant: it produces metadatasets in SDMX-ml format for the statistical domains included in a category scheme for which one ore more metadataflow(s) has been opened referring to specific MSDs

Reference metadata transmission Reference metadata in the ESS are mainly exchanged in SDMX format via the ESS-MH. Thus, what is implemented in the ESS-MH defines the rules for the exchange. E.g.: At the moment, the version of SDMX used for reference metadata in ESS-MH is 2.0 For IT experts: there are 2 types of SDMX messages for metadatasets the generic metadata set the metadata structure specific metadata set The generic metadata set is required in the ESS-MH CLOODT Hubertus (ESTAT) 16/09/2016 OK to let it in, but if interfaces that consume reference metadata using the SDMX standard are well defined they should be agnostic for what concerns the SDMX message type used. The type of message should then not be a show stopper for further processing

Access to ESS-MH At NSI level there are 2 different user profiles that access to ESS-MH: The National Administrator: manage the metadata files from all the statistical domains and from all the national Institutions manage national user profiles The National User: manage the metadata files for specific statistical domain(s) To access ESS-MH an ECAS account is needed

Interoperability between national metadata system and ESS-MH* Three possible scenarios Only ESS-MH is used to produce national metadata files * Nielsen et al.(2015) ESS reference metadata: implementation of standards and interoperability of national and European metadata systems Workshop of the Modernisation Committee on Standards: International Collaboration for Standards-Based Modernisation Meeting, 5 – 7 May 2015

Interoperability between national metadata system and ESS-MH* National metadata system is used to produce the metadata file and then it is manually imported in ESS-MH * Nielsen et al.(2015) ESS reference metadata: implementation of standards and interoperability of national and European metadata systems Workshop of the Modernisation Committee on Standards: International Collaboration for Standards-Based Modernisation Meeting, 5 – 7 May 2015

Interoperability between national metadata system and ESS-MH Metadata file is produced using the National metadata system and automatically imported in ESS-MH via EDAMIS * Nielsen et al.(2015) ESS reference metadata: implementation of standards and interoperability of national and European metadata systems Workshop of the Modernisation Committee on Standards: International Collaboration for Standards-Based Modernisation Meeting, 5 – 7 May 2015

Interoperability between national metadata system and ESS-MH Technical specification for file to be produced by the national metadata systems and imported in the ESS-MH has been produced by Eurostat E.g.: - only the SDMX-ML file with the metadata set or a zip file including the metadataset, a directory with the annexes, a directory with the images, and the related MSD (optional) can be imported - the name of the metadataset as well as the tag on the header should follow specific rules - the «annotations» tag can be used to transmit information on annexes and on the restriction for publication CLOODT Hubertus (ESTAT) 16/09/2016 will you explain these rules?

Interoperability between national metadata system and ESS-MH In order to accomodate domain specific reporting needs, html specific templates have been introduced in ESS-MH within the SDMX metadata attribute values

Interoperability between national metadata system and ESS-MH In practice, there are domain specific quality reporting elements included below SIMS level The interoperability between national metadata system and ESS-MH is more difficult CLOODT Hubertus (ESTAT) 16/09/2016 suggestion: may be explain this with a small example to make it concrete for a participant?

Interoperability between national metadata system and ESS-MH One of the objectives of this course is to promote cross-domain harmonisation on quality reports, reducing domain specific accomodation as far as possible

DEMO of ESS-MH Production environment https://webgate.ec.europa.eu/estat/spe/metaconv/ Demo on the «acceptance environment» https://webgate.acceptance.ec.europa.eu/estat/spe/metaconv/ Webinar on ESS-MH are periodically organised by Eurostat

References Eurostat / ESS SDMX info space: https://webgate.ec.europa.eu/fpfis/mwikis/sdmx/index.php/Main_Page SDMX website: https://sdmx.org SDMX content oriented guidelines: https://sdmx.org/?page_id=4345 SDMX information model detailed information: https://sdmx.org/wp-content/uploads/SDMX_2-1-1_SECTION_2_InformationModel_201108.pdf SDMX checklist for sdmx design projects: https://sdmx.org/?sdmx_news=checklist-for-sdmx-design-projects Nielsen et al.(2015) ESS reference metadata: implementation of standards and interoperability of national and European metadata systems Workshop of the Modernisation Committee on Standards: International Collaboration for Standards-Based Modernisation Meeting, 5 – 7 May 2015 Eurostat. Metadata Extra SDMX-ML annotations for ESS-MH