Interoperability issues in the implementation of SIMS

Slides:



Advertisements
Similar presentations
Eurostat J OINT UNECE/OECD/E UROSTAT MEETING OF THE GROUP OF EXPERTS ON BUSINESS REGISTERS 3-4 September 2013, Geneva Session 1: Economic globalisation.
Advertisements

Implementing ESS standards for reference metadata and quality reporting at Istat Work Session on Statistical Metadata Topic (i): Metadata standards and.
Eurostat achievements and challenges Emanuele Baldacci, Director European Commission - Eurostat Director Methodology; Corporate statistical.
1 Integration of the Eurostat and ESS Metadata Systems A. Götzfried Head of Unit B6 Eurostat.
SDMX IT Tools SDMX use in practice in NA
United Nations Economic Commission for Europe Statistical Division GSBPM in Documentation, Metadata and Quality Management Steven Vale UNECE
Quality declarations Study visit from Ukraine 19. March 2015
The Eurostat Metadata Handler Götzfried Eurostat (Head of Unit B6)
Towards more flexibility in responding to users’ needs
The ESS vision, ESSnets and SDMX
Item 6 - Introduction to ESS Metadata Handler
Group Discussions - Summary
Seminar on ESA 2010 Metadata
The CVD Metadata Handler
Usage of National Reference Metadata Editor (NRME)
ESTP TRAINING ON EGR Luxembourg – December 2014
MSDs and combined metadata reporting
11. The future of SDMX Introducing the SDMX Roadmap 2020
State of play of labour market and other domains of regional statistics Items 4.2 and 4.3 of the agenda Gorja Bartsch Eurostat Unit E4.
2. An overview of SDMX (What is SDMX? Part I)
Time Use Survey data processing and dissemination 17 July 2014
Eurostat – Units E2, B5 Cristina BLANARU
2. An overview of SDMX (What is SDMX? Part I)
ESS technical standards and tools for quality reporting
Draft EP/Council Regulation for processes, standards and
Opinions after the 24/25 February 2016 Plenary
SDMX Information Model: An Introduction
Implementation of SDMX in the ESS
Working Group on Population and Housing Censuses
X-DIS/XBRL Phase 2 Kick-Off
Statistical Information Technology
Giuliano Amerini Unit E6 (Transport)
Validation services developed in the ESS
ESS VIP ICT Project Task Force Meeting 5-6 March 2013.
The ESS reference metadata standards
Education and Training Statistics Working Group – 2-3 June 2016
3. An overview of the SDMX implementation process
August Götzfried Eurostat unit B 4
SDMX : General introduction H. Linden, Eurostat, Unit B5
A review of the 2011 census round in the EU, including the successful implementation of a detailed European legal base First meeting of the Technical Coordination.
ESQRS implementation in the in Labour Force Survey
SDMX Progress and implementation A. Götzfried, Unit B6
Dissemination Working Group John Allen
Legislative strategy for cross-cutting ESS legislation
Education and Training Statistics Working Group, May 2011
Item 7.3 (b) SDMX for UOE data collection
LAMAS Working Group 7-8 December 2015
Support for syntaxes (UBL and UN/CEFACT) Nicosia October 30, 2017
Measuring, reporting and communicating quality of National Accounts statistics (ESA 2010) in an integrated way with data production Christos LIOURIS,
Quality reporting in the ESS - State of play and next steps
SDMX Implementation The National Accounts use case
European Census Hub: a cooperation model for dissemination of EU statistics Paper prepared by Ioannis Xirouchakis Presentation: Christine WIRTZ, Eurostat.
M. Henrard, B5 N. Buysse and H. Linden, B6 Eurostat
3. An overview of the SDMX implementation process
Work Session on Statistical Metadata (Geneva, Switzerland May 2013)
European Statistical System Metadata Handler ESS MH (Super) Providers
ESS Vision and VALIDATION
ESS technical standards and tools for quality reporting
Introduction to reference metadata and quality reporting
The Role of Metadata in Census Data Dissemination
IA on quality reporting
7. Introduction to the main SDMX objects for metadata exchange
Future of EDAMIS Webforms
ESS conceptual standards for quality reporting
Interoperability of metadata systems: Follow-up actions
Approach to SIMS implementation
Improving the interoperability of metadata exchange in the ESS
Joint meeting of the working groups Environmental accounts and Monetary environmental statistics and accounts 15 May 2019 Quality reporting-National Metadata.
Presentation transcript:

Interoperability issues in the implementation of SIMS Item 2.2 – Standards Working Group 2-3 July 2018 Luca Gramaglia, Eurostat unit B5

"Once for all purposes" reporting One of the key objectives for the creation of SIMS was to achieve "once for purposes" reporting. One of the ambitions was that Member States would be able to map their own metadata systems to SIMS. Producing a SIMS-compliant report for Eurostat would have been simply a matter of harvesting the information already reported by domain managers for national purposes.

"Once for all purposes" reporting NSI Eurostat National concept 1 SIMS concept 1 National concept 2 SIMS concept 2 National concept 3 National concept 4 SIMS concept 3 National concept 5

European Interoperability Framework

European Interoperability Framework The EIF defines 4 layers of interoperability: Legal interoperability: Ensuring that organisations operating under different legal frameworks, policies and strategies are able to work together. Organisational interoperability: The way in which public administrations align their business processes, responsibilities and expectations to achieve commonly agreed and mutually beneficial goals. Semantic interoperability: Semantic interoperability ensures that the precise format and meaning of exchanged data and information is preserved and understood throughout exchanges between parties. Technical interoperability: This covers the applications and infrastructures linking systems and services. It include interface specifications, interconnection services, data integration services, data presentation and exchange, and secure communication protocols.

Quality and metadata reporting in the ESS from the EIF point of view Regulation 223/2009 CoP, QAF, GSBPM SIMS SDMX

Interoperability issues Different approaches to quality reporting in legal acts National vs European production processes High degree of customisation and low degree of reuse Use of non-standard syntax elements

Quality reporting in legal acts Layer affected Legal interoperability Description of the issue The specific arrangements for the implementation of the generic obligations established by regulation 223/2009 are often laid down in domain-specific implementing acts. These do not consistently refer to SIMS concepts and may therefore encourage diverging practices.

Concrete example Implementing Reg. 881/2017 on quality reports for the Census.

Potential solutions The problems caused by this issue should not be overemphasised. Eurostat usually maps the concepts mentioned in the legal acts to SIMS concepts even when SIMS terminology is not explicitly mentioned. However, to avoid potential issues, Eurostat will develop a common approach for the inclusion of metadata and quality reporting in legal acts. Improvement action Availability Develop a common approach for the inclusion of metadata and quality reporting in legal acts Q1/2019

National vs European production processes Layer affected Organisational interoperability Description of the issue There is not a one-to-one alignment between what is considered to be a single process or domain at European and national level. The number and granularity of metadata or quality reports requested by Eurostat differs from those produced for national purposes. This means that it may be difficult for member States to reuse for the purpose of European quality reporting quality information gathered for national purposes

Concrete example In the area of Short-Term business Statistics (STS), Member States are asked to provide 25 different metadata files to Eurostat. Many NSIs see however STS as a single statistical production process/domain, and therefore produces only one metadata file for national purposes. The end result is that NSIs have to do double-work: they have to produce one aggregated file for national purposes and 25 detailed ones for European purposes. The goal of "once for all purposes" reporting is not achieved.

Potential solutions Achieving a one-to-one correspondence between national and European processes is difficult, as this can very much depend on the internal organisation of each NSI. Mitigation actions can however be considered: Provide a clearer overview of the number and granularity of the metadata and quality reports requested from MS Better overview of active metadata flows in the ESS-MH Establish a link between EDAMIS datasets and ESS-MH metadata flows Give MS the option to provide certain information at domain level or even at country / organisation level.

Potential solutions

Potential solutions Improvement action Availability Clearly indicate in the ESS Metadata Handler which metadata flows are currently actively used for national metadata and quality reporting Q2/2019 Establish a link between the inventory of datasets in EDAMIS and the inventory of metadata flows in the ESS Metadata Handler Q4/2018 Give Member States the possibility to provide information at different levels of granularity Q4/2019

High degree of customisation Layer affected Semantic interoperability Description of the issue SIMS defines a set of concepts needed for metadata and quality reporting. In order to cater to the information needs of specific domains, the content of these items is often customised, either by using HTML templates or by introducing sub-concepts. Customisations risk breaking the semantic interoperability SIMS is supposed to provide, as they may change the meaning or expected content for a given SIMS concept.

Why is customisation so common? Some of the more common reasons that domain managers ask for the introduction of more specific information: Detailed methodological information needed for compliance monitoring purposes Standardisation of responses for certain concepts (e.g. providing a pre-defined set of possible answers via a code list) Receive information at the level of specific variables within a dataset / survey

Concrete example The LFS Quality Reports expect the following HTML template to be filled in for concept 6.2 (Sampling error). The requested information changes the meaning of the “Sampling error” concept by focusing on a specific aspect of it. Member States would be unable to directly reuse information compiled at national level for this concept, as it would not fit the structure of the template proposed by Eurostat.

Potential solutions A multi-pronged strategy can be pursued to confront this problem: Unneeded customisations can be prevented by a wider use of domain-specific guidelines. Sub-concepts should be preferred to HTML templates, as they offer a more transparent and SDMX-compliant manner of signalling customisations. As a general rule, customisations should not replace the base concept, but only further detail it. This means that the content of the base concepts should not be modified.

Potential solutions

(phase-out of grouping functionality needed for full implementation) Potential solutions Improvement action Availability Expand the use of domain-specific guidelines Q3/2018 Replace HTML templates by sub-concepts (phase-out of grouping functionality needed for full implementation)

Insufficient reuse of information Layer affected Semantic interoperability Description of the issue Domains that have implemented both the ESMS and ESQRS strcutures must currently submit two different files to Eurostat. As a result, the high degree of overlap between the two structures is not exploited and the information for shared concepts is duplicated. This goes against the principle of "once for all purpose" reporting.

Potential solutions Request that Member State supply a single report containing all SIMS concepts. The ESMS and ESQRS reports can then be extracted from this single input file. This approach has recently been successfully piloted in the HICP domain. Improvement action Availability Allow Member States to submit a single report containing all SIMS concepts in case both ESMS and ESQRS are needed Q3/2018

Use of non-standard syntax elements Layer affected Technical interoperability Description of the issue The SDMX standard offers the technical and syntactic basis for the exchange of metadata and quality reports. Over time however, Eurostat has introduced some custom elements that are outside of the SDMX standard and that therefore cause interoperability issues. The most notable among these is the so-called “grouping” functionality, which specifies how concepts in the MSD can be grouped in an HTML template for easier visualization.

Concrete example The "grouping" functionality allows ESS-MH administrators to define how sub-concepts should be displayed via an HTML template. The resulting SDMX-ML file thus mixes visualisation and structural elements in a way that is not easy for a national metadata provider to replicate independently (see screenshot below).

Potential solutions The grouping functionality is mainly used to organise sub-concepts within a table structure. This need can be addressed in an SDMX-compliant manner by representing the table as an SDMX dataset structured by an SDMX Data Structure Definition. The introduction of this new functionality will need to be accompanied by precise specifications for Member States who wish to build interoperable systems. Improvement action Availability Phase-out the grouping functionality Q4/2019

Potential solutions MSD DSD Concepts Dataflow Grouped by Structured by Annotation Concepts Dataflow SDMX dataset Represented by Structured by

Potential solutions Concept heading Webform generated based on DSD

No automated transmission of metadata files Layer affected Technical interoperability Description of the issue It is currently not possible to automate the submission of metadata files to Eurostat. Member States who are able to create SDMX-ML metadata files from their metadata systems must manually upload them to the ESS-MH, which is clearly inefficient.

Potential solutions Eurostat is currently investigating the possibility of allowing Member States to use EDAMIS to transmit metadata and quality reports. Improvement action Availability Automate metadata and quality report transmission via the use of EDAMIS Q1/2019

Interoperability layer Summary Interoperability layer Improvement action Availability Legal Develop a common approach for the inclusion of metadata and quality reporting in legal acts Q1/2019 Organisational Clearly indicate in the ESS-MH which metadata flows are currently actively used for national metadata and quality reporting Q2/2019 Establish a link between the inventory of datasets in EDAMIS and the inventory of metadata flows in the ESS-MH Q4/2018 Give Member States the possibility to provide information at different levels of granularity Q4/2019

Interoperability layer Summary Interoperability layer Improvement action Availability Semantic Expand the use of domain-specific guidelines Q3/2018 Replace HTML templates by sub-concepts (phase-out of grouping functionality needed for full implementation) Allow Member States to submit a single report containing all SIMS concepts in case both ESMS and ESQRS are needed Technical Phase-out the grouping functionality Q4/2019 Automate metadata and quality report transmission via the use of EDAMIS Q1/2019

Thank you for your attention! Any questions?