ESS Validation State of Play and next steps DIME/ITDG Steering Group Meeting Item 5 Zagreb, 26 June 2017 Martine.Hahn@ec.europa.eu Eurostat, Unit B1

Presentation Outline Goals and expected outcomes for ESS Validation Business Architecture for ESS Validation Validation rules (including VTL)

Goals and expected outcomes for ESS Validation Medium-term goals Business Outcomes Goal 1: Ensure the transparency of the validation procedures applied to the data sent to Eurostat by the ESS Member States. Increase in the quality and credibility of European statistics   Reduction of costs related to the time-consuming validation cycle in the ESS ("validation Ping-Pong") Goal 2: Enable sharing and re-use of validation services across the ESS on a voluntary basis. Reduction of costs related to IT development and maintenance

BA for ESS Validation Approved by TF Validation, ESS Entreprise Architecture Board and WG on Methodology Clarifies: To-be state for validation in the ESS Roles of Eurostat and Member States in the validation of data How common validation services could be used in ESS Includes: Validation principles Business, Information and Application views 3 optional scenarios for implementation by countries Detailed acceptance process

BA for ESS Validation: Principles The sooner, the better Trust, but verify Well-documented and appropriately communicated validation rules Well-documented and appropriately communicated validation errors Comply or explain Good enough is the new perfect

BA for ESS Validation: Severity levels Errors: rules must necessarily be satisfied for mathematical or logical reasons. Data containing errors are not considered acceptable. Warnings: highlight suspicious values. Data with warnings may be accepted upon further verification and/or justification. Information: highlight potentially suspicious values which do not usually require further justification for acceptance.

BA for ESS Validation: Tools and Services (1)

BA for ESS Validation: Tools and services (2) STRUVAL available at Eurostat in Process Manager and as shared service CONVAL-1 (no VTL) available at Eurostat in Process Manager CONVAL-2 (with VTL) Q1/2018 for test and Q4/2018 as shared service Validation Rule Manager Q3/2018 for test and Q4/2018 as shared service from COM website Validation Rule Registry Q4/2018 (for all 3 scenarios)

BA for ESS Validation: Options for implementation in NSIs Connect to central repositories Installation of service/tool in own environment, access to repositories (SDMX DSDs, validation rules) Connect to CSPA compliant service Integration of service into own workflow Connect to the process Transmission of pre-validated data to Eurostat, Eurostat runs service and delivers report Grants Applications from 9 June – 8 Sept 2017 Action Jan/Feb 2018 - 2019 (12 months) 350.000€ total, 25.000€ min/MS, EU 90% of eligible costs ESSnet deliverables

Validation rules Assessment of VTL for content validation rules in 5 domains Need to harmonise description of validation rules, link to DSD, VTL suitable, support needed for complex rules Proof of concept of a VTL interpreter (VTL sandbox) VTL 1.1 Q3/2017 Standard framework (template, guidelines, best practices, …) for describing and agreeing validation rules and roles Q3/2017 Roll-out to cover statistical domains Q4/2019

Communication The ESS Validation Video: https://www.youtube.com/watch?v=vuBJxGtg7qA Business Architecture for ESS Validation: here ESTP course on Validation 21-22 Nov 2017

Do you have any QUESTIONS Do you have any QUESTIONS ? What are your plans for implementing the ESS Validation architecture?

ESSnet Validat Integration Timeline: Jan 2017 and March 2018 6 countries: DE, LT, NL, PL, PT, SE Work Packages: WP1: Finalisation of the Methodological Handbook for Validation WP2: Creation of a common validation report for the ESS WP3: Pilot implementation of the 3 scenarios of the Business and IT architecture and cost/benefit analysis of the implementations WP4: Dissemination in the ESS

ESSnet Validat Integration WP1: Methodolocical Handbook Identified potential areas for improvement of Methodological Handbook Started rewrite of some sections Preparation of a survey of validation metrics in use in ESS WP2: Common Validation Report for the ESS Identified 126 items to report (reduced to 96) (Re-)Definition of machine-readable report structure ongoing

ESSnet Validat Integration WP3: Implementation of Business Architecture Proof of Concept of VTL translator almost finished Preparation of a model for cost/benefit analysis of implementation scenarios by end of June Start pilot implementations in July WP4: Four Regional Conferences (PT) Lisbon 13.11.2017-14.11.2017 (NL) The Hague 11.01.2018-12.01.2018 (LT) Vilnius 01.02.2018-02.02.2018 (RS) Belgrade 22.02.2018-23.02.2018

The Hague, 11.-12.01.2018 Vilnius, 01.-02.02.2018 Belgrade, 22.-23.02.2018 Lisbon, 13.-14.11.2017