Validation services developed in the ESS ESTP training course Item 7(2) Luxembourg, 21-22 Nov 2017 Luca.Gramaglia@ec.europa.eu Eurostat, Unit B1

Validation in the ESS – IT building blocks VRM STRUVAL CONVAL

Structural Validation Service (STRUVAL) First step of validation Performs structural validation of statistical data files based on structural information following the SDMX information model Checks that syntax and coding follow the definition given in an SDMX DSD, the completeness of the records and highlights basic consistency errors (e.g. if the same key is repeated twice) Already used in Eurostat and available to Member States

Structural Validation Service (STRUVAL) In order to use STRUVAL you need: An SDMX DSD or an SDMX dataflow An SDMX-ML file Potential future improvements: ability to read a wider variety of data formats

Content Validation Service (CONVAL) Performed after structural validation Includes the complete range of validation operations on data prepared for statistical production basic logical and content checks intra- and inter-file data plausibility consistency checks cross-domain source based checks A first version of CONVAL used in Eurostat and in a limited fashion as a shared service for Member States

Content Validation Service (CONVAL) In order to use CONVAL, you need A description of the validation rules Additional data files (if a validation of level 2 or above is required) CONVAL will receive as input validation rules written in the standard validation language VTL (feature to be implemented at the beginning of 2018). It is likely a subset of VTL will be covered first, to be expanded in the future. VTL

EDIT vs CONVAL What is the relationship between EDIT and CONVAL? CONVAL is a CSPA service: at its core, CONVAL is a specification of a set of functionalities and of a service interface. There can be several implementations of CONVAL. EDIT is a first implementation of CONVAL. This however may or may not change in the future…

Validation Rule Manager Allowing searching, browsing and managing validation rules Second prototype made in 2016/2017 To be developed over the course of 2018

Validation Rule Manager – 2nd prototype

Validation Rule Manager – 2nd prototype

Validation services and CSPA Both CONVAL and STRUVAL have been built according to the CSPA guidelines The services are described according to the standard CSPA templates, which will be made available in the new CSPA service catalogue First (but not only) Eurostat CSPA services

Availability timetable Validation services can be made available to Member States under different scenarios…

Scenario 2 Member States use common ESS services in their process for validation (as shared or replicated services) Shared service Replicated services

Scenario 3 Member States use common ESS processes for validation

Availability timetable Scenario 2 Scenario 3 STRUVAL Available for both replicated and shared Available CONVAL Available as shared in a limited fashion Available as replicated starting in Q3/2018 VRM N/A Available end-2018 / Beginning 2019

Thank you for your attention! Any questions?