Eurostat Sharing data validation services Item 5.1 of the agenda

Eurostat Content  Background  Policy context  Vision Infrastructure Project on Validation  Generic data editing tools in Eurostat  Next steps

Eurostat Background In October ITDG meeting Eurostat presented:  the status of the data editing activity and related software  the work of Vision Infrastructure Project on Validation aiming at improving data editing within the ESS

Eurostat ITDG 2011 conclusions  A formal unambiguous language was needed for encoding the rules so that they can be translated into specific data editing system rule syntaxes  It wished to be kept informed of developments in the VIP on Validation  Use of the generic software tools provided by Eurostat should be optional

Eurostat Policy context The initiative is based on the Communication from the Commission to the European Parliament and the Council on the production method of EU statistics (Document ESSC 2010/05/6). It calls for:  more harmonisation and standardisation of statistical methodologies for data validation within the ESS  harmonising the IT infrastructure and sharing IT tools as a way to facilitate the use of agreed statistical methods, leading to better quality and higher productivity in the processing of statistical data

Eurostat Vision Infrastructure Project on Validation  The objectives of the project  The most important deliverables  The common language

Eurostat The main objectives  To create a more efficient production chain with responsibilities clearly assigned to the different actors  To develop various standards for describing the editing step, describing the expression language, developing functional specifications and using the tools to be shared within the ESS

Eurostat  An unambiguous expression language to be used by statisticians for encoding the rules  Guidelines for selecting a set of rules to ensure a defined minimum standard of quality for the data exchanged  Guidelines for the assignment of responsibility in the data editing chain (Member States and Eurostat)  User requirements and functional specifications for a  Tool to edit and monitor compliance  Tool to specify the rules The key deliverables

Eurostat Presentation to ESSC Eurostat presented the initiative to the ESSC in May The outcome of the discussion was as follows:  The ESSC generally supported the approach and the first steps of the planned project on broad principles and guidelines for a review of the data checking policy in the ESS  Eurostat would present a more detailed proposal on the project parameters to the ESSC meeting in  The project would focus on a common language and optional software

Eurostat The common language challenge  The development of a language for encoding and documenting the rules, is one of the most important and the most challenging  As requested during the last ITDG meeting, the language to provide for automatic translation of the rules to the scripting language of existing MSs software  The tool assisting in the creation of the rules by statisticians and producing the rules and a document presenting the rules in the common language with information allowing human understanding of the rules

Eurostat Generic data editing tool progress For the integration of the software, merging principal data editing tools EBB and EVE in to one - Edit has been achieved Preparation of the integration of Edit with EDAMIS to allow data editing and error reporting on arrival at Eurostat through the Single Entry Point:  The EBB is ready to communicate with EDAMIS  EDAMIS is being equipped with a back-channel for sending data reports back to data providers  Integration tests are to be carried out for EDAMIS, EBB and the SDMX Registry (for SDMX data set descriptions)

Eurostat Edit New features Import of Data Structure Definitions from the SDMX Registry to apply corresponding checks for codelist and format Web version for editing by the MSs in the MSs without any software installation: if MSs wish to edit any of their other national data collections each of them would be able to set it up for themselves  Available in a stand-alone version, a server version and a freely accessible Web (Internet) version 1-way SSL and ECAS protected (mail: for

Eurostat ESS cooperation  The definition of an agreed ‘minimum data quality standard’ by the business Working Groups enhances transparency in compliance monitoring and leads to efficiency gains in the production chain  The use of a common language for checking will improve interoperability of the data checking process within the ESS  In terms of the software solution: Edit allows Member States to check their data files before sending them to Eurostat or third parties

Eurostat Next steps  The ESSC just endorsed continuation of VIPV  VIPV will report to the ESSC during the first half of 2013  Thematic Working Groups will work on agreements on the definition of a set of quality checks to be implemented (shared by Eurostat & Member States) to guarantee a ‘minimum quality standard’ for data to be exchanged  Implementation of the VIPV final results will begin  Working Group (ITDG decision) will ease implementation of the approved ideas and provide Eurostat with stakeholder feedback

Eurostat Next in tool development Based on the user requirements and functional specifications, software solutions and IT tools will be continually developed In 2013 development work will esspecially focus on:  Integration of data checking and Pull Requestor within EDAMIS  Edit User interface  Requirements based on VIPV and conclusions of ESSC and ITDG

Eurostat In line with VIPV proposals…  Generic tool today offers more features than in the past. Some MSs preparing, testing or using Edit for their data  In some data collections the stakeholders, anticipating the opportunity, have started organising a common data checking procedure (incl. CVTS, SBS, BOP, FDI and ITS) are beginning to use agreed set of rules  VIPV, ITDG and ESSC will give the direction of future developments

Eurostat The ITDG is invited to:  Take note of the progress made on this initiative  Comment on any issues of interest and make suggestions for future actions