1. Sharing data validation activities in the ESS.
Item 4.1 of the agenda
Sharing data validation activities in the ESS.
Recent developments and plans for 2012
Georges Pongas – B3
19-20 October 2010
IT Directors’ Group Meeting

2. Background ITDG 2010: Sharing of Validation tools
Eurostat presented a document on sharing selected validation tools at ESS level
Sharing of Validation tools
Is a crucial aspect of future infrastructure of the ESS reinforcing common standards in terms of rules and common data and metadata use.
In can result in considerable economies both for Member States and Eurostat.
Sharing of software services or software and (meta)data distribution can be envisaged.
Challenges: IT standards, interdependence of actors, operational support
18-19 October 2011
IT Directors’ Group Meeting

3. Data validation in the ESS
Data validation takes place:
In Member States – before transmission
In Eurostat – before further dissemination and processing
Several steps in validation:
Format validation
Codes validation
Data validation
1st level: basic checks – existence of mandatory fields, range checks, consistency of info inside file
2nd and 3rd level: consistency with historical data / data from other sources (other countries, other statistics)
4rd level: expert validation / in-depth analysis
18-19 October 2011
IT Directors’ Group Meeting

4. Data validation tools developed by Eurostat
eVE = eDamis Validation Engine
Allows for a final check before transmitting data to Eurostat
Covers format, codes and basic checks
For files in SDMX-ML format; linked to DSD
EBB = Editing Building Block
Allows importing of external reference files and databases
Can be configured to use many files simultaneously
For files with an agreed format applied by all data senders (csv, flr,sdmx-ml, sdmx-edi, databases)
Not only validation but also correction and computation
WEDES
-Web application for the validation of Energy statistics applicable by Member States and Eurostat (an extension of EBB).
18-19 October 2011
IT Directors’ Group Meeting

5. EBB = Editing Building Block
Main Functionalities:
Acceptance of various file formats and number of variables (limited by the DBMS column number capacity)
Validation programs are parametric
Not only validation but also variable creation
Possibility to manipulate incoming datasets
Information is persistent (data+metadata) and reusable
18-19 October 2011
IT Directors’ Group Meeting

6. Functionality in detail
File management:
Fixed length records
Variable length records (delimited)
Sdmx-ML
Gesmes files
Scripting and web services
Web version and stand alone version
18-19 October 2011
IT Directors’ Group Meeting

7. Validation Rules, Computations
Rules are logical expressions followed by:
The rule name
The rule severity
The rule warning message
A possible modification or creation of data depending on the rule result.
Rules can be horizontal or vertical (inter record)
Special computations (outliers)
Output statistics (summary) and details for errors (what error, where in the dataset).
18-19 October 2011
IT Directors’ Group Meeting

8. Dataset operations Copy file, select part of file Split file Aggregate
Rename
Merge
Append
Reorder lines or columns
18-19 October 2011
IT Directors’ Group Meeting

9. EBB New Functionalities (2012)
Secure use using the same solution as the Commission (ECAS, SMS)
Running EBB in unattended mode linked to eDAMIS.
Integration of EBB with eVE and SDMX Registry
18-19 October 2011
IT Directors’ Group Meeting

10. Currently applied in the domains
Foreign Trade (Estat, Enhanced application)
Esspross (Estat)
CVTS (MS)
ITS, FDI (MS)
SBS (In preparation)
From inside SAS (Estat)
18-19 October 2011
IT Directors’ Group Meeting

11. Proposed Organisation
The successful deployment of EBB needs the existence of a knowledge and support network at local level.
This network exists; It is called Local Coordinators and is the contact point for transmissions, and support for eDAMIS, eVE, Statel and SDMX.
It is proposed that this network includes also EBB in its support mandate.
Eurostat will create the conditions for this inclusion (courses, material,..)
18-19 October 2011
IT Directors’ Group Meeting