2. Validation Activities in the ESS What you will hear today…
New strategy discussed in the ITDG 18/19 October 2011
Validation tools offered by Eurostat
Current status
Planned activities
Suggested way forward
Presentation of EBB (Editing Building Block)
Eurostat Unit B5 – Statistical Information Technologies GLC 26th Meeting – 25/26 October 2011

3. ITDG Conclusions on sharing Validation Activities
Working groups define minimum requirements, formalised rules and appropriate levels for validation
Formal language for validation  avoid ambiguity of interpretation
Eurostat develops and provides an environment and set of tools and services  Member States can use if they want
Update mandate of GLC  to include shared (meta)data validation
Eurostat Unit B5 – Statistical Information Technologies GLC 26th Meeting – 25/26 October 2011

4. Validation in the ESS – Current Status
Data validation takes place:
In Member States – before transmission
In Eurostat – before further dissemination and processing
Several steps in validation:
Format validation
Codes validation
Data validation
1st level: basic checks – existence of mandatory fields, range checks, consistency of info inside file
2nd and 3rd level: consistency with historical data / data from other sources (other countries, other statistics)
4rd level: expert validation / in-depth analysis
Eurostat Unit B5 – Statistical Information Technologies GLC 26th Meeting – 25/26 October 2011

5. Tools in Eurostat – Current Status
EDAMIS
Web Forms
On screen
Replace small Excel questionnaires
Generates SDMX-ML
Validation Engine
Validates SDMX-ML
Within single file
Distributed through EWA and EWP
Can be used for confidential transmissions
Editing Building Block
Different modes possible
Stand alone
Client / Server
Web service
Complex validation possible
Any file format
Checks against other data sources (mirror checks, against previously provided data)
High performance for big volume
Wedes
Extends EBB to a Web application for validation
Currently used in Energy
Eurostat Unit B5 – Statistical Information Technologies GLC 26th Meeting – 25/26 October 2011

6. EBB = Editing Building Block
Main Functionalities:
Acceptance of various file formats and number of variables (limited by the DBMS column number capacity)
Validation programs are parametric
Not only validation but also variable creation
Possibility to manipulate incoming datasets
Information is persistent (data+metadata) and reusable
Eurostat Unit B5 – Statistical Information Technologies GLC 26th Meeting – 25/26 October 2011

7. Functionality in detail
File management:
Fixed length records
Variable length records (delimited)
Sdmx-ML
Gesmes files
Scripting and web services
Web version and stand alone version
Eurostat Unit B5 – Statistical Information Technologies GLC 26th Meeting – 25/26 October 2011

8. Validation Rules, Computations
Rules are logical expressions followed by:
The rule name
The rule severity
The rule warning message
A possible modification or creation of data depending on the rule result.
Rules can be horizontal or vertical (inter record)
Special computations (outliers)
Output statistics (summary) and details for errors (what error, where in the dataset).
Eurostat Unit B5 – Statistical Information Technologies GLC 26th Meeting – 25/26 October 2011

9. Dataset operations Copy file, select part of file Split file Aggregate
Rename
Merge
Append
Reorder lines or columns
Eurostat Unit B5 – Statistical Information Technologies GLC 26th Meeting – 25/26 October 2011

10. EBB New Functionalities (2012)
Secure use using the same solution as the Commission (ECAS, SMS)
Running EBB in unattended mode linked to eDAMIS.
Integration of EBB with eVE and SDMX Registry
Eurostat Unit B5 – Statistical Information Technologies GLC 26th Meeting – 25/26 October 2011

11. Currently applied in the domains
Foreign Trade (Estat, Enhanced application)
Esspross (Estat)
CVTS (MS)
ITS, FDI (MS)
SBS (In preparation)
From inside SAS (Estat)
Eurostat Unit B5 – Statistical Information Technologies GLC 26th Meeting – 25/26 October 2011

12. Validation Tools – Current Activities
EDAMIS Transmission Feedback
Connects EDAMIS and EBB
Providing validation reports
SDMX and SDMX Registry
Single repository encompassing SDMX compliant data structures
EBB to read DSDs from SDMX Registry
Formal language for validation rules
VIP Validation project and GLC to provide input
Eurostat Unit B5 – Statistical Information Technologies GLC 26th Meeting – 25/26 October 2011

13. ESS Validation Services – The way forward
Offer validation as an optional service
Tailor made for needs of the statistical domain
Using agreed standards for rules and reports
Let Member States choose if they use central services or own tools
From Eurostat tools to an ESS service
Combine strengths of EDAMIS and EBB
EBB needs knowledge and support network at local level
GLC to include EBB to its mandate
 Central contact network for EDAMIS, SDMX and Validation Services
Eurostat proposes a GLC Task Force on extension of GLC mandate to include technical coordination for Eurostat support to validation services.
Eurostat Unit B5 – Statistical Information Technologies GLC 26th Meeting – 25/26 October 2011