1. Workshop on the Validation of Waste Statistics
Item 2: ESS.VIP VALIDATION: Objectives, scope & concepts
Angel Simón Delgado
EUROSTAT

2. ESS.VIP VALIDATION VIP VALIDATION – First Phase
VIP VALIDATION – Deliverables in First Phase
ESS.VIP VALIDATON – Definition of the project
Concept of Excellence in Validation

3. ESS VIP-V First Phase Overall goal:
To develop validation solutions to be used by different production chains (horizontal integration), within the ESS (vertical integration)
Bottom-up approach:
Extensive consultation of all possible stakeholders
Participative management
Business driven approach
From pilots experience to general principles

4. Scope, objectives and outputs
ESS VIP-V First Phase
Scope, objectives and outputs
Documentation / Standardisation:
template and guidelines for process description
template and guidelines for a standard documentation of the validation process
Methodological analysis of Data Validation
Typology of validation rules
Standard definition of validation levels
Standard formalised “syntax” (understandable by business users) to express validation rules
Distribution of responsibilities in the production chain
Guidelines to be used for the attribution of responsibility in the whole production chain (MSs and Eurostat) by the WG.
Guidelines based on efficiency principles (Validation  Corrections: “the sooner, the better”)
Preparing for IS/IT solutions and architecture

5. Scope, objectives and outputs
ESS VIP-V First Phase
Scope, objectives and outputs
Towards IT/IS solutions and architecture:
Users’ requirements to develop a new software to allow business users to input validation rules and the corresponding error messages in a shared Central Repository of Validation Rules. The new software should be able to generate the rules in the Validation syntax developed by the project.
b) Validation Architecture defining the elements and their relationships in an integrated validation system ("common platform") to be used by:
Internal users, in an appropriate IS architecture to facilitate horizontal integration of IT/IS systems
All stakeholders in the production chain

6. Contribution to the VISION
More efficient production chain with clear attribution of responsibilities
Standard definitions, guidelines and validation syntax
Development of a common validation procedure
Common solutions to be shared within the ESS

7. Templates & Guidelines
ESS.VIP Validation First Phase - Deliverables
Documentation
Examples
Methodology
1.1 Inventory of documents
1.2 Analysis of inventory
1.5 Inventory of validation rules
1.6 Inventory of error messages
1.3 Validation & statistical processes
2.4 Analysis of validation typologies
3.1 Validation rules by typology
3.3 Error messages
1.4 Validation typologies
2.4 Analysis of validation typologies
2.5 Levels of validation
Solutions
Templates & Guidelines
3.2 Validation syntax
4.1 Functional specifications for GUI
2.1 Documentation of validation process
2.2 Documentation of statistical process
3.3 Error messages
3.4 Selection of validation rules
3.5 Improvement actions
3.6 Attribution of responsibilities

8. Data Deliverables: Validation levels Validation complexity Same file
Within an organisation
Within a domain
From the same source
Same dataset
Same file
Level 0: Format & file structure
Level 1: Cells, records, file
Between files
Level 2: Revisions and Time series
Between datasets
Level 2: Between correlated datasets
From different sources
Level 3: Mirror checks
Between domains
Level 4: Consistency checks
Between different organisations
Level 5: Consistency checks
Validation complexity

9. Deliverables: Typology of validation rules
File Structure
Filename
File type
Delimiters
Format
>1 file checks
Referential integrity
Code list
Cardinality
Mirror
Time series
Revised data integrity
Model-based consistency
1 file checks
Type
Length
Presence
Allowed character
Uniqueness
Range
1n file checks
Consistency
Control
Conditional

10. Validation Controls – Different actors – Different responsibilities
Deliverables: Guidelines for the attribution of responsibility of validation activities in the whole production chain
Guidelines for the allocation of responsibility, for the implementation of validation rules within the ESS based on an AGREEMENT Eurostat-NSI's with periodic performance revisions from both sides
Proposal for a generic business architecture of data validation:
Validation Controls – Different actors – Different responsibilities
Step 1
Step 2
Step 3
Step 4
Data preparation by the NSI's
Transmission of data and validation report
Loading data to production database
Additional processing & dissemination

11. Deliverables: Standard template for error/warning messages
Standard templates for error/warning messages and for validation report
Validation report structure:
Body
Footer
Rules applied
Error/warning messages
Total failures
No. errors
No. warnings
Header
Total records
Time stamp
Records failed
User ID
Sum of weights
Data checked
Maximum admissible error weight
(dataset name…)
Rate of acceptance
Maximum possible amount of error
Rate of performance
Error/warning message structure:
Rule ID
Severity
Rule type ID
Message text
Action
Failing data

12. Deliverables: VALS - Validation syntax
Standard syntax for validation language
To define a meta-language for the domain of statistical data validation to express, document and communicate validation rules
Trade-off between Human-understandable and Computer-parseable language
Implementation through Graphic User Interface to support business users to input and maintain validation rules and rule-sets
Types
Examples
Type check
validate ( type(A1.Rcount)='ΤΕΧΤ')
Range & math
validate( Table2.C_ between
(Table2.C_5_1 + Table2.C_5_2 + Table2.C_5_3 - tolerance) and
(Table2.C_5_1 + Table2.C_5_2 + Table2.C_5_3 + tolerance))
Code list check
validate ( match_codelist (A1.Quarter, CL_QRT) )
Range check
validate ( H.HB050 between 1 and 12 )
Rules & Metarules
validate ( age_range_rule.result = true and
country_codes_rule.result = true) …

13. ESS.VIP VALIDATION - Package 1:
Proposed approach
ESS.VIP VALIDATION - Package 1:
IMPLEMENTATION
Goals
Implementation of the methodological developments of VIP-V Phase I in the statistical domains/WGs
Maintenance and refinement of standards developed
User requirements for further developments
Evaluation, monitoring and reporting

14. ESS.VIP VALIDATION - Package 2:
Proposed approach
ESS.VIP VALIDATION - Package 2:
MICRO DATA
Goals
Vertical integration of the micro data validation within the ESS production processes taking into account the results of the first phase of ESS VIP-V
Extension of the functional specifications to apply to micro data validation
Integrated solutions for micro data validation

15. ESS.VIP VALIDATION - Package 3:
Proposed approach
ESS.VIP VALIDATION - Package 3:
SOLUTIONS
Goals
Adaptation of existing validation tools to the functional specifications issued from ESS VIP-V Phase I
Deployment of validation solutions to MSs
Distribution of validation rules in agreed language
Building-Blocks in an adequate web services architecture
Provision of web services validation solutions to be used by Member States before transmission to Eurostat

16. ESS.VIP VALIDATION - Package 4: EXTENSIONS AND GENERAL COORDINATION
Proposed approach
ESS.VIP VALIDATION - Package 4:
EXTENSIONS AND GENERAL COORDINATION
Goals
Overall coordination of the project
Coherence of validation approaches within ESS
Implementation of the meta-language within ESS
Analysis of links with other VIP & ESS VIP projects
Good practices identification
More sophisticated validation solutions:
Longitudinal validation
Mirror checks validation
ESS wide shared final warehouses

17. Elements in a validation system: ESS production chain
Eurostat
Member States
Single Entry Point
Statistical domain 1
Member States statistical production and validation
Statistical domain …
Validation Services
Web Inteface
Statistical domain n
Eurostat

18. Validation Service Validation Service Data Definition registry
Validation rules
repository
Validation Service
Data
Errors
Validation report
Metrics

19. Good practices in validation
Processes with good validation development
Document providing detailed validation process and rules, agreed by and available to Data Providers
Regular discussion in the WP and update of the document
Easy access to the latest version and to previous versions if relevant
Domain managers know which validation rules have been applied; they don’t need to know the details of HOW validation rules are technically implemented
A clear Validation summary report is provided to Data Providers; a detailed report is available
IT tools manage the feedback to users
Limits
Coherence between maturity of validation and maturity of process
Relative effort for harmonisation can be significant for small processes [/small teams]
Validation is dynamic (improvement) and needs resources to evolve

20. Towards the idea of "excellence" in validation
Review loop
Assess
Efficiency and completeness of rules are checked against error metrics and feedback from users
Adapt
Validation checks are re-designed if needed
Execute methods and tools
Process
Production process integrates the standard validation tools
Feedback
Standardised feedback to data providers on errors and corrections sent fast
Control
Validation metrics discussed with the data providers
Design validation checks
Optimal
Validation checks are sufficient, complete, efficient
Transparent
Validation checks are expressed in an understandable way and written in a document
Sound
Actions "if error" are defined
Implement within ESS
What?
Validation checks discussed and agreed with data providers
Where/who?
Optimal distribution of validation tasks amongst Eurostat and the data providers
How?
All validation checks in a centrally supported tool
Validation metrics are established

21. Current actions and next steps
Assessment of Eurostat domains in the field of validation
Functional specifications for Validation services to be accordingly adapted to the findings of the project
Tools development:
System to create/maintain a central repository of validation rules
Development and/or adaptation of IT Tools (EDIT, eDAMIS)
Set of standard documentation for domain managers for harmonisation of communication with data providers
Task Force to:
Identify best practices in ESS
Advice on implementation in the ESS
Optimisation of the validation process

22. Thank you
More information on: