1. ESS Validation Project State of Play and next steps
ESTP course
Item 05
Luxembourg, Nov 2017
Eurostat, Unit B1

2. Deliverables provided by ESS VIP Validation in 2015
Dimension
Deliverables
Methodology
ESS methodological handbook for validation (Version 1)
Standards
Standard ESS validation syntax (VTL 1.0)
Processes/Governance
Proposal for a Business and IT architecture IT
Prototype for a validation rule manager
Prototype for a structural validation service

3. Next steps: May ESSC conclusions
In May 2016, the ESSC approved a set of follow-up actions to the ESS.VIP Validation.
These actions focus on deploying into statistical production the deliverables of the ESS.VIP Validation.

4. Goals and expected outcomes
for ESS Validation
Medium-term goals
Business Outcomes
Goal 1: Ensure the transparency of the validation procedures applied to the data sent to Eurostat by the ESS Member States.
Increase in the quality and credibility of European statistics
Reduction of costs related to the time-consuming validation cycle in the ESS ("validation Ping-Pong")
Goal 2: Enable sharing and re-use of validation services across the ESS on a voluntary basis.
Reduction of costs related to IT development and maintenance

5. Main achievements Jan 2016 to Nov 2017
Business architecture for ESS Validation finalised and approved
VTL:
Assessment of VTL for content validation rules in 5 domains
Proof of concept of a VTL interpreter (VTL sandbox)
Contribution to improvement of VTL (via public consultation)
VTL 2.0 Pre-release under review
Tools and services:
STRUVAL available at Eurostat in Process Manager and exposed as shared service
CONVAL-1 (based on EDIT) available at Eurostat in Process Manager and exposed as shared service
Communication (Video, Training courses, Web pages, …)

6. Next steps (until Dec 2019) VTL 2.0 published in Dec 2017
Deployment in all statistical domains of standard framework for describing and agreeing on validation rules and roles (template, guidelines, best practices, …)
Deployment of Business Architecture in ESS
Validation services and registries available
Standardisation of ESS validation reports

7. Standard framework for validation rules and roles
practise of designing and agreeing on validation rules at Working Group level will be extended to all statistical domains until end 2019
validation rules will be documented using common cross- domain standards
clear validation responsibilities will be assigned to the different actors

8. Priority for deployment in Domains
Top 20 domains for eDAMIS transmissions in 2016
Cover about 80% of all transmissions (in total of 83 domains)
Include 7 of the 8 pilot domains
5 Priority levels:
8 Pilot domains (Animal, Milk, Asylum, BOP-ITS, Energy, HBS, NA, STS)
9 domains in top 20, with ratio V2/V1>= 50%
4 domains in top 20 with ratio V2/V1<50%
23 domains ranked 21 to 83 with ratio V2/V1>= 50%
39 domains ranked 21 to 83 with ratio V2/V1< 50%
Tasks:
Maturity assessment and documentation gathering
Standardisation of documentation
Discussions in Working Groups
Validation rules written in VTL
Validation services can be used

9. Priorities for deployment
Top 20 domains
Rank
Dir.
Unit
Process
Transm.
% Trans
Version 1
Version 2
Version 3+
V2/V1
Versions
Pilot
Priority order 1 G G3
STS
6605
10.00%
4888
1094
623
22.38%
1.35 X 2 C C2
NA-ESA
6255
9.47%
2569
1548
2138
60.26%
2.43 3 E E1
CROPROD
4849
7.34%
1578
841
2430
53.30%
3.07 4 G5
COMEXT
4384
6.64%
1287
682
2415
52.99%
3.41 5 E5
ENERGY
4310
6.52%
2852
838
620
29.38%
1.51 6 F F2
ASYLUM
2848
4.31%
2238
506
104
22.61%
1.27 7 C5
BPM6
2730
4.13%
572
383
1775
66.96%
4.77 8 G2
SBS
2575
3.90%
1817
469
289
25.81%
1.42 9
TOUR
2525
3.82%
1068
732
725
68.54%
2.36 10 E3
AIR
2399
3.63%
959
764
676
79.67%
2.50 11
REGWEB
2286
3.46%
1228
797
261
64.90%
1.86 12 C4
HICP
2000
3.03%
22.58% 13
MRTM
1977
2.99%
568
442
967
77.82%
3.48 14
ANI
1783
2.70%
1444
241 98
16.69%
1.23 15
MILK
962
1.46%
554
202
206
36.46%
1.74 16 D D4
EDP
954
1.44%
333
380
72.37%
2.86 17
ROAD
844
1.28%
417
189
238
45.32%
2.02 18 E2
WASTE
807
1.22%
558
156 93
27.96%
1.45 19 F5
ESSPROS
783
1.19%
252
188
343
74.60%
3.11 20 F3
LFS
726
1.10%
270
160
296
59.26%
2.69 9

10. Main cooperation instruments
Pilot domains
Task Force on Validation
ESSnet Validat Integration
Grants to implement the ESS Architecture
National Accounts Validation Task Force and Grants
Energy Statistics Validation Task Force

11. 8 Pilot domains Animal, Milk, Asylum, BOP-ITS, Energy, HBS, NA, STS
7 in the Top 20 transmissions to Eurostat (covering more than 30% of file transmissions)
Pre-validation tools already used (eDAMIS Web Forms, EDIT, GENEDI, SDMX tools, MS-Excel, …)
NSI and Other National Authorities
Aggregated and micro-data

12. Task Force on Validation
8 Participating countries: DE, EE, IE, IT, NL, PT, SI, UK
Mandate between June 2016 and Dec 2017:
Pilot implementation of methodological handbook
Pilot implementation of VTL
Finalisation of Business and IT Architecture
7th meeting was on 7th Nov

13. ESSnet Validat Integration
Participating countries:DE, LT, NL, PL, PT, SE
Builds on results of previous ESSnet Validat Foundation
Work Packages between Jan 2017 and March 2018:
(LT) Finalise Methodological Handbook
(NL) Create a standard validation report structure
(PT, PL, SE) Develop a cost/benefit model for the 3 scenarios of the Business and IT architecture
(DE) 4 Regional conferences (Lisbon, Den Haag, Vilnius, Belgrade)
More at

14. Grants Implementation of Business Architecture for ESS Validation
5 applications received (EE, HR, IT, LT, PL)
Total budget : about EUR
Budget probably available for next year as well (2nd wave)
Objectives:
fostering implementation of ESS Validation in countries.
Covers all types of activities to increase the maturity of countries to implement the ESS validation policy and to benefit from ESS shared development.
Example of activities:
pilot the implementation of Business Architecture for ESS validation in one or more statistical domains
implementation of principle “the sooner, the better” by deriving the validation rules agreed at earlier stages in the national production process

15. National Accounts Validation Task Force
15 Participating countries from NSI or Central Banks: (AT, BE, CZ, DE, DK, FR, GR, IE, IS, IT, NL, PL, SE, SI, UK) + OECD + ECB
Mandate started in 2014
Review validation checks performed in Eurostat
Clarification of methodological or practical aspects
Validation rules for an internal or external pre-validation tool
Collection and dissemination of associated metadata
Last meeting at Eurostat: October 2017

16. National Accounts Validation Grants
2015 Grants: BG, DK, EL, NL, SI
2016 Grants: CY, CZ, DK, LT
Pilot scenarios of the Business Architecture for Validation:
SI, (DK), (NL)
LT, CY, DK, EL, CZ
BG, DK, EL, (NL)
Grant coordination meeting at Eurostat: 20 October 2017

17. Task Force ESS shared Validation and SDMX in Energy Statistics
6 Participating countries from NSI or Ministries: (CZ, DE, HR, PL, SE, TK) + IEA (International Energy Agency)
Mandate:
Understanding and optimisation of the ESS shared validation process and underlying SDMX DSD definitions
Pilot projects for testing and fine tuning
Assessment of effectiveness, user-friendliness and needs on training and documentation
Proposal for an ESS wide implementation plan
Main input: Validation handbooks in Energy Statistics
First meeting at Eurostat: 12 September 2017

18. Training courses
=> ESTP course Nov => 4 ESSnet regional conferences

19. QUESTIONS ?
Time for lunch ?