1. Validation at Insee

2. Validation at Insee 1 – validation prior to transmission to EU
2 – French uses of Eurostat validation tools
3 – Additional support needs in the area of validation services
4 – Lessons learned from our efforts in modernising national validation systems

3. 1 – Validation prior to transmission to EU
Local solutions
the same as for National Dissemination :
a validation at each level of the management
French specificity : studies
Data feed a specific publication
On each level of the writing process, figures are checked
Various tools
Excel / LibreOffice spreadsheets and macro
Sas programs
R programs
Different from a statistics producer to another
Not yet a unique, homogeneous and shared solution
Is it really possible for a great variety of statistics productions ?
Challenging : an up-coming project ?

4. 2- Uses of Eurostat validation tools
EDIT tool :
Standalone versions (on each PC)
Due to the nature of the data : confidential
for Business demography, Ifats, EGR, TOURISM, EHISS (…)
Input in CSV
STRUVAL and CONVAL
For SDMX files
Short Term Statistics, NA, Asylum, Air
Encapsulated in eDamis

5. EDIT Standalone solution Installation on each PC (15 + 5)
Upload of the data (csv)
Checking operations (called Job)
Download of the feedback
corrections and new Edit checking operations until sucessful feedbacks would be received

6. EDIT
Feedbacks : informations on the kind of errors and where there are in the file
« Not so easy to understand at the beginning »
Difference between critical and non-critical errors
→ send an explanatory comment
Limits of the standalone : no possibility to share the work between different persons and to optimize the process.
For example one person cannot upload all the data and another one follow the checkings.
→ For the LC no visibility on the checking

7. STRUVAL AND CONVAL Used through edamis No specific installation needed
Use of the V dataset (sandbox, not in production)
Feedbacks (like with Edit tool) available in eDamis (in the « Received files » Menu)
Only the correct files are sent in production

8. STRUVAL AND CONVAL STRUVAL : for Structural validation
Implemented since 2nd semester of 2016
Structure of the header → if not correct the validation process stops
Right dimensions and attributes
Codes (respecting or not the codelists)

9. STRUVAL AND CONVAL CONVAL : for Content validation
Recently implemented for NA / a bit early to give an opinion
Example of some CONVAL feedbacks unsuccessful :
Incorrect combination of OBS_VALUE and OBS_STATUS
= inter dimensions / attributes checks
For NA we passed the two validation systems but there were still errors :-)
Up-coming ?
Inter series checks (Y / Y-1)
Threshold for some values and variations
Like in EDIT tool ? Possible with SDMX ?

10. 3- Additional support needed in the area of validation services
EDIT tool
First of all, a question : future of EDIT alongside STRUVAL and CONVAL ?
An intermediary solution until the SDMX become the format of all Datasets ?
Is Edit still extended to new domains ?
For now : basically improvement of the existing tool
To reduce the cost entry
A more user-friendly GUI
A more visible support team
Guidelines and wiki need to be promoted
Webinar maybe more often

11. 3- Additional support needed in the area of validation services
EDIT tool
First of all, a question : future of EDIT alongside STRUVAL and CONVAL ?
An intermediary solution until the SDMX become the format of all Datasets ?
Is Edit still extended to new domains ?
For now : basically improvement of the existing tool
To reduce the cost entry
A more user-friendly GUI
A more visible support team
Guidelines and wiki need to be promoted
Webinar maybe more often

12. 3- Additional support needed in the area of validation services
EDIT tool
About validation rules
To share more widely the rules
Some rules are too strict / sometimes more flexibility is needed (but difficult to apply, we understand it)
For confidential data
a secured application and no more standalone version, difficult to maintain / time-consuming

13. 3-Additional support needed in the area of validation services
For STRUVAL/ CONVAL :
Good idea to encapsulate the validation tool in edamis
Easy to use even with a basic knowledge of SDMX
Warning : Burden of edamis in the peak days (29/09/17) => delays for feedbacks
Some errors can still exist although successful feedbacks have been received

14. 4- Lessons learned from our efforts in modernising national validation system
Conforted us in our national checking operations
pointed out missing controls
Reduced the number of sendings
And the « ping-pong » exchanges with Eurostat

15. 4- Lessons learned from our efforts in modernising national validation systems
Validation from the NSI is a crucial need for Eurostat
To reduce the work of its teams
To homogenize the data received (so they could be used)
But the final users of Eurostat are also the NSI themselves
Virtuous circle
Useful for data sharing

16. 4- Lessons learned from our efforts in modernising national validation system
The validation tools from Eurostat are already a business process, level not yet reached in France
We're actually redesigning our dissemination model
→ a centralized dissemination warehouse feeded by various databases
→ Validation process : a challenging issue
EU has raised awareness of it
A bit early yet, but we will surely benefit from Eurostat experience on implementing CONVAL and STRUVAL

17. Thank you for your attention !
Any questions ?