1. EDIT data validation system Ewa Stacewicz EUROSTAT VALIDATION TEAM
The first module of the training is about the functionalities EDIT offers to the users

2. Content of the presentation
Introduction
Objects in EDIT, main principles of EDIT
Ways of using EDIT
Using EDIT - integration with other tools
E. EDIT demo
In an Introduction we will speak about how EDIT works and what we can do by using it.
Then, we move to the way EDIT can be used in connection to other existing tools.
Starting to see what it is inside EDIT, we will see first the main objects we can handle by using EDIT, their definitions, usefulness and the way they link to each other
The core EDIT activity being the validation, it is worth to see how EDIT is doing this and how can we use its facilities to validate and to get validation reports
Finally, some hints about future developments foreseen for EDIT

3. EDIT Introduction
Data validation and editing tool provided by EUROSTAT - used in several ESS.VIP projects
Allowing users to:
import data
run validation programs
browse or export validation results
Friendly web-based User Interface
Available as a standalone application – can be installed locally on the user's PC and serve for the validation of confidential data 3

4. EDIT implementations FSS (SO & NSNE) R & D ENERGY ICT BOP-IIP
New in 2017
FSS (SO & NSNE)
R & D
ENERGY
ICT
BOP-IIP
UNIDEMO (ACQ and IMM)
Updated in 2017
FSS microdata
ASYLUM
RESPER
AES
CVTS
COD
EGR
TOURISM
PRODCOM
SBS, BOP
STATISTICS
EUROSTAT: 27 statistical domains use EDIT
Member States and other institutions: 13 statistical domains use EDIT

5. EDIT principles User, programmer, administrator roles Basic EDIT use
Accepted data formats

6. User role in EDIT Dataset upload according to a Format
Create validation Job
Browse Error Report or export it
EDIT dataset
data file
VALIDATION FLOW
(1-click)
+ validation Program
validation Job
EDIT dataset

7. Programmer role
Programmer role - manages the metadata needed by the user to execute programs
Implements Formats (DSD)
Develops Programs for Datasets validation containing validation rules and dataset operations, prepares Lookups for code lists check
Sets up the unattended mode configuration
Dataset Instance (Dataset) – a collection of data rows according to the structure defined in Format;
A bidimensional table composed by rows and columns: Columns correspond to the fields defined in the format;
EDIT can import DSDs or code lists;
EDIT acts as a client for the SDMX Registry Web Services in order to fetch DSD files and code lists data;
The DSD file is broken down into EDIT components;
Key families are translated to EDIT formats;
Code lists are translated to EDIT Lookups;
An EDIT Program is created performing lookup validations and basic checks on the dimension fields;

8. EDIT VALIDATION PROGRAM
Programmer role Program development
Custom EDIT Scripting Language designed for data validation
Programs contain validation rules and dataset operations
Coming in 2018: VTL translator integrated with EDIT
VALIDATION RULES
IN VTL
EDIT VALIDATION PROGRAM

9. Administrator role
Administrator role - manages domains, users and permissions
Domain - self-contained grouping of EDIT elements available to a group of users;
Contains Formats, Datasets, Programs
No object in two different domains can interact with each other
Users have access to all datasets and Jobs results within a Domain

10. Accepted data formats
GESMES / BOP ITS, BOP FDI multi-year 2007, 2008, 2009 observations
UNA:+.? '
UNB+UNOC:3+FR2+4D :1637+IREF GESMES/TS'
UNH+MREF GESMES:2:1:E6'
BGM+74'
NAD+Z02+ECB'
NAD+MR+4D0'
NAD+MS+FR2'
IDE+10+EUROSTAT_BOP_01 reporting'
DSI+BOP_FDI_A'
STS+3+7'
DTM+242: :203'
DTM+Z02: :702'
IDE+5+EUROSTAT_BOP_01'
GIS+AR3'
GIS+1:::-'
ARR++A:FR:N:2:330:N:4A:E:9999:9999: :702:0:A:F+0:A:F+0:A:F‘
ARR++A:FR:N:2:330:N:4F:E:9999:9999: :702:0:A:F+0:A:F+0:A:F'
ARR++A:FR:N:2:330:N:7Z:E:9999:9999: :702:0:A:F+0:A:F+0:A:F'
ARR++A:FR:N:2:330:N:A1:E:1100:9999: :702:5824:A:F+5930:A:F+4204:A:F'
ARR++A:FR:N:2:330:N:A1:E:1495:9999: :702:5828:A:F+5932:A:F+4206:A:F'
CSV (with or without header) /SBS, CVTS, TOURISM
9H; 2008; LT; 2; B-N_X_K642; 11930; 16236; ; ; ; ; UNIT; ; ; ; ; ; TT0; ; ; ; ; D08
9H; 2008; LT; 3; B-N_X_K642; 11930; 1001; ; ; ; ; UNIT; ; ; ; ; ; TT; ; ; ; ; D08
9H; 2008; LT; 4; B-N_X_K642; 11930; 529; ; ; ; ; UNIT; ; ; ; ; ; TT; ; ; ; ; D08
9H; 2008; LT; 30; B-N_X_K642; 11930; 17766; ; ; ; ; UNIT; ; ; ; ; ; TT; ; ; ; ; D08
9H; 2008; LT; 2; B-E; 11930; 1138; ; ; ; ; UNIT; ; ; ; ; ; TT; ; ; ; ; D08
9H; 2008; LT; 3; B-E; 11930; 104; ; ; ; ; UNIT; ; ; ; ; ; TT; ; ; ; ; D08
9H; 2008; LT; 4; B-E; 11930; 61; ; ; ; ; UNIT; ; ; ; ; ; TT; ; ; ; ; D08
FLR
E ZZZZZ
E ZZZZZ

11. Ways of using EDIT Standalone – running on a PC
Client – server – running in a Data Centre
As a web service – called by other applications (CONVAL)
INPUT FILE
You can use EDIT from within Internet by providing an authentication – ECAS username and password.
EDIT can be downloaded and used on a computer even if it is not connected to Internet. This fits well cases of confidential data and have the advantage of managing local resources in a configured way. For example, you can run the validation program repeatedly in a quicker way.
Client - server
VAL REPORT
VAL PROGRAM_ID
Status: Available for internal usage and for testing with some countries 11

12. EDIT variants for external users
EDIT Public Web server
EDIT standalone version for local installation
user from MS
user from ESTAT
Local authentication
All components running locally:
Tomcat server & PostgreSQL DB
Confidentiality is preserved
ECAS authentication
Non-confidential data
In preparation: remote EDIT validation service for confidential data

13. EDIT variants for ESTAT users
EDIT ESTAT server in the secure environment
EDIT ESTAT server in the standard environment
user from ESTAT
user from ESTAT
Confidential data
Non-confidential data
Unattended mode
EDAMIS back channel
Integration with other systems in Eurostat

14. EDAMIS Integration
EDAMIS can send data to EDIT by placing the files in a configurable location;
EDIT detects metadata based on the EDAMIS naming convention;
EDIT performs the processing in unattended mode;
EDIT acts as a client for the EDAMIS Feedback Channel Web Service in order to publish the results of a job execution.

15. Manual feedback EDIT & EDAMIS integration – Feedback modalities
Valid
Data
EDIT
Dom. Manager
Error
Report
Examples:
SBS
ENERGY
FSS
COD
Manual feedback 16

16. Automatic feedback EDIT & EDAMIS integration – Feedback modalities
Valid
Data
EDIT
Examples:
ASYLUM
Tourism
EGR
RD & GBAORD
ICT
Error
Report
Automatic feedback 17

17. Integration of pre-validation in EDAMIS
Purpose: Improve the user experience and the quality of data sent
Status: Part of EDAMIS 4, available Q2 2018

18. EDAMIS
notification

19. Validation report It contains:
Job results – information about the job plus an overview of the validation results;
Error statistics – summary of the errors;
Error report – detailed list of errors;
Acceptance/rejection algorithm implemented in the program.

20. Errors statistics

21. Detailed error report
Pass to EDIT and show the result of the job previously launched.

22. Detailed statistics report

23. More info on EDIT Take part in one of the next EDIT Webinars:
registration via
For next webinars please check the section
Editing & Validation Events on:

24. Thank you for your attention!
Any questions?