EDIT data validation system Ewa Stacewicz EUROSTAT VALIDATION TEAM The first module of the training is about the functionalities EDIT offers to the users

Content of the presentation Introduction Objects in EDIT, main principles of EDIT Ways of using EDIT Using EDIT - integration with other tools E. EDIT demo In an Introduction we will speak about how EDIT works and what we can do by using it. Then, we move to the way EDIT can be used in connection to other existing tools. Starting to see what it is inside EDIT, we will see first the main objects we can handle by using EDIT, their definitions, usefulness and the way they link to each other The core EDIT activity being the validation, it is worth to see how EDIT is doing this and how can we use its facilities to validate and to get validation reports Finally, some hints about future developments foreseen for EDIT

EDIT Introduction Data validation and editing tool provided by EUROSTAT - used in several ESS.VIP projects Allowing users to: import data run validation programs browse or export validation results Friendly web-based User Interface Available as a standalone application – can be installed locally on the user's PC and serve for the validation of confidential data 3

EDIT implementations FSS (SO & NSNE) R & D ENERGY ICT BOP-IIP New in 2017 FSS (SO & NSNE) R & D ENERGY ICT BOP-IIP UNIDEMO (ACQ and IMM) Updated in 2017 FSS microdata ASYLUM RESPER AES CVTS COD EGR TOURISM PRODCOM SBS, BOP STATISTICS EUROSTAT: 27 statistical domains use EDIT Member States and other institutions: 13 statistical domains use EDIT

EDIT principles User, programmer, administrator roles Basic EDIT use Accepted data formats

User role in EDIT Dataset upload according to a Format Create validation Job Browse Error Report or export it EDIT dataset data file VALIDATION FLOW (1-click) + validation Program validation Job EDIT dataset

Programmer role Programmer role - manages the metadata needed by the user to execute programs Implements Formats (DSD) Develops Programs for Datasets validation containing validation rules and dataset operations, prepares Lookups for code lists check Sets up the unattended mode configuration Dataset Instance (Dataset) – a collection of data rows according to the structure defined in Format; A bidimensional table composed by rows and columns: Columns correspond to the fields defined in the format; EDIT can import DSDs or code lists; EDIT acts as a client for the SDMX Registry Web Services in order to fetch DSD files and code lists data; The DSD file is broken down into EDIT components; Key families are translated to EDIT formats; Code lists are translated to EDIT Lookups; An EDIT Program is created performing lookup validations and basic checks on the dimension fields;

EDIT VALIDATION PROGRAM Programmer role Program development Custom EDIT Scripting Language designed for data validation Programs contain validation rules and dataset operations Coming in 2018: VTL translator integrated with EDIT VALIDATION RULES IN VTL EDIT VALIDATION PROGRAM

Administrator role Administrator role - manages domains, users and permissions Domain - self-contained grouping of EDIT elements available to a group of users; Contains Formats, Datasets, Programs No object in two different domains can interact with each other Users have access to all datasets and Jobs results within a Domain

Accepted data formats GESMES / BOP ITS, BOP FDI multi-year 2007, 2008, 2009 observations UNA:+.? ' UNB+UNOC:3+FR2+4D0+100929:1637+IREF000243++GESMES/TS' UNH+MREF000001+GESMES:2:1:E6' BGM+74' NAD+Z02+ECB' NAD+MR+4D0' NAD+MS+FR2' IDE+10+EUROSTAT_BOP_01 reporting' DSI+BOP_FDI_A' STS+3+7' DTM+242:201009291637:203' DTM+Z02:20072009:702' IDE+5+EUROSTAT_BOP_01' GIS+AR3' GIS+1:::-' ARR++A:FR:N:2:330:N:4A:E:9999:9999:20072009:702:0:A:F+0:A:F+0:A:F‘ ARR++A:FR:N:2:330:N:4F:E:9999:9999:20072009:702:0:A:F+0:A:F+0:A:F' ARR++A:FR:N:2:330:N:7Z:E:9999:9999:20072009:702:0:A:F+0:A:F+0:A:F' ARR++A:FR:N:2:330:N:A1:E:1100:9999:20072009:702:5824:A:F+5930:A:F+4204:A:F' ARR++A:FR:N:2:330:N:A1:E:1495:9999:20072009:702:5828:A:F+5932:A:F+4206:A:F' CSV (with or without header) /SBS, CVTS, TOURISM 9H; 2008; LT; 2; B-N_X_K642; 11930; 16236; ; ; ; ; UNIT; ; ; ; ; ; TT0; ; ; ; ; D08 9H; 2008; LT; 3; B-N_X_K642; 11930; 1001; ; ; ; ; UNIT; ; ; ; ; ; TT; ; ; ; ; D08 9H; 2008; LT; 4; B-N_X_K642; 11930; 529; ; ; ; ; UNIT; ; ; ; ; ; TT; ; ; ; ; D08 9H; 2008; LT; 30; B-N_X_K642; 11930; 17766; ; ; ; ; UNIT; ; ; ; ; ; TT; ; ; ; ; D08 9H; 2008; LT; 2; B-E; 11930; 1138; ; ; ; ; UNIT; ; ; ; ; ; TT; ; ; ; ; D08 9H; 2008; LT; 3; B-E; 11930; 104; ; ; ; ; UNIT; ; ; ; ; ; TT; ; ; ; ; D08 9H; 2008; LT; 4; B-E; 11930; 61; ; ; ; ; UNIT; ; ; ; ; ; TT; ; ; ; ; D08 FLR 2010010011 010252000405595911005909580E 01ZZZZZ 2691.966 2734482.0 0.0 2010010011 010252000405595911004009600E 01ZZZZZ 237.543 341202.0 0.0

Ways of using EDIT Standalone – running on a PC Client – server – running in a Data Centre As a web service – called by other applications (CONVAL) INPUT FILE You can use EDIT from within Internet by providing an authentication – ECAS username and password. EDIT can be downloaded and used on a computer even if it is not connected to Internet. This fits well cases of confidential data and have the advantage of managing local resources in a configured way. For example, you can run the validation program repeatedly in a quicker way. Client - server VAL REPORT VAL PROGRAM_ID Status: Available for internal usage and for testing with some countries 11

EDIT variants for external users EDIT Public Web server EDIT standalone version for local installation user from MS user from ESTAT Local authentication All components running locally: Tomcat server & PostgreSQL DB Confidentiality is preserved ECAS authentication Non-confidential data In preparation: remote EDIT validation service for confidential data

EDIT variants for ESTAT users EDIT ESTAT server in the secure environment EDIT ESTAT server in the standard environment user from ESTAT user from ESTAT Confidential data Non-confidential data Unattended mode EDAMIS back channel Integration with other systems in Eurostat

EDAMIS Integration EDAMIS can send data to EDIT by placing the files in a configurable location; EDIT detects metadata based on the EDAMIS naming convention; EDIT performs the processing in unattended mode; EDIT acts as a client for the EDAMIS Feedback Channel Web Service in order to publish the results of a job execution.

Manual feedback EDIT & EDAMIS integration – Feedback modalities Valid Data EDIT Dom. Manager Error Report Examples: SBS ENERGY FSS COD Manual feedback 16

Automatic feedback EDIT & EDAMIS integration – Feedback modalities Valid Data EDIT Examples: ASYLUM Tourism EGR RD & GBAORD ICT Error Report Automatic feedback 17

Integration of pre-validation in EDAMIS Purpose: Improve the user experience and the quality of data sent Status: Part of EDAMIS 4, available Q2 2018

EDAMIS notification

Validation report It contains: Job results – information about the job plus an overview of the validation results; Error statistics – summary of the errors; Error report – detailed list of errors; Acceptance/rejection algorithm implemented in the program.

Errors statistics

Detailed error report Pass to EDIT and show the result of the job previously launched.

Detailed statistics report

More info on EDIT Take part in one of the next EDIT Webinars: registration via ESTAT-VALIDATION@ec.europa.eu For next webinars please check the section Editing & Validation Events on: https://webgate.ec.europa.eu/fpfis/mwikis/ESSValidServ

Thank you for your attention! Any questions?