1. Training Course on EDIT
For Users

2. Outline of the module Introduction
Using EDIT - integration with other tools
Objects in EDIT for Users
EDIT Graphical User Interface
Future developments

3. A - Introduction

4. EDIT is a tool for data validation - data edit/imputation
What is data validation? - An activity aimed at verifying whether the value of a data item comes from the given set of acceptable values:
What is data editing? - The activity aimed at identifying erroneous entries and correcting them if necessary. Example: the response is missing or incorrect.

5. How EDIT works shortly? Define a format
A format contains a description of the data in a dataset
A dataset is a set of data according to a specific format
Define a format
Define a program containing rules and file operations to be executed on the dataset(s)
Uploads dataset(s) from external files
For users
Execute the job
Get the report containing errors (if any)

6. EDIT User types
'User‘ - Executes programs on datasets and accesses the reports.
'Programmer‘ - Manages the metadata needed by the user to execute programs;
Implements 'formats‘;
Implements ‘validation rules’ by means of 'programs';
Defines other operations on files by mean of 'programs';
Sets up the unattended mode configuration.
'Administrator'
Manages users and permissions.

7. 'User' type functionalities
‘Change Password’
Allows users to change their password;
‘Dataset Import/Export’
Allows users to import and export data to and from EDIT as well as monitor any ongoing import/export processes;
‘Job Execution’
Allows users to execute programs on imported datasets and view/export the results of the execution.

8. The 'User' Workflow
Data Import
Job Execution
Job Results
Data Export

9. The link between 'User workflow' and 'User interface'

10. What can we do by means of a ‘program’?
Run programs containing mainly validation rules / computations:
A1 – Single column – only a column is involved;
A2 – Multiple columns – two or more columns within a single record are involved;
B - Vertical – multiple records involved;
C - Hierarchical – multiple datasets involved.
Perform dataset operations: Copy, Merge, Alter, Aggregate, etc.
Use specialised functions like outlier detection: Terror, Hidiroglu-Berthelot, σ-Gap;
Accepted formats: SDMX-ML, GESMES, CSV, FLR.

11. multi-year 2007, 2008, 2009 observations
Accepted data formats
GESMES (BOP ITS, BOP FDI)
UNA:+.? '
UNB+UNOC:3+FR2+4D :1637+IREF GESMES/TS'
UNH+MREF GESMES:2:1:E6'
BGM+74'
NAD+Z02+ECB'
NAD+MR+4D0'
NAD+MS+FR2'
IDE+10+EUROSTAT_BOP_01 reporting'
DSI+BOP_FDI_A'
STS+3+7'
DTM+242: :203'
DTM+Z02: :702'
IDE+5+EUROSTAT_BOP_01'
GIS+AR3'
GIS+1:::-'
ARR++A:FR:N:2:330:N:4A:E:9999:9999: :702:0:A:F+0:A:F+0:A:F‘
ARR++A:FR:N:2:330:N:4F:E:9999:9999: :702:0:A:F+0:A:F+0:A:F'
ARR++A:FR:N:2:330:N:7Z:E:9999:9999: :702:0:A:F+0:A:F+0:A:F'
ARR++A:FR:N:2:330:N:A1:E:1100:9999: :702:5824:A:F+5930:A:F+4204:A:F'
ARR++A:FR:N:2:330:N:A1:E:1495:9999: :702:5828:A:F+5932:A:F+4206:A:F'
CSV (with or without header) (SBS, CVTS,TOURISM)
9H; 2008; LT; 2; B-N_X_K642; 11930; 16236; ; ; ; ; UNIT; ; ; ; ; ; TT0; ; ; ; ; D08
9H; 2008; LT; 3; B-N_X_K642; 11930; 1001; ; ; ; ; UNIT; ; ; ; ; ; TT; ; ; ; ; D08
9H; 2008; LT; 4; B-N_X_K642; 11930; 529; ; ; ; ; UNIT; ; ; ; ; ; TT; ; ; ; ; D08
9H; 2008; LT; 30; B-N_X_K642; 11930; 17766; ; ; ; ; UNIT; ; ; ; ; ; TT; ; ; ; ; D08
9H; 2008; LT; 2; B-E; 11930; 1138; ; ; ; ; UNIT; ; ; ; ; ; TT; ; ; ; ; D08
9H; 2008; LT; 3; B-E; 11930; 104; ; ; ; ; UNIT; ; ; ; ; ; TT; ; ; ; ; D08
9H; 2008; LT; 4; B-E; 11930; 61; ; ; ; ; UNIT; ; ; ; ; ; TT; ; ; ; ; D08
multi-year 2007, 2008, 2009 observations
FLR example 1
001E
001E
001E
001E
FLR example 2
E ZZZZZ
E ZZZZZ

12. B - Using EDIT - integration with other tools

13. Ways of using EDIT
As a web-based application – called by other applications;
Standalone – running on a PC;
Client – server – running in a Data Centre.

14. EDIT as Web-based application
Web-based Interface
Unified interface for both the standalone version and the server deployment;
EUROSTAT Look & Feel;
Light interface, simplified workflows.
ECAS account is needed.

15. EDIT running standalone
Downloadable package;
Standalone installation supported by Windows XP and Windows 7;
Simple installation wizard;
Full functionality;
Standard authentication is requested.

16. Client - server mode for EDIT
EDIT runs on a UNIX machine;
The current setup is EDIT installed at Eurostat & other DGs;
Contains all registered domains (= user specific workspaces) as by default imbedded;
ECAS credentials needed for external users.

17. EDAMIS integration
EDAMIS allows transmitting data files through a single entry point;
EDAMIS can send data to EDIT by placing the files in a configurable location;
EDIT detects metadata based on the EDAMIS naming convention;
EDIT performs the processing in unattended mode.

18. SDMX integration
Statistical Data and Metadata Exchange (SDMX) initiative is sponsored by seven institutions (the BIS, the ECB, Eurostat, the IMF, the OECD, the UN and the World Bank);
SDMX describes and universalises the way to exchange statistical data and metadata;
EDIT can import SDMX-ML datasets.

19. C - Objects in EDIT for Users
Datasets instantiations - lookups;
Programs, jobs

20. 1 - Dataset instantiations
Dataset Instance (Dataset) – a collection of data rows according to the structure of a format;
A two dimensional table composed by rows and columns:
Columns correspond to the fields defined in the format;
Records – no limit on size or number.

21. Dataset example – Table AES (Adult Education Survey)

22. The description of the table AES

23. Example: 'Format' – 'Dataset instantiation'

24. The same format – different datasets

25. Lookup tables – code lists
Lookup – An auxiliary dataset containing a list of values to be used for validating codes;
Code lists – usually lookup tables refer to code lists;
One can use several code lists inside the same program – as many as needed for the given data sets – 'Country', NACE, NUTS;
Several versions of the same code list can be used from within the same program, if needed.

26. 2 - Programs, jobs
Program – a set of operations to be performed on a specified dataset definition (format);
No specific dataset is associated with a program, only formats (dataset definitions) should be specified;
Job – the association between a 'Program' and concrete 'Dataset Instances';
Possible operations types of rules/checks: Single and Multiple column(s), Vertical and Hierarchical.

27. Validation report It contains:
Job results – information about the job;
Error statistics – summary of the errors;
Error report – detailed list of errors.

28. Error statistics
The error statistics are displayed in a table format and it consists of the following columns:
Rule name: The name of the program rule that failed;
No of Failures: Individual rows that the error appeared through job execution;
Rule Message: Rule’s error message as defined in the program.

29. Errors statistics Rule Name No of Failures Rule Message RC07 10
Error : This region’s code is not valid
RC185 1
Error : IntPrv does not contain the expected values
SC04 8
Error : Invalid value (if MAINSTAT in(20, 31, 32, 33, 34, 35, 36, -1) then JOBISCO should be -2)
SC05
Error : Invalid value (if MAINSTAT in(20, 31, 32, 33, 34, 35, 36, -1) then LOCNACE should be -2)
SC32
Error : Invalid value (if SpkPrv04 <> 1 AND pskPrv05 <> 1 then SpkEquip should be -2)
SC33
Error : Invalid value (if SpkPrv04 <> 1 AND pskPrv05 <> 1 then SpkPHelp should be -2)
CC04
Error : Invalid value ()

30. Detailed error report No MESSAGE SEVERITY EXP NAME PARTITION
AUXILIARY DATA 1
This region code is not valid
Error
RC07
ROW_NUMBER=1
REGION= “JP” 2
Invalid value (if MAINSTAT in(20,31,32,33,34,35,36,-1) then JOBISCO should be -2)
Warning
SC04
ROW_NUMBER=3
MAINSTAT=20 JOBISCO=2 3
Invalid value (if MAINSTAT in(20,31,32,33,34,35,36,-1) then LOCNACE should be -2)
SC05
MAINSTAT=20 LOCNACE=7 4
ROW_NUMBER=4
REGION=EG 5
MAINSTAT=31 LOCNACE=6 6
ROW_NUMBER=6

31. D - EDIT GRAPHICAL USER INTERFACE

32. EDIT - Log in

33. EDIT Home page Menu options User profile information
Here password can be changed

34. Defining dataset: import dataset
Go in >Dataset>> Import dataset
Screen part I
Select a file on your hard drive
Select a file type (CSV / GESMES / FLR / SDMX)
Reuse saved parameters
Starting line
Save properties for further use

35. Defining dataset: import dataset
Screen part II
Select a format
Reuse saved configuration
Select columns to import
Use the arrows to add remove fields
Provide a name for the new dataset
Save configuration for further use
Click to import

36. Defining dataset: import dataset
Unsuccessful import
Click to download the importing report in text format
Status is FAILED

37. Defining dataset: import dataset
Successful import with warnings
In the report, two records were skipped (lines 2 and 5)
Click to download the importing report in text format
Status is COMPLETED

38. Defining dataset: import dataset
Successful import
After importing, EDIT redirects you to the search dataset screen
Click to look at the content imported
Delete a selected dataset
Status is COMPLETED

39. Defining dataset: import dataset
Click to hidden fileds
Select fields to be hidden in the display
Hidden fields
EDIT hides the selected fields

40. Defining dataset: import dataset
Unfold the Basic filtering options
Select an logical operator
Enter a value
Select a field in the datatset (e.g. WEIGHT)
The corresponding records are filtered

41. Defining dataset: import dataset
Unfold the Advanced filtering options
Create an expression aided by the lists of fields, operators and functions
Click to apply the search criteria
The corresponding records are filtered

42. Defining dataset: import dataset
Customize your view
Export in CSV format

43. Defining dataset: search dataset
Search criteria
Restore an archived dataset
Export the dataset in CSV format
List of already imported datasets
View details of the dataset with filtering options
Delete the dataset
Archive the dataset

44. Defining dataset: Import/Export dataset
Import/Export history search
Search criteria
View details of the dataset with filtering options
List of Import/export history
Delete the dataset

45. Defining jobs: Create a job
Menu option
Search criteria
Click to create a job for this program
List of existing programs to be executed

46. Defining jobs: Create a job
Enter a name and a description
Choose the dataset to validate (if several)
Execute the job

47. Defining jobs: Create a job
When the validation is finished the date is displayed
During the validation process, only cancellation is possible
Validation is RUNNING

48. Defining jobs: Create a job
Delete the job
Copy the job
When the validation is finished the date is displayed
Click to view the results
Validation is COMPLETED

49. Defining jobs: Create a job
VIEW RESULTS OF A JOB
Click to view the Error table

50. Defining jobs: Create a job
VIEW ERROR TABLE OF A JOB
Filtering by Error fields
Unfold Basic filtering
Unfold Advanced filtering
Error message number
Export the error table (CSV)

51. Defining jobs: Create a job
Message contained into the program
Severity used into the program
Name of the rule into the program
Row number where the error occured into the dataset
Click to view details of the error
Variable values defined into the program

52. Defining jobs: Create a job
DETAILED VIEW OF ERROR
Select the dataset fileds to display
Error information
Dataset record (fields selected)

53. Defining jobs: Create a job
EXPORT ERROR REPORT OF A JOB
Click to Export the error table in CSV format

54. Defining jobs: Create a job
EXPORT ERROR REPORT OF A JOB
Choose CSV or FLR format
CSV parameters
Error fields selected
Optionally, select Ascending or Descending order for any error field
Export table

55. Defining jobs: Create a job
VIEW PROGRAM DETAILS
Content of the program

56. Defining jobs: job statistics
Menu option
Job statistics

57. Defining jobs: search job
Enter the search criteria
Delete the job
Copy the job
The corresponding jobs are displayed (all jobs if no selected criteria)
Click to view the results

58. E - Future developments
Internationalisation – to offer the translation of the menus in other languages;
Gesmes full integration (registry);
SDMX 2.1 formats.

59. Useful links To EDIT page: http://ec.europa.eu/eurostat/edit
To VIPv page: CIRCAbc -> Eurostat -> VIP Validation Project
Generic data validation and editing service: mailto:
EDIT as web – client -
CIRCAbc for:
EHSIS:
BoP:
BoP:

60. Thank you for your attention!
Questions?
Thank you for your attention!