EDIT – Eurostat’s editing tool SDMX - TWG Paris, 13-14 Dec. 2012

EDIT - features Eurostat generic data editing and imputation tool The process relies on a Scripting Language capable of expressing complex rules; Data and metadata are isolated into independent ‘Domains’; Available in a stand-alone version, a server version and a freely accessible Web version 1-way SSL and ECAS protected; The Web version allows editing statistical data by anyone registered without any software installation.

Some functional capabilities Allow to define formats by using an editor in the User Interface. Imports/Exports Metadata from/to external files; Imports/Exports from/to Oracle; Imports Auxiliary Data (lookup datasets); Accepts GESMES, CSV, SDMX-ML and flat files; Executes programs on imported dataset instances. Data set operations Editing and deterministic imputation Outlier detection TRAMO, Hidiroglou-Berthelot and sigma-gap

Integration Capabilities Integration with EDAMIS – Eurostat's Single Exchange Point (ongoing): detects incoming files and process them in unattended mode; publishes validation results to the EDAMIS Back Channel; Integrated with the Euro SDMX Registry: fetches DSDs into EDIT structures; loads code lists from the Registry;

EDIT main principles Treatment of micro and macro data; Scripting principle (symbols/placeholders); Editing seen as computations; Multi - dataset approach; Cube approach in computations.

EDIT Terminology Domain – isolated workspace inside EDIT; Dataset Definition (Format) – defines the structure of the data; Identified by name inside a Domain; Dataset – collection of data rows according to the structure of a format; Program – a set of operations to be performed on a specified Format; Key set – a set of fields which uniquely identify a row in a data set; Partition – a sub-set of data identified by a fixed sub-set of the key set; Transposition – key set fields which uniquely identifies a row inside a partition. 6

Scripting Language Capabilities Custom Scripting Language designed specifically for data editing Attempt to be as simple as possible and still enough flexible to fit the requirements of any known / analyzed domain; The programs describe the rules and are composed of a set of steps with inputs and outputs; Drawbacks: Programs difficult to be written by non-programmers; Does not follow pure cube approach.

Working with EDIT Define a format (input file characteristics); Write a program composed by: Field rules – treating a single cell Horizontal Rules – at the level of records; Vertical Rules – sub set of the data set is seen as transposed; Hierarchical rules – on two or more interlinked datasets; Dataset Operations; Import the data set instance; Import auxiliary data (other datasets, lookup tables); Establish or import program parameters Execute a job = run the program against all the imported or set data. 8

Rules - examples 1. RECORD FL171 {CONDITION (NOT isNull (A1bis)) -> inLookup (A1bis, NACE, "CODE"); ERRMSG "Rule FL171 failed for field [A1bis]: NACE rev 1.1" SEVERITY "Warning" (A1bis) ; } 2. RECORD pureRecord { PRICE := 20;} 3. RECORD conditionalRecord {CONDITION isNull(VALUE); THEN {VALUE := PRICE * QUANTITY;} ELSE { PRICE := VALUE / 5; QUANTITY := VALUE / PRICE; } 4. VERTICAL pureVertical { EXPRESSION { KEYS COUNTRY, CTYPE, MONTH, PRODUCT; // dimensions – now the data set is a cube TRKEYS COUNTRY; //divide the cube in sub-cubes by country VALUE['TOTAL'] := nvl(VALUE['TOTAL'],0); } }

Editing rules - overview 1. Cell level rules may involve determining: whether the entry of any cell is an invalid blank; whether the recorded entries are among a set of valid codes for the cell; 2. Horizontal validation rule – at the level of a record - usually specified on the basis of extensive knowledge of the subject matter of data. Example: combinations of fields which are jointly unacceptable. 3. Vertical validation rule - involve a data integrity check for entries across a collection of related records: Examples: total number of imports for a given product is equal to the sum of imports from individual countries; stock value in the beginning of month is equal to the closing stocks in the previous month; 4. Hierarchical validation rule – checks involving one or more datasets hierarchically interlinked.

Eurostat's Meta-Language SDMX - TWG Paris, 13-14 Dec. 2012

VIP on Validation (VIP = Vision Infrastructure Project) It is an ongoing project; Main task of the project: organize and optimize data editing among MSs and Eurostat for ESS data collections. Main deliverables: set of standards including a common Meta Language Additionally, a GUI driven IT system capable to generate rule sets in a meta language and standardised documentation of the validation rules understandable by business users

The need for a common language A formal unambiguous language was needed to allow rules encoding so that they can be translated into other existing syntaxes; This can help to create a more efficient production chain with responsibilities clearly assigned to the different actors

Accompanying tools Guidelines for selecting a set of rules to ensure a defined minimum standard of quality for the data exchanged; Guidelines for the assignment of responsibility in the data editing chain (Member States and Eurostat); User requirements and functional specifications for: A tool to edit and monitor compliance; A tool to specify the rules.

Scope of the Meta – Language (ML) A formal unambiguous language for encoding validation rules; Friendly to statisticians – if possible, the rules to be expressed in a human understandable way; To be able to treat both micro data and aggregate data; To allow exchange of validation rules between organizations; Able to work with cubes and with bi-dimensional data sets (e.g. micro-data).

The Meta Language (ML) – under development Information model: The simpler the information model – the more flexible the language; Data model = bi-dimensional data sets consisting of rows and columns; To allow working with all types of incoming files; Operators/functions/calculations: Statistical needs oriented; Act on data model objects = data sets; Allow expression of logical operators and computations.

Documentation of the rules The same operator/function/expression may be used for expressing different statistical meaning A rule is documented by: A list of ML - expressions used; A set of parameters; Documentation – a text provided by the user when implementing the rule