eDAMIS Validation Possibilities Validation at the Single Entry Point; based on SDMX No installation or configuration in Member States eDAMIS Web Forms: Real-Time Validation (in Production for some years) eDAMIS Web Portal: New Validation Engine (available since eDAMIS 3.0, July 2010) eDAMIS Web Application Server side validation for all eWA versions Local validation in eWA 3.1 (using rules from the server) Eurostat Unit B5 – Statistical Information Technologies GLC 25th Meeting – 12/13 October 2010

eDAMIS Validation Engine (eVE) Batch Validation for eWP and eWA Transmissions New version available in eDAMIS 3.0 For SDMX-ML formatted files XML format validation Code validation using SDMX Data Structure Definitions Data Validation Eurostat Unit B5 – Statistical Information Technologies GLC 25th Meeting – 12/13 October 2010

eVE – Data Validation Features Same Validation Rule Syntax as Web Forms Within one file and reference period Different rule sets per reference period possible Country specific rules Mandatory values, Range checks Basic expressions, comparison (+ - * / < > =) Mathematical expressions (SUM, AVG, MIN, MAX, …) Conditional checks (IF…THEN…ELSE) Logical expressions (AND, OR, NOT) Eurostat Unit B5 – Statistical Information Technologies GLC 25th Meeting – 12/13 October 2010

eDAMIS Validation Engine for … Domain Managers Based on SDMX DSDs Links to SDMX Registry Same syntax as eWF for Data Validation Less iterations for transmissions lowers workload Data Senders All transmission channels Support confidential datasets Validation transparent to data senders Full automatic transmission and level 1 validation workflow Eurostat Unit B5 – Statistical Information Technologies GLC 25th Meeting – 12/13 October 2010 5

Workflow eDAMIS Validation Engine SDMX Registry DSDs Web Service Browser SDMX Converter CSV Settings MS Database Eurostat Production Unit eDAMIS Server Validation SDMX eWP eWA Report Report Eurostat Unit B5 – Statistical Information Technologies GLC 25th Meeting – 12/13 October 2010

Successful Pilot Projects in Member States Fisheries Pilot (May 2010) Eurostat Unit E2 (Matthew Elliott) Workshops in Sweden, Latvia, UK, Romania Remote Testing in CBS, Netherlands Aviation Pilot (September 2010) Eurostat Unit E6 (Hubertus Cloodt) Workshop in Statistik Austria Good Feedback from both Pilot Projects Eurostat Unit B5 – Statistical Information Technologies GLC 25th Meeting – 12/13 October 2010

Situation in Fishery Statistics Submission of many different formats to Eurostat: Time consuming to process; Difficult to validate. Eurostat responsibility for sharing data with international organisations (FAO and regional fisheries organisations) Conflict with other formats adopted by the EC for fisheries data (DG MARE and JRC); Challenge - simplify collection and share data (reducing MS burdens, also being looked at by the Standing Committee for Agricultural Statistics); Opportunity – changes to legislation and move to new production database (MDT/Oracle). Information required is set out in technical annexes to various legislation but it allows MSs some flexibility in the format they send data in. About half send data in flat file format. The remainder use an Excel format provided by FAO and in some cases of their own design. Validation of data is particularly difficult for fisheries data. There are around 10,000 valid species codes. Many species are specific to certain geographical regions and plausibility checking of the species/area combinations can be laborious. The greater the automation of validation and particularly pre-validation, the more time saved in processing and contacts with MSs. MSs send data to ESTAT for stats purposes, JRC for scientific assessments and MARE for control. Data collection mechanisms are different and at different levels of sophistication. DG MARE under the umbrella of CFP reform are looking to update their data collection to make it more efficient and facilitate improved quality. MSs are also required to share information (sales and catch) between themselves and a common format will help this (and possibly allow development of common IT solutions). Having a common format gives greater scope for comparing what is sent to different institutions. Minimising duplication saves their time in preparing reports and ours allowing more time on data quality and making the best use of the data. SDMX as a solution feeds in to work to improve the Eurostat fisheries data collections generally and in particular as part of the revamp of the entire production process from collection to dissemination. A central part of this is the development of a new MDT production databse. Eurostat Unit B5 – Statistical Information Technologies GLC 25th Meeting – 12/13 October 2010

Overview SDMX – for Fisheries From here Various file formats Various code lists Between Member States European Institutions Other organisations Going there Single file format SDMX Shared Data Structures Harmonized Code Lists Eurostat Unit B5 – Statistical Information Technologies GLC 25th Meeting – 12/13 October 2010

Overview – Transition to SDMX in Fisheries SDMX Workshop in March: Benefit of SDMX for senders? SDMX too complicated? Links with Coordinating Working party for Fisheries Statistics established: development of code lists. Pilot Project with some Member States launched Cooperation with DG MARE Workshop Road Trip to SE, LV, UK in May, RO in September. Remote Testing for NL Fisheries Statistics Working Group - June This provides an overview of what we have achieved from formally announcing the SDMX initiative for the March Workshop (though we had give advance warning through the SCAS) The Workshop on 5 March was very detailed and technical. It was well received but it was very difficult to arrange this for the right audience – attendance was diverse – managers, IT specialists, fisheries specialists, NSIs and some fulfilling several roles. We got the impression that many of you thought that it looked overly complicated – “you have taken something simple” Workshop launched the Pilot for testing of generation and processing of SDMX in MSs. Was achieved in spite of the Icelandic Volcano (whose name I will not try to pronounce) This was also used to examine the possibilities for wider SDMX use – by Commission Services and within MSs. Useful meeting in London with the UK MMO. Visits to SE, LV and UK Testing packs also sent to NL and RO. Need to discuss results bilaterally and also possibility of additional visits (RO) Eurostat Unit B5 – Statistical Information Technologies GLC 25th Meeting – 12/13 October 2010

Pilot - Technical Scope Catch for Major Fishing Area 27 (NE Atlantic) used for the pilot; eDAMIS will be used as transmission system; eDAMIS Validation performed for: Format validation, Code list validation ( DSD) Value validation: Detect some species that should not be in the area Check for mandatory fields and duplicates if possible Eurostat Unit B5 – Statistical Information Technologies GLC 25th Meeting – 12/13 October 2010

Technical Workflow – Tested during Pilot SDMX Converter CSV to SDMX using the DSD eDAMIS Upload SDMX to web portal eDAMIS Validation Get validation report Eurostat Unit B5 – Statistical Information Technologies GLC 25th Meeting – 12/13 October 2010

Feedback from Member States SDMX is no “Rocket Science” Number 1 Issue is to harmonize Code Lists The SDMX Registry is seen as a useful tool to manage data structures and code lists centrally Deadlines and Reports should be harmonized between organizations A Single Entry Point for data for the whole Commission would make life easier for data senders Tools and Guides were easy to use The information on the Validation reports was considered easy to interpret and use for data correction Eurostat Unit B5 – Statistical Information Technologies GLC 25th Meeting – 12/13 October 2010

eDAMIS Validation Engine – Live Demo Based on pilot project dataset View of a Data Sender Eurostat Unit B5 – Statistical Information Technologies GLC 25th Meeting – 12/13 October 2010