WFD RBMP user training 2009 WISE GIS/IT workshop BfG, Koblenz 18/ Jon Maidens, Atkins Danmark a/s

Overview

Who are Atkins? Commission contract from October 2008 to support reporting on River Basin Management Plans Main activity up until now reporting tools Core team: Jon Maidens Mette Wolstrup Claire Allaway

Goals of this mornings session Understanding of the RBMP reporting process Introduction to tools built to support reporting (parallel to user guidance) NOT to go through all the schemas but an overview on how to interpret or get help

Context Testing period June - September Schemas and tools available for testing Early support to help set up the information flow Official deadline 22nd March 2010

Sessions (I) Session 0: Overview Reporting process Supporting the process Session 1: Schemas Reporting sheets to schemas Understanding the schemas Change of schemas Final schemas Help resources Session 2: XML and MS Access DB Access database Conversion tool

Sessions (II) Session 3: QA/QC Rules Desktop validation tool ReportNet Session 4: Workflow Desktop ReportNet Session 5: Reporting GeoData Overview Good practice

Sessions (III) Session 6: Support User guidance Online resources Helpdesk Session 7: Phase 2 testing Overview

The reporting process

Submission workflow EEA/ Commission

Schemas

Schemas (I) Administrative arrangements (Schema RBDSUCA) Surface Water Bodies (Schema SWB) Groundwater Bodies (Schema GWB) Register of Protected areas (Schema ProtArea) Surface Water and Groundwater Methodologies which is divided into: (Schema SWBMethods for Surface Water Bodies) and (Schema GWBMethods for Groundwater Bodies) River Basin Management Plans and Programmes of Measures (Schema RBMP_POM) There is a WFDCommon schema which provides common elements to the other schemas which is not tested on its own.

Schemas (II) WRc CIS Working Group D, WISE GIS Group Reporting sheets do not provide all the technical specifications needed to develop the data exchange formats nor provide guidance to the data provider. These technical specifications may lead, where necessary, to adaptations of reporting requirements in order to facilitate the electronic data exchange. All Reporting sheets agreed with the Member States and involved in the WFD reporting process for 2010 have now been consolidated into ‘Guidance Document No. 21: Guidance for reporting under the Water Framework Directive’

Schemas (III) Streamlining of previous Articles 3 and 5 schemas Article 8 schemas – field names and enumeration lists Articles 11 and 13 schemas – translation and expansion of reporting sheets agreed summer 2008 Varying levels of complexity and information requirements

Schema testing 2009 First phase of schema testing in February todetermine issues or problems with population of schemas at technical level, such as: Bugs Bugs generated in translation of Reporting Sheets Misinterpretations by MS populating schema Volunteer MS (comments from AT, DE, FR, HU, SE, UK) Appr. 60 comments received Update to schemas made for the second phase and inclusion of Economics reporting sheets

Schema reporting RBD level except for RBDSUCA which is at national level

What to report? Given the time since 2005 reporting and the process to update and streamline the content of reporting, the Commission expects Member States to update all the information in Article 8 exception, as long as information not changed

Schemas dealing with Administration Areas, Water Bodies, Monitoring Stations and Protected Areas Objects

Objects(I) Each Surface Water Body has statuses for Biological, Hydromorphological and Physicochemical Quality Elements (QEs) that can change over time. For each Surface Water Body that fails to reach good status the QE causing the failure (exceedance) should be defined. Surface Water Bodies have statuses for Pollutant, Pesticide and Chemical quality elements that can change over time. For each Surface Water Body that fails to reach good status, the QE causing the failure (exceedance) should be defined together with any exemptions.

Objects(II) Each Groundwater Body has a status for Chemical QE that can change over time. For each Groundwater Body that fails to reach good Chemical Status the reason for failure should be defined by recording the chemicals that cause the failure. Each Good Chemical status needs to define the Background levels for the Groundwater Body. Each Groundwater Body has a status for Good Qualitative Status that can change over time. For each Groundwater Body that fails to reach good status, the QE causing the failure (exceedance) should be defined together with any exemptions. A Groundwater body has a set of trend status. These statuses can change over time.

Objects(III) Surface Water Bodies and Goundwater Bodies may be influenced by pressures (impacts). Each pressure and impact is of a pre-defined type. Surface Water Bodies and Groundwater Bodies may be linked to a number of Protected Area types

Schemas dealing with Methodologies Methods

Methods (I) For each River Basin District the method for identifying Surface Water Bodies and Groundwater Bodies is required based on the various categories of Water Body. For each River Basin District the typologies being used for Surface Water Bodies need to be defined. If System B typology has been used then different information is required for each category of Surface Water Body as well as a common definition for “Other System B types”. For each River Basin District the Surface Water Classification methodologies need to be defined..

Methods (II) For Surface Water Biological, Hydromorphological and PhysicoChemical Quality Element Classification an entry is required for each Quality Element sub-division, its boundaries and the typologies and intercallibration types that the boundaries apply to. For Surface Water Pollutant, Pesticide and Chemical Quality Elements an entry is required for each Quality Element that is being measured. If the national standard differs from the European Standard then details of the national standard is required. For each River Basin District the Groundwater Classification methodologies need to be defined for each pollutant or indicator that is used.

Methods (III) For each River Basin District the methods for identifying Pressures and Impacts for both Surface Water Bodies and Groundwater Bodies must be defined broken down by the Pressure Types. For each River Basin District the methods for applying exemptions for both Surface Water Bodies and Groundwater Bodies must be defined broken down by the various exemption types. For each River Basin District the way in which the results are to be interpreted for Surface Water and Groundwater Body Statuses must be described. For each River Basin District any data gaps are identified for both Surface Waters and Groundwaters

Schema dealing with River Basin Management Plan and Programme of Measures Plans

Plans (I) A River Basin Districts submits many River Basin Management Plans over time. Each River Basin District Plan defines a number of key dates and describes a Public Participation Schedule. A number of Reference Documents are associated with each River Basin District Management Plan. Each River Basin District must report on their Basic Measures, their Other Basic Measures and declare any Supplementary Measures if they exist. Each River Basin District will declare any high-level Pressures that require Supplementary Measures. None may exist.

Plans (II) Each River Basin District declares the specific Basic and Supplementary Pressure that are being applied to each specifically declared Pressure. A Supplementary Measure can embrace more than one Supplementary Pressure and vice versa. All Pressures are of a given Type. A River Basin District Management must define the costs of the Measures broken down into sub-divisions

Plans (III) If any Significant Surface Water pressures exist these are defined at a Sub-Unit level for each River Basin District. Load details are required for Significant Point Source and Diffuse Source Pressures. Extraction volumes are required for each Significant Abstraction Pressure. If any Relevant Main Groundwater Pressures exist these are defined at the RBD level. Load details are required for Relevant Point and Diffuse Source Groundwater Pressures. Extraction volumes and details are required for each Relevant Abstraction Pressure. Recharge volumes and details are required for each Relevant Recharge Pressure. Economics

Recognising the schema linkages

Reading the schemas

Structure of information asked for Structure: Simple v complex Data types: String, numeric, lat/long, Mandatory, optional, choice Codes: Defined in WFDCommon or in the schema itself

Reading the schemas

Guidance Streamlined schema user guidance available online (developed by WRc) Breaks down the schemas by element Associated annotations and enumeration lists Important notes/narrative/explanations/warnings also included

Summary Version 2.0 of the schemas available for second phase of testing over June Read the guidance Explore the schemas using a visual tool (e.g XMLSpy)

Access database

WFD database Databases managed at RBD level Database structure which supports the schemas Organising information for submission XML conversion tool links to database to create XML schemas Import datasets or manual entry

Database No forms Database tables Internal keys to hold the information together Portable

User support Naming convention Required information marked with asterisk (*) but not enforced Annotations in table field description (character limitation) Faithful to field type and size (exceptions) Code lists

Database design

Access Database Design

Table dependencies

Table design

Schema to tables

Simplifying the structure

Database map

Manual data entry – levels of data

Level 4 - Internal IDs for data management Input each previously reported Protected Area code (i.e. previously reported, before March 2010, under other directives). For each newly reported Protected Area input a code and use PROT_AREA schema to provide details.

Manual data entry Top down approach Database maintains refential integrity Code lists from parent table to help entry Multiple column code lists at level 4

Issues for testing Database design Database size Ease of data import

Data update Historic data management Need to maintain linkages Separate schema for submitting

Documentation RBMP user guidance will include database diagrams around each schema User guidance

XML conversion tool

Support for generating the XML Tool which links to Access databases and generates XML files for the appropriate schemas Download and install RBD level Important not to change database structure Independent of database Validation carried out in desktop tool

Conversion tool

Article 8 schema conversion To allow for resubmission of previous Article 8 reporting added functionality to the XML converter Select one of the three Article 8 schemas and the tool will convert it

QA/QC

Entry point for those producing their own XML

Validation rules Technical level validation ensures that numbers are entered as numbers, no characters; that the XML file is well-formed and so on. Business rule validation handles the more complex relations in the report, e.g. where data is mandatory if and only if other fields have a certain value. The validation rules are meant to ensure data meet the requirements of the reporting sheets. The rules also check that the reported data have the correct format to be stored in the WFD database model.

Element types and limits Submitted data needs to conform to the data types that have been used for each element in the schema: String: all characters allowed. Integer: only integers are allowed. Decimal: any number is allowed. Percentage: must be between 0 and 100. Date: must be a valid date in the format yyyy-mm-dd. URL: must be a valid URL format.

Required / Conditional / Optional Required: minimum occurrence is set to > 0. Conditional: minimum occurrence is set to 0 (conditional check) Optional: minimum occurrence is set to 0 (and no conditional check)

Multiple occurrences Minimum occurrence = 1, maximum occurrence = infinity Minimum occurrence = 0, maximum occurrence = infinity Minimum occurrence = 1, maximum occurrence = n Minimum occurrence = 0, maximum occurrence = n Minimum occurrence = n, maximum occurrences = n

Code checks valid codes are selected from the attached codelist defined in the WFD Common schema, or are selected from the enumeration list defined within the element itself e.g. YesNoCode / CountryCodeType / etc.

Complex validation checks Conditional Choice check Data structure check Cross-schema check All listed in the RBMP submission user guidance

Quality Logical consistency ID management

Within schema checks

Cross schema checks (I)

Cross/Within Schema checks (II)

XML validation tools

Two options for validation Desktop tool ReportNet

Desktop validation tool Installation downloaded Validates the XML files Retrieves schema from ReportNet Accesses QA/QC validation on ReportNet Same checks and output as ReportNet

Desktop validation tool

Click on the links for details

First check format

Second check – QA/QC rules

ReportNet Workflow

ReportNet validation – first check

ReportNet validation – second check

Cross/within schema checking Initiated from separate button in CDR Checks all submitted schemas

ReportNet

CDR: Central Data Repository Submission at country level for a specific obligation One obligation for WFD RBMP reporting Password protected – upload by designated MS focal point only

CDR Workflow Upload Validation performed (dependent schemas) Overwrite to correct errors Close envelope on completion Manual check by EEA/Commission Resubmission to correct (full submission only)

CDR Workflow

CDR Workflow resubmissions If envelope not released, overwrite previous submission. If envelope closed, create new envelope [RBD]_resubmission Use same file name as file replacing [country code]_[RBD]_[schema]_[date].xml Remember: full submission only Release envelope on completion

Envelope management RBD level reporting Envelopes will be pre-created One envelope for national level schema

Workflow testing CDR test area ReportNet envelope management Guidance document ReportNet QA/QC validation (cross schema checking) Closing of envelopes Letter generation

Reporting GeoData

Spatial information reporting (I) Features NOT points River Basin Districts (RBDSUCA.xsd) Sub-units (RBDSUCA.xsd) Surface Water bodies (SWB.xsd) Ground Water bodies (GWB.xsd) Protected areas (ProtArea.xsd)

Spatial information reporting (II) All objects defined in the schema will be present in the spatial dataset Attribute information for these spatial objects are defined in the associated schemas and attribute information should only be reported against the schemas

Spatial information reporting (III) Shapefile format (mandatory fields) Upload to same ReportNet folder Follow the naming convention [Country ID]_[River Basin District ID]_[Feature set name]_[Date] Positional accuracy for reported data should be better than 125 metres (1: ) and a maximum of 500 meters (corresponding to a scale of 1: ) ETRS-89.

Protected areas The WFD includes the following types of protected areas: Areas designated for water abstraction Areas designated for protection of economically significant aquatic species Recreational waters Nitrate Vulnerable Zones Special Protected Areas For the purposes of submission a separate layer (shapefile) should be provided

Quality checking Completeness Logical consistency (in schema QA/QC) Topological consistency specific to WFD General topological consistency

Logical consistency A water body can only be assigned to one RBD A monitoring station must be assigned to a least one water body A RBD must have at least one subunit (if MS doesn’t report subunits) can RBD be used as subunit Groundwater bodies must be assigned to only one RBD (even if they have parts outside of the respective RBD); Associated monitoring stations must be located within the boundaries of the respective groundwater body.

Topological consistency The objects of one type should be positionally consistent with spatial objects of related types. Reported elements should be considered as reference data and geometric consistency with other themes may be achieved if these other themes use the reported data as background data during the production or the validation of their own data, e.g. River Basin Districts. Groundwater bodies Groundwater bodies need not cover the entire territory of a country; Groundwater bodies CAN overlap; Overlapping groundwater bodies must not intersect if groundwater bodies lying upon each other are reported within one file

Topological consistency (II) Surfacewater bodies Rivers, lakes, transitional and coastal areas CANNOT overlap Rivers must not intersect (nodes at intersections) Rivers, lakes and transitional and coastal areas must be covered by subunits Rivers and lakes CANNOT overlap with coastal waters or transitional waters Outlet of each river must touch coastline A coastal area must not have gaps Coastal area must touch transitional waters, national boundaries or subunits RBD and subunits CANNOT overlap and have no gaps RBD and subunits must cover the extent of the MS RBD must contain at least one river The coastal areas must not be larger than one nautical mile from the coastline

General topological consistency Overlaps, overshoots, undershoots, slivers to be avoided. GIS software topology validation checks should be performed for connectivity issues before uploading the geometry files.

Data harmonisation Across data themes Across national borders WISE follows Inspire Hydrography data specification guidance

Centroids Data files in the reported xml schemas are related to the centroids for the associated spatial object types: Surfacewater bodies Groundwater bodies Protected areas Reported centroids are to be consistent with the reported spatial data set Quality check – reportnet tools

ID management and life cycle The location of a point features changes (e.g. if a monitoring station is moved upstream or downstream); threshold 125 m in accordance to the positional accuracy recommended for GIS datasets (according to the scale 1:250,000); The location or length of a line feature changes (e.g. if a river water body is divided or merged with another); The location or size of a polygon changes (e.g. if a River basin District is divided or merged with another).

Life cycle schema

Metadata Complete metadata information for each shapefile is needed in order to adequately describe the file, its contents and its origin.

Resources WISE GIS guidance on circa Shorter GIS reporting guidance

Documentation

Support documentation Schema user guidance Reporting user guidance: Access tool help XML Conversion tool Desktop Validation tool QA/QC validation tool help ReportNet workflow GIS reporting guidance

Online resources

Obligation and information

Support files online

Testing summer 2009

Phase two testing objectives Schemas Submission workflow Database to XML conversion tool XML validation tool Documentation Suggestions!

Phase 2 testing Scheduled to start in beginning of June Schemas available for download Help desk contact Supporting documentation and tools online Register so can be set up on ReportNet

Testing support Help desk Specific dialogue where appropriate

Capturing any issues Log What was tested and by whom What the problems were Actions required: i) schema change ii) documentation change iii) other action Reviewing the schema test data

Finally Just try the tools with a test database

Notification if wish to participate in 2. testing phase and receive notifications when tools and documents are ready Questions?