Synthesis of the technical activity and State of development of the on-line system V1 By Dick M.A. Schaap – technical coordinator Madrid, March 09

SeaDataNet Participating Countries 49 Partners = 40 Data Centres + ICES + IOC-IODE + JRC + specialists in statistics, products and informatics

SeaDataNet Infrastructure objectives SeaDataNet aims to set up and operate an efficient Pan-European distributed infrastructure for managing marine and ocean data by connecting: 40 National Oceanographic Data Centres (NODC’s), national oceanographic focal points, and ocean satellite data centres, in Europe these Data Centres are mostly divisions of major national marine research institutes and based in 35 countries, surrounding the European seas SeaDataNet aims to ensure a comparable quality of data sets and to make data sets easily accessible on-line through a unique portal, while the data sets are stored and managed at the Data Centres.

SeaDataNet Infrastructure versions Version 0 = Continuation and maintenance of existing Sea-Search systems and population of the Common Data Index (CDI) V0 system Version 1 = Harmonised and upgraded metadatabases adopting ISO standard; Transparent data access involving all Data Centres via implementation of Common Data Index (CDI) V1 system and downloading services. Version 2 = Adding OGC viewing services and further virtualisation of data access.

SeaDataNet Technical Task Team Oversees and coordinates the technical developments Includes 11 core technical partners 3 Meetings in the third year of SeaDataNet: Paris, France May 2008 Hamburg, Germany November 2008 Madrid, Spain, March 2009 Bilateral meeting RIHMI – MARIS concerning DM - RSM: Obninsk, Russia, August 2008

SeaDataNet Technical Task Team Methodology: Agenda with relevant topics, leading speakers, open discussion and brainstorming, possible subgroups for further explorations and analyses, resulting in technical specifications of system architecture and its modules, followed by implementation, evaluation and operation All presentations and documents are included in the SeaDataNet Extranet in the TTT section. Note: All Project Partners are encouraged to visit this regularly.

SeaDataNet User’s portal schema V1 services for metadata Data downloading services Visualization services (WMS) for V2 Ifremer Database BODC Database BSH Database... Download managers in Data centers CSREDIOS EDMED CDI Shopping basket Requests status manager datametadata CSR CDI Project info Metadata & Data catalogues Data request Status of request General requestMetadata request Organisation + data source id Data download at BODCat BSHat MARIS + EDMERP Entry point for access hits EDMO EDIOSEDMED EDMERPEDMO Software Vocabularies StandardsCross search AAA Registr. My transact. User Register User registration

SeaDataNet Interoperability Interoperability is the key to distributed data management system success. This is achieved in SeaDataNet via: Using common quality control protocols and flag scale Using common and controlled vocabularies, including international content governance Adopting the ISO metadata standard for all metadata directories Providing XML Validation Services to quality control the metadata maintenance Providing standard metadata entry tools Using harmonised Data Transport Formats (NetCDF, ODV ASCII and MedAtlas ASCII) for data sets delivery Adopting of OGC standards for mapping and viewing services Using SOAP Web Services in the SeaDataNet architecture Organising training and instruction workshops for transfer of expertise and hands-on training

SeaDataNet Quality Control Guideline A guideline (V1) of recommended QC procedures, reviewing NODC schemes and other known schemes (e.g. WGMDM guidelines, World Ocean Database, GTSPP, Argo, WOCE, QARTOD, ESEAS,SIMORC, etc.). QC methods for CTD (temperature and salinity), Current meter data (including ADCP), Wave data, Sea level data A scheme of QC flags to be used in SeaDataNet.These flags are for assigning to individual data values. They are not for allocating to whole data series, or to accompanying information. Compiled in discussion with IOC, ICES and JCOM, to ensure an international acceptance and tuning. Important feedback from the joint IODE/JCOMM Forum on Oceanographic Data Management and Exchange Standards (January 2008), joined by SeaDataNet and international experts to consider on-going work on standards and to seek harmonisation, where possible. Now extending the guideline with QC methods for surface underway data, nutrients, geophysical data, and biological data. V2: June 2009.

SeaDataNet Common Vocabularies Use of common vocabularies in all metadatabases and data formats is an important prerequisite towards consistency and interoperability. Set-up and population of Common Vocabularies. The SeaDataNet Vocabulary service is based upon the NERC DataGrid (NDG) vocabulary Web service. For end-users there is a vocabulary Client Interface for searching and browsing and to export selected entries in csv format. The Web service is compliant to WS Basis Profile 1.1, which is adopted as standard for all Web services in SeaDataNet. Content governance of the vocabularies is very important and is done by a combined SeaDataNet and MarineXML Vocabulary Content Governance Group (SeaVoX), moderated by BODC, and including experts from SeaDataNet, MMI, MOTIIVE, JCOMMOPS and more international groups. SeaVox operates by mailing list server.

SeaDataNet Common Vocabularies Upgraded interface in SeaDataNet style Web services upgraded to be compliant to WS Basic Profile 1.1 Expanding the number of lists Web services available via BODC and user client via MARIS

SeaDataNet infrastructure V1 Discovery services EDMED - Data Sets EDMERP – Research Projects CSR – Cruise Summary Reports EDIOS – Monitoring systems EDMO – Marine Organisations CDI – Common Data Index

SeaDataNet Discovery services TTT Activities undertaken for: Reviewing and streamlining the logical formats of each of the Directories Expanding the number of Common vocabularies, further population and upgrading of Vocabularies Web services Defining XML schema’s and formats, using the ISO metadata standard as basis Defining and developing maintenance modalities for each of the Directories

SeaDataNet Discovery services TTT Activities undertaken for: Defining and developing new User Interfaces for each of the Directories Defining and implementing XML Validation Web services, that will be used to validate XML output from data centres, before import into the public Directories Developing Web services for the Directories

Vocabs EDMO … … … … CDI EDMERP CSR EDMED EDIOS SeaDataNet : Formats review Review of the format and use of common vocabularies for each of the Directories to achieve harmonisation and integration, and paving the path to data access via the CDI Data Access …

SeaDataNet XML Schema’s The ISO content model is the basis for the XML formats and exchange schema’s (XSD). Guidance document has been prepared on how to use XML for SeaDataNet, including how to declare references to Common Vocabularies, EDMO and EDMERP For each Directory has been prepared: Description of the format and XML tags XML Schema XML example file These documents are available at the SeaDataNet Extended XML Schema’s have been prepared, using Schematron and OCL to support the checking of mandatory fields, use of codes from the Common Vocabularies and use of organization codes from EDMO. These schema’s are used in the XML Validation Web services

SeaDataNet Maintenance modalities Maintenance: Depending on the Directory, the following maintenance modalities are provided: Online maintenance via online Content Management System (incl master editing options) XML export from local system Local XML export can be produced by partners via: own software using upgraded version of MIKADO software, that also will replace the present MS Access tools for EDMED and EDMERP MIKADO can also be used for bulk production of CDI XML files from the partner’s database(s)

SeaDataNet Maintenance modalities DirectoryOnline CMS by partners XML exchange via use of MIKADO at partners XML exchange via local system at partners Online CMS by moderator XML Validation service EDMED (BODC = authority) SDN DC’s initial content conversion free text => vocabs via ‘sandbox’ Stand-alone incl. local storage + QC – loop for content Synchronisation Generated from in-house system Validation of XML syntax and use of Vocabs, EDMO and EDMERP EDMERP (MARIS = authority) SDN DC’s manage their national records + ‘sandbox’ for institutes Stand-alone incl. local storage Generated from in-house system Validation of XML syntax and use of Vocabs, EDMO and EDMERP CSR (BSH = authority) Online entries by chief scientists Stand-alone incl. local storage Generated from in-house system Validation of XML syntax and use of Vocabs, EDMO and EDMERP

SeaDataNet Maintenance modalities DirectoryOnline CMS by partners XML exchange via use of MIKADO at partners XML exchange via local system at partners Online CMS by moderator XML Validation service CDI (MARIS = authority) Tool embedded in local system Generated from in-house system Validation of XML syntax and use of Vocabs, EDMO and EDMERP EDIOS (BODC = authority) SDN DC’s initial revision + new entries via ‘sandbox’ Stand-alone incl. local storage Generated from in-house system Validation of XML syntax and use of Vocabs, EDMO and EDMERP EDMO (MARIS = authority) SDN DC’s manage their national records Vocabularies (BODC = authority) BODC with SeaVox governance mailing list User Register + AAA service (IFREMER = authority) SDN DC’s manage their national records

SeaDataNet Content Management Systems EDMERP CSREDMED EDMO

SeaDataNet MIKADO Java tool Available under multiple environments : Microsoft : Windows 2000, XP, VISTA APPLE Unix - Solaris Linux MIKADO Java code Native Drivers MYSQL ORACLE POSTGRES SQLServer Bridge Drivers using Microsoft ODBC (ACCESS, EXCEL, SQL SERVER) DATABASE JDBC Java DataBase Connectivity EXCEL File XML files for SeaDataNet catalogues CSR EDMED EDMERP CDI [EDIOS] Manual Automati c Other Drivers Downloaded from ad hoc Websites (Copied in the dist/lib MIKADO directory)

SeaDataNet new User Interfaces EDMERP CSR EDIOS EDMO

SeaDataNet Discovery services - status Reviewing and streamlining the logical formats of each of the Directories – READY Expanding the number of Common vocabularies, further population and upgrading of Vocabularies Web services – READY Defining XML schema’s and formats, using the ISO metadata standard as basis – READY (EDIOS READY SOON) Defining and developing maintenance modalities for each of the Directories: EDMO, EDMERP, and CSR – READY VIA ONLINE CMS EDMED upgrading V0 to V1 – READY VIA ONLINE CMS EDMED, EDMERP, CSR and CDI – READY VIA MIKADO V1.5 Common Vocabularies and AAA - READY VIA MASTER CMS IMPORT OF XML FOR EDMERP, CSR, EDMED, CDI - READY EDIOS – TO EXTEND MIKADO WITH THE NEW XML Defining and implementing XML Validation Web services, that will be used to validate XML output from data centres, before import into the public Directories – READY FOR CDI, EDMED, EDMERP and CSR ; EDIOS SOON

SeaDataNet Discovery services - status Defining and implementing new User Interfaces READY FOR EDMO, CDI, EDMERP, CSR, EDIOS, Common Vocabularies and AAA services UNDER DEVELOPMENT FOR EDMED Content upgrading from V0 to V1 – CONTINUED INPUT AND SUPPORT NEEDED FROM THE NODC’s !! EDMO READY EDMERP ALMOST READY AND V1 LAUNCHED CSR WELL UNDERWAY AND V1 LAUNCHED, BUT NEEDS MORE INPUT BY NODC’S EDMED ALMOST READY, SOME PARTNERS ARE BEHIND SCHEDULE EDIOS – SO FAR ONLY DONE FOR UNITED KINGDOM AND BLACK SEA AS PILOTS => OTHER PARTNERS TO START SOON ONCE EXTENDED MIKADO AVAILABLE (END 2009) – COOPERATION WITH EUROGOOS REGIONS! Note: CDI upgrade is undertaken as part of the Data Access activities;

SeaDataNet Discovery services - status Implementing Web services READY FOR EDMO, EDMERP, Common Vocabularies and AAA services LATER ALSO PLANNED FOR EDMED, CSR, CDI and EDIOS

SeaDataNet : Training and instruction Training Workshop has been held at IODE in Oostende – Belgium at from 16 to 19th June 2008 for all SeaDataNet data managers: To train partners in the use of the new V1 formats, interfaces and maintenance tools (MIKADO, online CMS, Web services, Validation services, Vocabularies) To train partners in performing data quality control and assessments, using ODV software To train partners in analysing and making data presentations, using ODV – DIVA software To present and explain the new CDI V1 – data access procedure and system

SeaDataNet Data Access Services SeaDataNet aims to make data sets easily accessible on-line through a unique portal, connecting 40 National Oceanographic Data Centres (NODC’s), national oceanographic focal points, and ocean satellite data centres, in Europe

SeaDataNet Data Access components Common Data Index (CDI) system including a Shopping and Transaction mechanism Using harmonised Data Transport Formats for data sets delivery SeaDataNet Data Policy and User Licence Authorisation and Authentication Services, based upon a Central User Register and vocabulary of Data Access Restrictions

Authentication, Authorization and Administration Metadata systems and website with standards and tools are public domain Single Sign On system required for access to distributed databases User must register once in order to get a personal login - password Web form to provide necessary information User agrees with “SeaDataNet User Licence” After processing, login/password sent by ( check) Licence is part of the SeaDataNet Data Policy, that is intended to be fully compatible with the Directive of the European Parliament and of the Council on public access to environmental information, the INSPIRE Directive, IOC, ICES, WMO, GCOS, GEOSS and CLIVAR data principles. Choice in V1 for CAS system (= Centralized) to provide AAA Web services NODCs intermediate in the registration of their national users to the SeaDataNet central user register

SeaDataNet AAA service Interface for central log-in

SeaDataNet Common Data Transport Formats V1: data sets are accessable by downloading services. Delivery of data sets to users requires common data transport formats, which interact with other SeaDataNet standards (Vocabularies, Quality Flag Scale) and analysis & presentation tools (ODV, DIVA) The following formats have been defined: SeaDataNet ODV4 ASCII for profiles, time series and trajectories SeaDataNet MedAtlas as optional extra format. NetCDF with CF compliance for gridded data sets ODV4 and MedAtlas have been outfitted with a SeaDataNet semantic header International cooperation is underway from SeaDataNet with the CF community and UNIDATA for a common NetCDF format (Core Data Model – CDM) for the oceanographic and meteorological domains, including a semantic header

SeaDataNet : Conversion Tools NEMO tool ready for conversion of ASCII files to SeaDataNet ODV and MedATLAS Med2MedSDN V1.0 ready for conversion of MEDATLAS files without the SeaDataNet extensions to MEDATLAS files with SeaDataNet extensions (SeaDataNet MEDATLAS) Transformation Web Service for lat – lon coordinates to Marsden Squares and vice versa

CDI is a fine-grained index to individual data measurements (such as a CTD cast or moored instrument record). This is in contrast to EDMED, which is an inventory of data sets, each of which comprises a large number of measurements. CDI XML Schema is based upon the ISO standard CDI V0 = Central and homogeneous metadatabase, but for Data Access users are confronted with the different user interfaces of the SeaDataNet Data Centres (which can provide online search and access / ordering facility / request) CDI V0 coverage: > CDI entries from 36 Data Centres from 29 countries SeaDataNet Common Data Index (CDI)

SeaDataNet CDI V0 – Data coverage – points

SeaDataNet CDI V0 – Data Access example RIHMI-WDC – Russia

Central and homogeneous metadatabase plus transparent access to the distributed data sets via a unique user interface at the SeaDataNet portal and downloading services SeaDataNet Delivery Services – CDI V1

SeaDataNet CDI V1 system components CDI V1 Metadatabase and User Interface: Searching and browsing of metadata of data sets, managed by SeaDataNet Data Centres Central User Register - Web service: Details of users, their organizations and addresses, license period, Id-Passwords, SeaDataNet Roles Shopping Basket – User Interface: Preparing a user request of multiple data sets, handling validation of users and roles via the Central User Register, and routing requests to the Request Status Manager Request Status Manager – Web service + User Interface: Processing and administration of all requests and data deliveries (downloads), for users in communication with data centres. For data centres to oversee all transactions.

SeaDataNet CDI V1 system components Data Centre Download Manager – Java Tool: Handles communication with Request Status Manager and takes care that requested files are made ready for downloading by User (if OK) in the agreed formats via a local website address at the Data Centre

SeaDataNet Download Manager V1.1 Functions Communication with RSM Web service about process progress Processing order lists => retrieving requested data files Retrieving data sets from: file management systems database systems (e.g. Oracle, MySQL, MS-SQL, Sybase, ….) If data sets are retrieved from a database => export of data files in SDN ODV4 format Note: Data sets from a file management system must be pre- processed in SDN transport formats (ODV4 and optional MedATLAS for ASCII type data, or NetCDF (CF) for grid type data) Combining data set files in daily zip files in a user download directory Generating dynamic user download webpage, following SDN template, to give user overview and option of downloadable zip files

SeaDataNet CDI V1 – Data Access example Search Results Include in Basket Shopping list Submit + Authentication Request Confirmed Data Check Status In RSM Ready at DC x Download SDN format

SeaDataNet CDI V1 - status CDI V1 database, interface & transaction system - READY; Mikado software tool for CDI V1 generation + Validation service – READY Download Manager (DM) V1.1d – READY Full system tested and operational – READY Alternative without DM = V1 Interim - READY 22 Data Centres well underway with installing the DM V1.1 and configuring their infrastructure for being a full V1 data centre. So far 9 Data Centres fully operational with > datasets accessable Other Data Centres are requested to start implementing CDI V1 soon.

SeaDataNet CDI V1 – Quick Search interface Using an intuitive approach which is easy for users

SeaDataNet CDI V1 – Quick Search interface With Show / Hide Map option for display of search results

SeaDataNet CDI V1 – Extended Search interface More search capabilities, but more complicated for first-time users

SeaDataNet CDI V1 shopping principle

SeaDataNet CDI V1 – Some search results

SeaDataNet CDI V1 – Shopping Basket

SeaDataNet CDI V1 – Request Status Manager (RSM) Log-in page

SeaDataNet CDI V1 – RSM: logged in as User Overview of requests per Data Centre and Status

SeaDataNet CDI V1 – RSM: logged in as User Status = Ready for Download + Download option

SeaDataNet CDI V1 – RSM: logged in as User User personal Download webpage at Data Centre

SeaDataNet CDI V1 – Monitoring and Tracking All submitted requests, their users, the data sets concerned, and processing of these requests is administered in the Request Status Manager SeaDataNet Data centres can also log on to the Request Status Manager and oversee all requests, handle ‘tbd’ requests and prepare a number of online reports.

SeaDataNet CDI V1 – RSM: logged in as data centre Data centre RSM: overview Standing Orders

SeaDataNet CDI V1 – further implementation 22 Data Centres well underway with installing the DM V1.1 and configuring their infrastructure for being a full V1 data centre. So far 9 Data Centres fully operational with > datasets accessable Other Data Centres are requested to start implementing CDI V1 soon. Preparing CDI V1 XML records following the upgraded CDI XML schema and using the new MIKADO software Installing the Download Manager software Choosing between pre-processed data files OR conversion via the Download Manager Converting data files to SeaDataNet formats, in case of choice for pre-processed mode

SeaDataNet CDI V1 interim as temporary alternative All Data Centres must populate the CDI V1 database: Preparing CDI V1 XML records following the upgraded CDI XML schema and using the new MIKADO software V1 interim = no installation of Download Manager, but data centres will be informed by the Request Status Manager by of all requests Data centres have to process all requests manually using the Request Status Manager Delivering data sets in SeaDataNet Formats (conversion via new NEMO tool) – pre-processing required. Effect is, that users will experience a common approach to all data centres. The difference is behind the scenes. Thereafter centres should move to full V1 status by installing DM

SeaDataNet V1 presentation and analysis tools Ocean Data View (ODV) General data analysis and visualization software >10,000 registered users Upgrading ODV software into ODV4 for: Extending ODV‘s graphical display capabilities and interactive controls for automatic and visual quality control and data quality flagging Seamless connection to SeaDataNet output: SDN ODV4 data formats, quality flag scale Integration of DIVA gridding software Important tool for the SeaDataNet regional Data Products DIVA software (Data-Interpolating Variational Analysis) allows to spatially interpolate observations on a regular grid in an optimal way. The analysis is performed on a finite element grid allowing for a spatial variable resolution and a good representation of the coastline and isobaths.

SeaDataNet V1 presentation and analysis tools

SeaDataNet Website Upgraded The SeaDataNet Website has recently been upgraded and users can find all presented services including background information.

Thank you