JERICO WP5 Data management and distribution Alessandro Crise Istituto Nazionale di Oceanografia e di Geofisica Sperimentale JERICO Project Kick-Off meeting Maison de la Recherche Paris, May 2011

JERICO WP5 Data management and distribution: main goal The main goal is to design the JERICO Data Flow and Management infrastructure based on the implementation of an end-to-end ‘ open and free ’ conduit from data to users and upon the guidelines proposed in the GOOS Report 148 (2005). The implementation will be consistent and complementary with the major European initiatives for the establishment and coordination of infrastructures for the management and distribution of data and products in Europe which can use JERICO data (SeaDataNet, EuroGOOS Regional Centres, MyOcean). YET ANOTHER DATA MANAGEMENT SYSTEM?: NO!

WP5 Structure JERICO will need to ensure that the flow of real-time and delayed mode will be reliable, accessible and easy to distribute. The WP5 structure : Creating value for measured data: from raw data issued from the sensors to validated data stored and marked. Facilitating data and meta-data flow to existing data repository infrastructures (SeaDataNet); Helping in delivering real-time JERICO data to ROOSs and My Ocean

Task 5.1: Quality assurance: create value for measured data Task responsible: OGS Partners: HCMR Duration M1-M42 This task is aimed at adding value to observations by associating them with characteristic uncertainties. By definition, the “expected” accuracy associated with an observation is an attribute of the measurement chain that produces it.

Task 5.1: Quality assurance objectives The objectives are : Collecting pre-existing knowledge regarding the key identified sensors and critically revise state-of-art literature on in lab and in situ performances of the most popular sensors (in conjunction with WP4) defining procedures for ascribing a total uncertainty to measurements that will account for the actual circumstances and conditions under which the measurements were taken. Testing and verifying uncertainty computational procedures

Task 5.1: Quality assurance methodologies. Methodology A novel model of a-priori uncertainty will be proposed as a sum of fixed and time-dependent source of error and uncertainty (precision and accuracy, drifts, experimental uncertainties, the representativity error, etc.) The variables temperature, salinity and fluorescence/chlorophyll will be considered at first. Controlled experiments will be performed in order to document and validate the different uncertainty components composing the final uncertainty estimates. All the relevant assessments carried out in WP4 will be included as basic information for the combined uncertainty estimation. WP4.1 will provide an updated summary of these achievements to WP5.1, no later than the dates set down in the WP4 D4.2 delivery schedule (M14)

Task 5.2: Harmonization of Delayed Mode data management procedures with SEADATANET Task responsible: IFREMER, Partners: HCMR, MUMM, OGS Duration: M1-M42 This task will define, establish and oversee the data management infrastructure for dealing with delayed-mode data in JERICO. The goal is to provide JERICO users with relevant informations for seamless data integration with the EU SEADATANET initiative.

Task 5.2 Methodology - Survey the existing delayed-mode data handling practices of JERICO partners. -Formulate a viable proposal for a common JERICO delayed-mode data management platform that can serve to reinforce the SEADATANET effort; -the guidelines established by DG-Mare/EMODNET and WISE-Marine will be also taken in account when doing this. - Create common vocabularies for JERICO delayed-mode data formats and meta-databases, building upon the work that has already been done in SEADATANET. -Implement delayed-mode data and metadata formats that are compatible with those of SEADATANET (in compliance with the EU INSPIRE Directive); -the Sensor Web Enablement family of the OGC consortium family of standards (SensorML, Sensor Registry, O&M) will be considered in the implementation. - Ensure easy sharing and secure archival of delayed-mode data within JERICO by employing common data transport formats and storage criteria making use of the experience gained by SEADATANET.

Task 5.2 Metadata implementation Discovery metadata To let all know what have been deployed during the project (where, what, how, who…) To help to retrieve data related to observatories To be compliant with European directives (Inspire) Enabling technology Use of SeaDataNet EDIOS directory to describe observatories Typing / Generation of metadata using SeaDataNet procedures and tools (Mikado) complemented with automatic tools developed in Emodnet-Physics to better use the RT Netcdf files provided in RT by the ROOSs

Task 5.2 Data implementation Delayed mode data Delayed mode quality check Uniform file format (based on OceanSites NetCDF, compatibility with CF convention) + distribution in ASCII flat file format (Ocean Data View) Full metadata (Common Data Index) generated by Mikado for long term reuse Data access via web portal (all users)

Task 5.3 Harmonization of Real Time data anagement procedures with MyOcean and EuroGOOS & the institution of data access services for JERICO TOP activities Task responsible: IFREMER, Partners: CNR, NIVA, IMR, HCMR,PUERTO, SMHI Duration: M1-M42 This task will define, establish and oversee the data management infrastructure for dealing with real-time data in JERICO. The goal is to provide for seamless integration with the EuroGOOS and EU Marine Core Service’s MyOcean component and in order to render JERICO real-time data accessible for use in generating downstream services and products.

Task 5.3 Methodology Survey the existing real-time data handling practices of JERICO partners. Formulate a viable proposal for a common JERICO real-time data management platform that can serve to reinforce EuroGOOS and MyOcean Create common vocabularies for JERICO real-time data formats and metadatabases, based on any existing recommendations by MyOcean/EuroGOOS for doing this. Implement real-time data and metadata formats based on any existing recommendations by EuroGOOS and MyOcean for doing this. Reconcile, wherever possible, the data quality assurance procedures/protocols for real-time data amongst the JERICO partners (link to WP4, WP5.1). Ensure easy exchange of real-time data within JERICO by employing common data transport formats and storage criteria, based on any existing recommendations by EuroGOOS/MyOcean for doing this.

WP5.3 Implementation Real-Time data Real-Time Quality check based on EuroGOOS/MyOcean procedure that would need to be extended to new parameters whenever possible Uniform file format (based on OceanSites NetCDF, compatibility with CF convention) ftp file server (with an index file) per region Oceanotron (OpenDap protocol) for direct access by remote software

WP5 in JERICO Data flow WP5 act as mediator in JERICO Data Flow Model A priori quality assurance (WP4) Data management procedures (WP5) Time-dependent data uncertainty estimation Real-time data management Delayed mode data management Value added products and data dissemination (WP6) EuroGOOS ROOSs engagement (WP2) Demonstation of data provision (WP7) SeaDataNet MyOcean Emodnet Physical Parameters EuroGOOS

Black Sea IOBAS/Bulgaria All parameters Arctic: IMR / Norway All parameters Baltic Sea: SMHI/Sweden Temp & Salinity : BSH/Germany Current: SMHI/Sweden Sea Level: DMI/Denmark Bio : FMI-Syke/Finland Mediterranean Sea : HCMR/Greece Mooring : HCMR/Greece XBT/CTD: ENEA/Italy Drifter & Argo: OGS/Italy Glider and Argo: Coriolis/France South West Shelves: Puertos Del Estado/Spain Mooring : PdE Underway data: Coriolis/France North West Shelves: BSH/Germany Temp & Salinity : BSH/Germany Current: SMHI/Sweden Sea Level: DMI/Denmark Global Ocean : Coriolis/France Link with international network s: Coriolis/France Argo: Coriolis,BODC,BSH European Vessels: NIVA/Norway v MyOcean Insitu TAC implementation of EuroGOOS regional approach

MyOcean In-situ TAC role Limited number of parameters: Physical: T&S, current, sea level Biogeochemical: Chlorophyll/Fluorescence, Oxygen, Nutrients Integrate in-situ data in product accessible through global and regional portals Common format Common NRT QC Common Quality flags Common distribution tools Single access point Ensure a minimum level of quality on the data delivered In Near real time ( 24h to a week) In delayed mode Assessment of the products at basin scale

The European Commission, represented by DG MARE, concluded a service contracts aiming to: Create a prototype components of the European Marine Observation and Data Network (EMODnet). Provide access to archived and real-time data on physical variables in Europe's seas and oceans determine how well the data meet the users needs. EMODNET PHYSICAL PARAMETERS

THE EMODNET PP PILLARS EMODnet PP EMODnet PP aims to bridge the communities behind MyOcean, EuroGOOS and SeaDataNet in order to strengthen their position inside the Commission agenda on IS and expand in future the participation to all partners MyOcean EuroGOOS SeaDataNet JERICO WP5

NEW FEATURES Add new stations and models Add profiles Make use of the results from ECOOP Add data from FerryBox and RV Common quality control Link to EDIOS SEPRISE data in SDN ODV format Put data on GTS E-HYPE EuroGOOS SEPRISE network

Cooperation between MyOcean in-situ TAC and SeaDataNet MoU signed Beginning 2010 Observing systems Marine governmental agencies and research institutes (including NODCs) Other observations (e.g. scientific cruises…) International organisations (IOC/IODE, ICES, ICSU World Data System…) delayed-mode real-time Metadata (e.g. EDIOS) Metadata (e.g. EDIOS) Vocabularies Metadata ( EDIOS, CDI, CAMIOON, SensorML) Yearly regional products (climatologies, datasets) Yearly regional products (climatologies, datasets) Re-analysis users Users Operational users MyOcean in-situ TAC SeaDataNet Direct users Data integration (QC, « best copy », archival, re-analysis datasets Data integration (QC, « best copy », archival, re-analysis datasets Technical WG

JERICO role in the European Data Management landscape International organisations (IOC/IODE, ICES, ICSU World Data System…) delayed-mode real-time Metadata (e.g. EDIOS) Metadata (e.g. EDIOS) Vocabularies Metadata ( EDIOS, CDI, CAMIOON, SensorML) Yearly regional products (climatologies, datasets) Yearly regional products (climatologies, datasets) Re-analysis users Users Operational users MyOcean in-situ TAC SeaDataNet Direct users Data integration (QC, « best copy », archival, re-analysis datasets Data integration (QC, « best copy », archival, re-analysis datasets Technical WG Observing Network Information & evidence pipeline JERICO Coastal Observatory Network (WP7) Web Portal real-time Assessment database ‘EMECO’ like Data XML, CSV ‘EMECO- like’ Web – enabled Datatools (WP6)

WP5 Milestones/Deliverables

ACTIONS To propose a formal agreement with SDN, EuroGOOS/MyO for JERICO data provision, to ensure more visibility to JERICO project To find an agreement with DG MARE/EMODNET for the availability to JERICO of the developments achieved/to be achieved in EMODNET Physical Parameter lot.

Thanks!