Improvements in database of data products and services Giorgio Santinelli, Gerrit Hendriksen, Wouter Schoonveld First Annual Meeting EMODnet-Chemistry3 Roskilde, Denmark 12 april 2019

Index P01 terms next to P35 in ODV collections ETL procedure for automated data ingestion WPS to return data from database 12 april 2019

P01 terms next to P35 in ODV collections AWI was asked to include also P01 terms in the resulting ODV collections next to the P35 parameters. Done in a release of the ODV software, circulated to all Regional Coordinators, and which includes per CDI an extra column with all related original P01 terms (so not 1 : 1 paired to P35 terms). Deltares ingested the ODV collection by retrieving all the P01 terms. 12 april 2019

P01 terms next to P35 in ODV collections 12 april 2019 P01 terms next to P35 in ODV collections Most ideal situation is to store P01 in the aggregated ODV collections and serve the P35 value to end-users. What are the options? Full new delivery of all existing data (incl. the P35 – P01 information), so start over again with ‘clean slate’ Fix only contaminants in biota data And/or only new data with the P35 – P01 information (so from 2018 on add new data) 12 april 2019

Data model changes Current situation data model New additional table 12 april 2019 Data model changes Current situation data model New additional table CDI id Local_cdi_id Edmo location observation id cdi_id Parameterid Datetime Depth value Obs-p01 Id observationid [P01] 1:n n:1 1:n parameter Id altlabel preflabel p35 Data model changes are necessary to be able to add a list of P01 parameters to a p35 parameter, or better, to an observation. Because for each observation it could be different list of P01 parameters. The way the new additional table (obs-po1 called) is setup can be part of the discussion. With a combined foreign key of parameterid, cdi_id and date from the observation to a array of P01’s associated with that specific combination of location, parameter and observations. 12 april 2019

ETL procedure for automated ingestion For uploading and further maintenance of the database of aggregated data Deltares has developed a number of scripts and it would like to handover the responsibility to another partner. We need a staging process whereby the loading is first done and tested in another version of the database before it is made public in the production database. 12 april 2019

Proposed setup of data ingestion Regional coordinators Ocean browser automated quality control, feedback & approval Geoserver + PyWPS ETL PostgreSQL/PostGIS BackOffice for final approval? 12 april 2019

WPS to return file of observation Working on an API to easily query parts of the database As first step. Generation of a WPS service that returns data in a certain format. Test: http://emodnet02.cineca.it/wps?DataInputs=[mindatetime=2000-01-01T00:00:00.000Z;maxz=10;maxdatetime=2001-01-01T00:00:00.000Z;bbox=0,0,45,45;width=512;markersize=0.5;minz=0;p35=EPC00007]&service=wps&request=Execute&Identifier=observed_cindex&version=1.0.0 12 april 2019

WPS to return file of observation Ready to implement a WPS that can return a file of aggregated data products needed or selected. It is depending on: depth - limits Bbox Time - limits P35 Download option? What format? 12 april 2019

Thank you Questions? 12 april 2019