New Round of Regional data collections Deltares EMODnet Chemistry 3 New Round of Regional data collections Deltares 21 april 2019

Index Introduction Present status – Current situation ODV collections and PG database WPS and visualizations Interaction with Oceanbrowser including CDI metadata Loading new datasets: present procedure and future challenges Old and new data collections Different format Maintain products Ideas/Discussion 21 april 2019

Current situation From ODV collections to oceanbrowser 4 – step approach WPS, WMS OGC services Enriched ODV files

PostgreSQL/PostGIS database 21 april 2019

WPS and Visualisations WPS services are used to call the PG database and: generate transparent overlays for OceanBrowser interface of locations produce list of links for direct access to metadata and SeaDataNet services Show profiles, timeseries, timeprofiles 21 april 2019

Interaction with OceanBrowser Provision of transparent image as overlay of OceanBrowser interface 21 april 2019

Interaction with OceanBrowser Provision of image and link to metadata for SeaDataNet download 21 april 2019

Lesson learned 1/2 Last moment odv data products with various omissions In file formats (small errors) Inconsistent headers Parameters not available in BODC vocabs Changes in formats of ODV due to new version of ODV software 21 april 2019

Lesson learned 2/2 Getting the services up and running in the cloud hard: Slow startup due Communication with cloud provider User rights issues Technically complex Docker container with WPS services to avoid user right issues Reprogramming of the services already created Reconsiderations between work done on the server and work done on the browser (i.e. plotting on the server or in the Oceanbrowser?) 21 april 2019

Loading new datasets (until now) Description of the current ODV data collection Relation between file and database is maintained via filename link of each observation Output of previous DIVA products Deltares has a semi-automated procedure for uploading new collections 21 april 2019

Loading new datasets (from now on) Description of new data collection NEW ODV File format has been tested (successfully imported) … 21 april 2019

Differences and Effects At first glance, no critical differences for the import routines No effects on: Uploading procedure Metadata handling Columns handling 21 april 2019

Optimisations 1/2 Overall Performance of the services for visualisation of timeseries, profiles, timeprofiles on retrieving the data, but also with the interaction between Oceanbrowser (benchmarking) WP4.2 Task 1: Improving the performance and tweaking graphics of the existing advanced viewing services – Deltares + MARIS + OGS Splitting main DB observations table in multiple tables per parameter Optimization of the WPS processes Enhancing plot visualisation with interactive plotting libs (bokeh, plot.ly) 21 april 2019

Optimisations 2/2 Performance of the services for visualisation of timeseries, profiles, timeprofiles (stress test) WP4.2 Task 2: Developing dedicated maps on contaminants as part of the advanced viewing Maps Cross domain maps (bathymetry and profile in formation for specific area) Statistics (time series statistics for instance, data availability over time/depth,) 21 april 2019

Performance 21 april 2019

Ideas and discussion Deltares developed uploading scripts (OSS) Shift responsibility for uploading new data collections to EMODNet consortium Proposal: Automated upload from RCs Code on Github Provision of loading API (OSS and manuals) Ex. EMODNet Ingestion 21 april 2019

Ideas and discussion Data ingestions should be further automated Via portal Better performance by changing the datamodel Load cdi’s Load jsonb per cdi, per parameter 21 april 2019

Conclusions Need of improving performance Change of database schema and update of WPS Re-thinking of ingestion of the new data New maps and statistics 21 april 2019