1. New Round of Regional data collections Deltares
EMODnet Chemistry 3
New Round of Regional data collections
Deltares
21 april 2019

2. Index Introduction Present status – Current situation
ODV collections and PG database
WPS and visualizations
Interaction with Oceanbrowser including CDI metadata
Loading new datasets: present procedure and future challenges
Old and new data collections
Different format
Maintain products
Ideas/Discussion
21 april 2019

3. Current situation From ODV collections to oceanbrowser
4 – step approach
WPS, WMS
OGC services
Enriched ODV files

4. PostgreSQL/PostGIS database
21 april 2019

5. WPS and Visualisations
WPS services are used to call the PG database and:
generate transparent overlays for OceanBrowser interface of locations
produce list of links for direct access to metadata and SeaDataNet services
Show profiles, timeseries, timeprofiles
21 april 2019

6. Interaction with OceanBrowser
Provision of transparent image as overlay of OceanBrowser interface
21 april 2019

7. Interaction with OceanBrowser
Provision of image and link to metadata for SeaDataNet download
21 april 2019

8. Lesson learned 1/2
Last moment odv data products with various omissions
In file formats (small errors)
Inconsistent headers
Parameters not available in BODC vocabs
Changes in formats of ODV due to new version of ODV software
21 april 2019

9. Lesson learned 2/2
Getting the services up and running in the cloud hard:
Slow startup due
Communication with cloud provider
User rights issues
Technically complex
Docker container with WPS services to avoid user right issues
Reprogramming of the services already created
Reconsiderations between work done on the server and work done on the browser (i.e. plotting on the server or in the Oceanbrowser?)
21 april 2019

10. Loading new datasets (until now)
Description of the current ODV data collection
Relation between file and database is maintained via filename link of each observation
Output of previous DIVA products
Deltares has a semi-automated procedure for uploading new collections
21 april 2019

11. Loading new datasets (from now on)
Description of new data collection
NEW ODV File format has been tested (successfully imported) …
21 april 2019

12. Differences and Effects
At first glance, no critical differences for the import routines
No effects on:
Uploading procedure
Metadata handling
Columns handling
21 april 2019

13. Optimisations 1/2 Overall
Performance of the services for visualisation of timeseries, profiles, timeprofiles on retrieving the data, but also with the interaction between Oceanbrowser (benchmarking)
WP4.2 Task 1: Improving the performance and tweaking graphics of the existing advanced viewing services – Deltares + MARIS + OGS
Splitting main DB observations table in multiple tables per parameter
Optimization of the WPS processes
Enhancing plot visualisation with interactive plotting libs (bokeh, plot.ly)
21 april 2019

14. Optimisations 2/2
Performance of the services for visualisation of timeseries, profiles, timeprofiles (stress test)
WP4.2 Task 2: Developing dedicated maps on contaminants as part of the advanced viewing
Maps
Cross domain maps (bathymetry and profile in formation for specific area)
Statistics (time series statistics for instance, data availability over time/depth,)
21 april 2019

15. Performance
21 april 2019

16. Ideas and discussion Deltares developed uploading scripts (OSS)
Shift responsibility for uploading new data collections to EMODNet consortium
Proposal:
Automated upload from RCs
Code on Github
Provision of loading API (OSS and manuals)
Ex. EMODNet Ingestion
21 april 2019

17. Ideas and discussion Data ingestions should be further automated
Via portal
Better performance by changing the datamodel
Load cdi’s
Load jsonb per cdi, per parameter
21 april 2019

18. Conclusions Need of improving performance
Change of database schema and update of WPS
Re-thinking of ingestion of the new data
New maps and statistics
21 april 2019