1. Improvements in database of data products and services
Giorgio Santinelli, Gerrit Hendriksen, Wouter Schoonveld
First Annual Meeting EMODnet-Chemistry3
Roskilde, Denmark
12 april 2019

2. Index P01 terms next to P35 in ODV collections
ETL procedure for automated data ingestion
WPS to return data from database
12 april 2019

3. P01 terms next to P35 in ODV collections
AWI was asked to include also P01 terms in the resulting ODV collections next to the P35 parameters. Done in a release of the ODV software, circulated to all Regional Coordinators, and which includes per CDI an extra column with all related original P01 terms (so not 1 : 1 paired to P35 terms).
Deltares ingested the ODV collection by retrieving all the P01 terms.
12 april 2019

4. P01 terms next to P35 in ODV collections
12 april 2019
P01 terms next to P35 in ODV collections
Most ideal situation is to store P01 in the aggregated ODV collections and serve the P35 value to end-users.
What are the options?
Full new delivery of all existing data (incl. the P35 – P01 information), so start over again with ‘clean slate’
Fix only contaminants in biota data
And/or only new data with the P35 – P01 information (so from 2018 on add new data)
12 april 2019

5. Data model changes Current situation data model New additional table
12 april 2019
Data model changes
Current situation data model
New additional table
CDI id
Local_cdi_id
Edmo
location
observation id
cdi_id
Parameterid
Datetime
Depth
value
Obs-p01 Id
observationid
[P01]
1:n
n:1
1:n
parameter Id
altlabel
preflabel
p35
Data model changes are necessary to be able to add a list of P01 parameters to a p35 parameter, or better, to an observation. Because for each observation it could be different list of P01 parameters. The way the new additional table (obs-po1 called) is setup can be part of the discussion. With a combined foreign key of parameterid, cdi_id and date from the observation to a array of P01’s associated with that specific combination of location, parameter and observations.
12 april 2019

6. ETL procedure for automated ingestion
For uploading and further maintenance of the database of aggregated data Deltares has developed a number of scripts and it would like to handover the responsibility to another partner.
We need a staging process whereby the loading is first done and tested in another version of the database before it is made public in the production database.
12 april 2019

7. Proposed setup of data ingestion
Regional coordinators
Ocean browser
automated quality control,
feedback & approval
Geoserver + PyWPS
ETL
PostgreSQL/PostGIS
BackOffice for final approval?
12 april 2019

8. WPS to return file of observation
Working on an API to easily query parts of the database
As first step. Generation of a WPS service that returns data in a certain format.
Test:
12 april 2019

9. WPS to return file of observation
Ready to implement a WPS that can return a file of aggregated data products needed or selected. It is depending on:
depth - limits
Bbox
Time - limits
P35
Download option?
What format?
12 april 2019

10. Thank you
Questions?
12 april 2019