An OPeNDAP server for CORIOLIS data centre Context and functions Architecture and technical description Planned improvements T. Loubrieu, A. Forest (IFREMER)

DAP4COR : An OPeNDAP server for CORIOLIS data centre CORIOLIS is an operational data centre for ocean in-situ measurements. As such CORIOLIS : Collects measurements for various observation systems (ARGO profilers, measurement from research ships, XBT from Volontary observation ships, moorings, …) Apply various quality controls on datasets. Disseminates datasets as asked by users (ocean forecasting systems, ARGO, Carbocean, …) The datasets are fully managed in an Oracle RDBMS until they are disseminated as netCDF files in ftp directories or requested through a GUI web interface.

DAP4COR : external usage requirements Open a standard interface on CORIOLIS database : Enable interactive subsetting of the datasets (even with scripts) Provide an oceanography oriented, high level, interface so that data management and low level information are hidden to the end-user. Promote CORIOLIS data centre interoperability Use DAPPER ocean in-situ vertical profile DAP model. DAP4COR is not intended to be distributed and installed outside of CORIOLIS data centre.

DAP4COR : internal requirements Demonstrate a robust and highly capable federating service for dissemination : Improve reactivity in the development of new dissemination tools. Facilitate the maintenance of the multiple dissemination tools by making them independant from the low level RDBMS model. Avoid datasets redundancy (Oracle RDBMS / netCDF files) to ease data update management. Better resources management by using a unique RDBMS pool of connexion. Use J2EE architecture (java + tomcat) because of system infrastructure and available skills.

DAP4COR : functions DAP4COR is Web Application enabling : OPeNDAP request interpretation for ocean in-situ profile dataset subsetting OPeNDAP, DAPPER-like response generation from RDBMS profile data. DAP4COR Dataset configuration (on SQL criteria on the RDBMS native storage). DAP4COR command line client for saving extraction as NetCDF ARGO profiles file.

DAP4COR : request syntax example base URL dap4cor-coriolisdataset.separator [dds|das|ascii|dods]output extension ? separator location.profile.PRES projected variable, separator location.profile.TEMP projected variable & separator location.TIME> selection criteria & separator location.TIME< selection criteria &separator location.attributes.PLATFORM_NUMBER="DACF" selection criteria

DAP4COR : user’s subsetting criterias available Platform code Geographical area Time Vertical reference : depth, pressure Available measurements Measurement values ARGO profiling float T/S diagram On a small ocean box Dissolved oxygen Profiles for 2 weeks Mediterranean Salinity profiles One month

DAP4COR : dataset configuration Main data (location, time, measurements) are stored in the same 3 tables. Ancillary data are distributed into numerous exotics tables (CORIOLIS « expert » domain). DA4COR Datasets can be configured according to user needs by defining low level SQL criteria on the Oracle DBMS : Subset of platforms (gliders, elephant seals, moorings array…) Instrument types (CTD, XBT, XCTD, …) Quality assesment level (real time, near real time, delayed mode…) Geographical area (ocean basin, WOCE reference section), by using Oracle Spatial functions (GIS). … Thanks to that, a most of the RDBMS model complexity is hidden to the user. The most relevant datasets are configured on server as SQL pre-defined SQL criteria without data redundancy.

DAP4COR : dataset configuration examples WOCE reference section, for example AX7 Mediterannean sea Measurements from elephant seal Gliders dataset GTS datasets (BATHY, TESAC messages) Sea Surface dataset (for satellite calibration) Pre-processed profiles : pressure/depth conversion, interpolation on standard vertical levels, parameters conversion…  To Be Done …

DAP4COR : response format Ocean in-situ vertical profiles as proposed in PMEL’s DAPPER server : One sequence of profiles For each profile : First level information (time, location + quality control results) Metadata (profile attributes) Sequence of the measured levels (vertical reference value, measured parameters values, quality control information) Maximum number of vertical profiles in each response : 1000

DAP4COR : command line client (dap4cor2netCDF) Windows/Unix or Linux portable software. Uses DAP4COR opendap request as input. Saves the subset as NetCDF ARGO profile file. Command line tool which can be used as batch (usefull for operational requirements).

DAP4COR : system architecture tomcat application server DAP4COR JDAP DAP4COR Datasets configuration tomcat JDBC connection pool Oracle RDBMS Spatial functions Components we rely on : JDAP Tomcat application server (enabling load balancing) Tomcat JDBC connection pool Oracle RDBMS Oracle spatial functions (GIS)

DAP4COR : a 3 layers software architecture DAP request interpretation/data model generation based on java API (JDAP 1.1.7) In-situ Vertical Profile business interface Implemented in order to manage : raw profiles Or pre-processed profiles (confidential information hidden, vertical reference conversion, interpolated on standard vertical levels…)  To Be Done Design as a « profile reader » so that memory overflow does not occur. : It reads the lower layer / write the upper layer very frequently so that the smallest possible amount of data is kept is that layer memory. JDBC convenient resultSets

DAP4COR : profile business interface Main methods : setConstraint() : initializes the OPeNDAP subsetting criterias (dataset, projected parameters, selection parameters) getStationRoughNumber() – getStationPreciseNumber() : maximum profile quota analysis. Prepare() : initializes the requested profiles subset (before reading) Profiles subset reading methods (mapped on DAPPER sequences order) : nextStation() : move to next vertical profile getStationElementValue() : get global information on the current vertical profile (time, location, platform, …) nextLevel() : move to next vertical level for the current profile nextProfileLevelElementValue() : get and move to next measurement for the current vertical level. Interface implementation is a parameter of the dataset configuration so that we can define, for example : A dataset of pre-processed profiles with homogeneous vertical references (pressure/depth converted if necessary) …

DAP4COR : planned improvements Enable big dataset request (thanks to OPeNDAP indexed requests ?) Improve performance especially when only the global profile information (time, location, platform code, …) are requested. Implement alternative profile interface for profile pre-processing (vertical level conversion, interpolation on standard vertical levels…) Provide a new OpeNDAP interface for other CORIOLIS datasets : time series at a fixed location along track observations Keep on following the DAPPER observation data model convention and improve compliance with it.

DAP4COR : links Server URL : Documentation and related tools : ftp://ftp.ifremer.fr/ifremer/coriolis/tools/dap4cor Containing : FAQ document Java client which enable saving datasets as NetCDF ARGO profiles file (dap4cor2netcdf). Matlab client script examples Python client script example