1. GROOM Annual meeting Data Management issues Trieste 4-5 June 2013 Quoi de Neuf à Coriolis en 2008 ? GMMC 13-15 Octobre 2008 S Pouliquen & Coriolis team Developing a Data Management System for Glider data in Europe Sylvie Pouliquen GROOM Annual meeting 4-5 June 2013 Developing a Data Management System for Glider data in Europe Sylvie Pouliquen GROOM Annual meeting 4-5 June 2013

2. GROOM Annual meeting Data Management issues Trieste 4-5 June 2013 The Objectives of an integrated data system Data accessible easily from a unique point Data coherent in term of :  Data format  Data Quality  Processing chain ( clearly documented) Serve both Operational and Research Users  Data are available in near real time ( within less than 24 hours)  Data are available in delayed mode after calibration and /or validation

3. GROOM Annual meeting Data Management issues Trieste 4-5 June 2013 What About Gliders Glider activities are presently driven by individual research drivers Gliders can deliver real-time data for core parameters ( T, S Current, Chl, O2) that are useful for both research an operational users Benefit from what has been developed by Argo and OceanSites  Developing integrated Data Management system  Common Data format to users  Real –time QC of core parameters Gliders are complementary to other platforms and synergy should be developed  Developing a deployment strategy for other needs than pure research ( ie. GMES Marine Core Service in Europe)

4. GROOM Annual meeting Data Management issues Trieste 4-5 June 2013 What is the starting point for GROOM Data management is done by the different research communities using their own methods Some harmonization activities started in FP5-MFSTEP and FP6-MERSEA projects and FP7 MyOcean Data exchange in Real time is working on a best effort schema through EGO but without any commitment nor from providers or data managers Link with GMES MCS is done though Coriolis providing Glider data as profiles No agreement neither on RTQC or DMQC but best practices on RTQC through MyOcean INSTAC not widely known by the glider teams

5. GROOM Annual meeting Data Management issues Trieste 4-5 June 2013 Groom Data Management activities Task 3.2

6. GROOM Annual meeting Data Management issues Trieste 4-5 June 2013 GROOM Data Management Objectives Data accessible easily from a unique point Data coherent in term of :  Data format  Data Quality  Processing chain ( clearly documented) Serve both Operational and Research Users  Data are available in near real time ( within less than 24 hours)  Data are available in delayed mode after calibration and /or validation

7. GROOM Annual meeting Data Management issues Trieste 4-5 June 2013 Objective of task 3.2 Improve coherency of Glider dataset in Europe Facilitate access to Glider data by defining a Data system for Glider data The system is built to exchange glider data  Must first be useful for Glider users and Glider operators and therefore provide access to as much as possible information (metadata, scientific, technical data) provided by Gliders  Delivering of Glider Data either on GTS or FTP for Operational users ( i.e. MyOcean) will be carried on as specific delivery system that will convert the GDAC data into products usable by these operational users

8. GROOM Annual meeting Data Management issues Trieste 4-5 June 2013 Who does what ?

9. GROOM Annual meeting Data Management issues Trieste 4-5 June 2013 GROOM Data flow PIS Scientific Users Operational Centers GDAC France Final correction Data to DAC - GDAC RT DATA QUALITY CONTROL Data assembly Centres GTS FEED RT within hours DAC Within hours DAC Glider Operators

10. GROOM Annual meeting Data Management issues Trieste 4-5 June 2013 Roles of Glider Actors PI : Principal Investigators  Team or scientists who define the glider mission, deploy the glider and carry out post-recovery delayed mode QC Glider Operators  Team in charge of steering the glider, collecting all the metadata and the deployment information required for processing, collect all the data transferred in realtime by the glider. Collect the post- recovery high resolution data

11. GROOM Annual meeting Data Management issues Trieste 4-5 June 2013 Roles of Glider Actors the Dacs : the DAC is the facility set up by one or more nations/institutes to provide RealTime and Delayed mode glider data to the users. It  collects the data from the Glider Operator,  converts to standard exchange format,  applies standardized real-time quality control,  delivers data to the GTS and GDACs within few hours of the surfacing and to PIs,  coordinates glider data handling for the gliders under their control.

12. GROOM Annual meeting Data Management issues Trieste 4-5 June 2013 List of DACs CountryDACGlider OperatorDelayed Mode PI UK BODCUEA, SAMS, NERC (NOC, POL) CEFAS, BAS UEA, SAMS, NERC (NOC, POL), CEFAS, BAS France CORIOLISLOCEAN, LOV, MIO, LEGOS, LPO, DT-INSU LOCEAN, LOV,MIO, LEGOS, LPO Italy CMRE To Be Defined OGS/CORIOLI S OGS Germany HZG CORIOLISGEOMAR,AWI SPAIN CORIOLISPLOCAN SOCIBSOCIB, CSIC Norway UiB/IMRUiB Cyprus OC-UCY Greece CORIOLISHCMR ?To Be Defined Finland To Be Defined Ireland To Be DefinedMI Poland CORIOLISIOPAS

13. GROOM Annual meeting Data Management issues Trieste 4-5 June 2013 Roles of Glider Actors GDAC : The GDAC operates the data services where the master copies of the data resides. It doesn’t perform any additional individual glider QC activities.  Central point for data distribution on Internet for all GROOM gliders  Can perform data format transformation, of set up additional services ( OGC viewing service, OpenDap/Oceanotron download services,…) to fulfil additional needs.

14. GROOM Annual meeting Data Management issues Trieste 4-5 June 2013 Common data format

15. GROOM Annual meeting Data Management issues Trieste 4-5 June 2013 Improve Coherency of Glider dataset in Europe Worked on the definition of a Glider, of a deployment for a glider.  A glider is moving platform that is steerable. It can have a propeller and this information must be recorded in the metadata. Define Level of processing:  Level0 : Data provided by the glider without any unit transformation or geophysical interpretation.  Level1: Geophysical parameters with a quality indicator set up by automatic QC procedures together with the data acquired by the glider. This is the level shared in Near Real Time.  Level2 : Geophysical parameters calibrated after glider recovery together with quality flag information, if possible error estimation together with the non-corrected data provided at Level1. This is the level shared in Delayed Mode  Level3 and after : Product derived from glider data ( gridded fields, additional parameters calculated …) This is not addressed in the present GROOM data management activities.

16. GROOM Annual meeting Data Management issues Trieste 4-5 June 2013 How to Describe a Glider Sensor S Takes on board Deployment N Performs Description of the glider for this deployment What are the sensors on board What is the configuration Is described by Transmits for each dive Technical information measurements Scientific measurements

17. GROOM Annual meeting Data Management issues Trieste 4-5 June 2013 Improve Coherency of Glider dataset in Europe Worked on common data format to share the Glider data based on OceanSites NetCDF format already used in EuroGOOS/ MyOcean/ SeaDataNet  Defined how to store Metadata to register the mission description and its evolution in time when changed through down link  Enhanced OceanSites format for Scientific information by adding if necessary the new parameters sampled  Defined how to store the technical information by definition common vocabulary  First version of the user manual was delivered  Common tools to produce these Netcdf files are under way

18. GROOM Annual meeting Data Management issues Trieste 4-5 June 2013 RT Processing chain modules Source data Seaglider Slocum Other gliders Processing modules Data delivery Common Readers in matlab that convert Manufacturers Data into.mat files Conversion of.mat output to EGO format NetCDF for transmission Real Time quality control routines PIGDACGTS Generic Description -Platform -Sensors My Glider description

19. GROOM Annual meeting Data Management issues Trieste 4-5 June 2013 Data Stream in Real time Glider Operator DAC GDAC Sci and Tec data Sci and Tec users Customize to meet other needs Sci Data GTS Users MyOcean Users DATADATA DATADATA Manufacturer Format Common Format

20. GROOM Annual meeting Data Management issues Trieste 4-5 June 2013 Define Real Time QC procedure Action taken to work on common Near Real Time QC procedures in agreement with what exist already within EuroGOOS/MyOcean/SeaDataNet  Adopt EuroGOOS/MyOcean NRT QC procedures for T&S, Chl and O 2  Enhance if necessary these procedures to take into account Glider specific behavior  Develop new recommendations in Partnership with Myo/SDN for additional parameters available in real time

21. GROOM Annual meeting Data Management issues Trieste 4-5 June 2013 Data Stream after recovery Glider Operator DAC GDAC Common Format users HRHR HR TECHR TEC High Resolution scientific data Real Time Data in Common format ------   ----Delayed mode High resolution data Manufacturer Format Common Format High Res Manufacturer Format Customize to meet other needs Sci Data GTS Users MyOcean Users

22. GROOM Annual meeting Data Management issues Trieste 4-5 June 2013 Post recovery procedures The Goal : correct the data transmitted in real time  Check RTQC again. Additional run taking into account past and future data is possible (future data not available in RT !)  If necessary Merge RT data with flash card data in order to fill the gaps left by the RT data transmission in terms of resolution and/or parameters measured  Cross-calibration along the whole deployment with reference to in-lab water samples measurements when possible.  Working group to develop recommendations for T&S, Chla, Oxygen, Current

23. GROOM Annual meeting Data Management issues Trieste 4-5 June 2013 Possible extention outside Europe through the EGO Cost action

24. GROOM Annual meeting Data Management issues Trieste 4-5 June 2013 GROOM Data flow PIS Scientific Users Operational Centers GDAC France Final correction Data to DAC - GDAC RT DATA QUALITY CONTROL Data assembly Centres GTS FEED RT within hours DAC Within hours DAC Glider Operators

25. GROOM Annual meeting Data Management issues Trieste 4-5 June 2013 Possible extention outside Europe PIS Scientific Users Operational Centers Final correction Data to DAC - GDAC RT DATA QUALITY CONTROL Data assembly Centres GTS FEED Glider Info Center? RT within hours Monitoring GDAC ? Archive ? WWW FTP GDAC France DAC Within hours Glider Operators

26. GROOM Annual meeting Data Management issues Trieste 4-5 June 2013 Data Format and QC procedures Data Format : based on OceanSites netcdf format with CF convention also used by IMOS /Australia  Compare the format metadata to be sure that we are coherent (link with ODIP EU project): name the same thing the same way list of mandatory fields that should be in all files Share tools to generate netcdf files NRT QC procedures : Both IMOS and QUARTOD procedures are taken as input for defining the EU RTQC procedures for core parameters. Post-recovery procedure : still and R&D activity but collaboration with international partners should be encouraged

27. GROOM Annual meeting Data Management issues Trieste 4-5 June 2013 Quoi de Neuf à Coriolis en 2008 ? GMMC 13-15 Octobre 2008 S Pouliquen & Coriolis team Thank You Questions ? Thank You Questions ?