Groom-gliders data-management workshop Brest, December 2012 Groom gliders data management n In 2012 : 18 714 vertical profiles from 26 platforms.

Slides:



Advertisements
Similar presentations
OBSERVATIONS & PRÉVISIONS CÔTIÈRES Training Workshop – Ostende – October 2009 REFORMATTING TOOLS Med2MedSDN and NEMO M. Fichaut.
Advertisements

10 th Argo data management 2009 Toulouse Argo format and CF compatibility OceanOBS09 Strengthen and enhance the international framework under GCOS, GOOS,
Air-Deployable Profiling Floats
Groom-gliders Trieste data-management meeting Trieste, June 2013 T&S real-time QC n 1. Platform identification : valid wmo ID n 2. Impossible date test.
1 OBJECTIVES To generate a web-based system enables to assemble model configurations. to submit these configurations on different.
An OPeNDAP server for CORIOLIS data centre Context and functions Architecture and technical description Planned improvements T. Loubrieu, A. Forest (IFREMER)
FP7-Infra : Design studies for European Research Infrastrutures 1st October 2011 – 31st December 2014 Duration 39 months – Periods : 2 (month.
Pierre Jaccard1 MyOcean Quality Control for Ferryboxes MyOcean Tutorial, NERSC, Bergen.
Kick-Off Seadatanet – Heraklion – 7-11 June Atlantic and global products C. Coatanoan, F. Gaillard, E. Autret, T. Loubrieu.
Training course – Ostende – 2-6 July 2012 Training Workshop – Ostende – 1-4 March 2010 Practical work on NEMO Converting files M. Fichaut.
10th Argo data management 2009 Toulouse User’s manual comment Here is a list of comments received on User’s manual version 2.2 since August 28 th. Jean-Philippe.
Ocean glider data management: Argo concepts, GROOM, OGC sensor observation services Justin Buck + many collaborators British.
Work Package 2 / 3 TECHNOLOGICAL & PROCEDURAL HARMONISATION FixO3 General Assembly 14 th to the 16 th October 2014, Heraklion-CRETE ESTOC, PLOCAN,
MEDIN Data Guidelines. Data Guidelines Documents with tables and Excel versions of tables which are organised on a thematic basis which consider the actual.
Work Package 2 / 3 TECHNOLOGIAL & PROCEDURAL HARMONISATION FixO3 General Assembly 14 th to the 16 th October 2014, Heraklion-CRETE Maureen Pagnani, BODC/NOC,
OBSERVATIONS & PRÉVISIONS CÔTIÈRES 3 rd SeaDataNet training course – Ostende – June 2008 NEMO reformatting tool v1 M. Fichaut.
7 th TTG meeting – Trieste – 3-4 March, 2015 SeaDataNet tools, new developments IFREMER : M. Fichaut, A. Briand, M. Larour, T. Loubrieu, V. Tosello ALTRAN.
AON Data Questionnaire Results 21 Respondents Last Updated 27 March 2007 First AON PI Meeting Scot Loehrer, Jim Moore.
NRT data stream from the gliders Slocum and Spray to Coriolis EGO meeting Paris 2-3/10/06.
INSTRUCTIONS We are looking to develop a detailed understanding of what information about an Platform needs to be displayed somewhere in the user interface,
Editing RT QC flag in delayed mode ? Virginie Thierry DMQC 4 Toulouse, 28 septembre 2009.
Glider data (*.*bd files) via Iridium or rsync from WHOI, Umaine or UMaryland Disk NetCDF/Acquire WMO ID IOOS Glider DAC ERDDAP OPeNDAP sequence or download.geoJson,.json.nc,.ncCF,.ncCFMA.csv,.tsv,
Object-Oriented Analysis & Design Subversion. Contents  Configuration management  The repository  Versioning  Tags  Branches  Subversion 2.
SAMOS-GOSUD Meeting. Boulder 2-4 May Potential collaboration between the Coriolis project and the Samos initiative L. Petit de la Villéon. Ifremer-France-
Progress with migration to SVN Part3: How to work with g4svn and geant4tags tools. Geant4.
EGO – COST - Groom-gliders Kiel meeting Kiel, June 2014 Data-management activity n Within Groom and with COST support, the glider data- management group.
GROOM Annual meeting Data Management issues Trieste 4-5 June 2013 Quoi de Neuf à Coriolis en 2008 ? GMMC Octobre 2008 S Pouliquen & Coriolis team.
Topics describe the Data Buoy Cooperation Panel (DBCP) –aims, –achievements and –Challenges network status developments of data buoy technology JCOMM.
_______________________________________________________________CMAQ Libraries and Utilities ___________________________________________________Community.
Argo workshop in Ghana, December Argo data status & data access.
Argo & New Telecom. Systems Technical Workshop Introduction and Summary. ADMT#11 October 2010, Hamburg/Germany.
IOOS National Glider Data Assembly Center
OBSERVATIONS & PRÉVISIONS CÔTIÈRES SeaDataNet annual meeting, Madrid, March 2009 How to prepare data for integration in SeaDataNet.
SWGData and Software Access - 1 UCB, Nov 15/16, 2006 THEMIS SCIENCE WORKING TEAM MEETING Data and Software Access Ken Bromund GST Inc., at NASA/GSFC.
Copyright © 2010, SAS Institute Inc. All rights reserved. SAS ® Using the SAS Grid.
ICM – API Server & Forms Gary Ratcliffe.
An Introduction to the Argo Data Sytem South Pacific Workshop 11 – 14 October 2005 Mark Ignaszewski FNMOC.
INSTRUCTIONS We are looking to develop a detailed understanding of what information about an Instrument needs to be displayed somewhere in the user interface,
ICM – API Server Gary Ratcliffe. 2 Agenda Webinar Programme API Server Overview JSON-RPC iCM API Service API Server and Forms New services under.
Standard Metadata in Scientific Data Formats September 19, 2007 Flash at:
10 th Argo data management 2009 Toulouse What is new at GDACs ?
EMODNET Preparatory Actions Progress meeting for ur-EMODNET preparatory actions, 24 November 2009 Presentation of Progress Chemical Lot Alessandra Giorgetti.
Institute for the Protection and Security of the Citizen HAZAS – Hazard Assessment ECCAIRS Technical Course Provided by the Joint Research Centre - Ispra.
CARBOOCEAN Data management and SOCAT Benjamin Pfeil, Are Olsen, Jeremy Malzcyk, Steve Hanhin, Alex Kozyr and many others Partner 16 WDC-MARE Partner 19.
What problems are we trying to solve? Hannes Tschofenig.
2 nd training course – Ostende – May, 2014 What’s new in NEMO 1.5?
3rd Training Workshop June 2008, Ostende Management of CSR Anne Che-Bohnenstengel, BSH  Metadata Formats  Defined Vocabularies  Content Management.
Scenario use cases Szymon Mueller PSNC. Agenda 1.General description of experiment use case. 2.Detailed description of use cases: 1.Preparation for observation.
CTD Data Processing Current BIO Procedure. Current Processing Software Matlab Migrating to R & Python Code Version Control SVN Migrating to GitHub.
Sensor Web Enablement (SWE) developments for fixed monitoring platforms and research vessels By Dick M.A. Schaap – SeaDataNet Technical Coordinator with.
ATom data management plan: Data products and submission requirements Observed data and model products will be submitted in NetCDF or ICARTT format www-air.larc.nasa.gov/missions/etc/ESDS-RFC-019-v1.1_0.pdf.
SeaDataNet Technical Task Group meeting JRA1 Standards Development Task 1.2 Common Data Management Protocol (for dissemination to all NODCs and JRA3) Data.
MIKADO – Generation of ISO – SeaDataNet metadata files
SeaDataNet tools NEMO, OCTOPUS, MIKADO
NASA Earth Science Data Stewardship
Simulink Interface Layer (SIL)
NEMO – Reformating tool
GML in CDI and CSR ISO using Ends&Bends
Practical work on NEMO Converting files
IOOS National Glider Data Assembly Center
IOOS National Glider Data Assembly Center
OCTOPUS – SeaDataNet Format conversion tool
Outline RTQC goals to achieve Description of current proposal Tests
How to handle multiple vertical references
Dick M.A. Schaap – Technical Coordinator SeaDataNet Training Workshop
MIKADO: Generation of CDI ISO19139 XML files
Tools for generation of SDN NetCDF (CFPOINT) files
International Argo Data Management Team
Data aggregation and products generation
Practical work on NetCDF - CFPOINT
Presentation transcript:

Groom-gliders data-management workshop Brest, December 2012 Groom gliders data management n In 2012 : vertical profiles from 26 platforms

Groom-gliders data-management workshop Brest, December 2012 Workshop objectives n Task 1: Complete and validate the glider’s data management user’s manual n Assigned to : Thierry ?

Groom-gliders data-management workshop Brest, December 2012 Workshop objectives n Task 2 : Setup a Glider matlab data processing –Work at DAC level n For a deployment, from raw Iridium data to the Glider NetCDF file containing: –Metadata –Data (times-series with profile information) –Technical data –Type of gliders : Slocum, seaglider, other ? n Assigned to : Justin (?), Tomeu (?), Jean-Philippe (?), –Subtask 2.1: read Slocum data (Daniele) –Subtask 2.2: read Seaglider data (BODC team) –Subtask 2.3: generate the glider NetCDF file (Thierry, Lise) –Subtask 2.4: manage metadata (Tomeu) Fixed metadata, transmitted metadata (navigation commands) –Subtask 2.5: manage technical data (Mark, Daniele, Justin)

Groom-gliders data-management workshop Brest, December 2012 Workshop objectives n Task 3: Setup real-time QC and apply it on the glider NetCDF glider file n Assigned to: Tomeu, Justin

Groom-gliders data-management workshop Brest, December 2012 Workshop objectives n Task 4 : manage the non-pumped (and pumped) CTD errors or other sampling issues Daniele

Groom-gliders data-management workshop Brest, December 2012 Workshop objectives n Task 6: oxygen data management n Convert oxygen observations (MOLAR_DOXY) to micromole/kg (DOXY) as done on Argo floats n Justin

Groom-gliders data-management workshop Brest, December 2012 Workshop objectives n Task 7: organize a collaborative development n Investigate the use of a forge such as Ifremer “Fusion forge”, subversion to distribute the matlab softwares, mantis to manage actions – n Thierry

Groom-gliders data-management workshop Brest, December 2012 Workshop objectives n Task 8 : how to manage sea water current from the glider trajectory n When reported by the float : real-time current n When reported from a PI : delayed mode-current

Groom-gliders data-management workshop Brest, December 2012 Workshop objectives n Task 9 : specify a policy for data distribution on GTS n Mark, Justin, Thierry

Groom-gliders data-management workshop Brest, December 2012 Workshop objectives n Manage the time sampling issues from different CPUs, sensors, clock drifts n Manage duplicate times

Groom-gliders data-management workshop Brest, December 2012 Messages from glider *.SBD *.DBD Messages from glider *.SBD *.DBD Decoded data *.m *.dat Decoded data *.m *.dat EGO decoded data *.m *.dat EGO decoded data *.m *.dat EGO NetCDF file metadata data technical data EGO NetCDF file metadata data technical data mapping glider variable EGO variable mapping glider variable EGO variable variable attributes standard_name, unit, long_name… deployment metadata sensors, serial numbers… deployment metadata sensors, serial numbers… glider metadata name, wmo_platform_code, maker… glider metadata name, wmo_platform_code, maker…

Groom-gliders data-management workshop Brest, December 2012 Slocum SBD data processing, a fisrt step n Read and decode SBD –input : sbd received from the float –manufacturer script –result : « initial *.m *.dat » –no metadata needed n Read and transform « initial *.m *.dat » –Associate manufacturer variables with EGO variable names sci_water_temp -> TEMP, m_present_time, m_present_secs_into_mission –Slocum-EGO metadata file n Write EGO NetCDF file –EGO variables and data TEMP –glider metadata file platform_code, wmo_platform_code –glider deployment metadata file deployment, deployment_start, deployment_end –EGO NetCDF metadata file sea_water_temperature

Groom-gliders data-management workshop Brest, December 2012 Slocum binaries real-time data processing, a first step n Step 1 : read and decode *.tbd *.sbd files from glider to (*.m, *.dat) –The *.sbd and *.tbd files are transmistted in real-time. The *.sbd files contain a subset of glider technical data (nav) The *.tbd files contain a subset of glider scientific data (sci) This subset is configured by the operator in a sbdlist.dat file listing the data (engineering, scientific) to be transmitted. There is a minimum set of data to be transmitted under definition by Gerd working group –The script is provided by the manufacturer –The glider operator usually does not configurate the script to filter binary data –The script is operated by the glider operator, not by the data centre –The data centre archives the files received in real time

Groom-gliders data-management workshop Brest, December 2012 Slocum SBD data processing, a first step n Step 2 : read and decode initial (*.m *.dat) into NetCDF –input : « initial *.m *.dat » –Ego matlab script, metadata : json Read input data Rename manufacturer variables with EGO variable names sci_water_temp -> TEMP, sea_water_temperature, Celsius_degree… Add EGO attributes to variables Write the NetCDF file –Output: initial EGO NetCDF data file n Step 3 : set global attributes of EGO NetCDF file from –glider ncml metadata file platform_code, wmo_platform_code –glider deployment ncml metadata file deployment, deployment_start, deployment_end n Step 4 : add sensor description variables

Groom-gliders data-management workshop Brest, December 2012 JSON metadata n That ends up in the EGO netcdf n Glider family variables to EGO variable names {EGO} –Variable attribtes : CF, SDN, unit and a prefered name n EGO-NetCDF metadata {at DAC level based on latest version} –Format version, data_type, date_update, … n Deployment metadata {glider team} –global attributes and variables (area, launch_date, cruise_id) n Glider metadata {glider team} –Global attributes and variables for glider description (platform_code, wmo…) n Sensor metadata (multiples by deployment?) {glider team} –Description (sensor model, serial number…) n Calibration information(n_param,n_cal’n) (1sensor:many cal’ns 1var:many cal’ns) {technicians/glider team ?} –Calibration date, calibration equation, calibration parameters n Translation table JSON – informs writer n EGO-NetCDF fields (optional/mandatory) {at DAC level} n Sensor model common attributes {manufacturer provides}

Groom-gliders data-management workshop Brest, December 2012 Collaborative development proposal Subversion repository n The EGO-gliders data processing softwares and tools can be managed in a repository n svn checkout --username *** n svn list --username *** gliders/trunk

Groom-gliders data-management workshop Brest, December 2012 Collaborative development proposal Mantis for action mangement n Mantis is orginally a bugtracker. I propose to use it to manage our data management actions n On the top right,

Groom-gliders data-management workshop Brest, December 2012 Collaborative development proposal Mantis for action mangement n Mantis is orginally a bugtracker. I propose to use it to manage our data management actions n On the top right, select “ego-gliders” n Simple workflow –Submitted pale yellow –Accepted orange –Delivered green –Closed grey –Pending camel (à la mode cette année)

Groom-gliders data-management workshop Brest, December 2012 Collaborative development proposal Testlink for validation n A series of tests are specified in testlink n For each new version of the EGO code, a test report is updated on Testlink

Groom-gliders data-management workshop Brest, December 2012 n Glider –Deployment 1 –Deployment 2 Sensor 1 Sensor 2 –Parameter 1 –Parameter 2 »Calibration 1 »Calibration 2