Outline RTQC goals to achieve Description of current proposal Tests

Slides:



Advertisements
Similar presentations
Better Data, Better Science! [ Better Science through Better Data Management ] Todd D. OBrien NOAA – NMFS - COPEPOD.
Advertisements

10 th Argo data management 2009 Toulouse Argo format and CF compatibility OceanOBS09 Strengthen and enhance the international framework under GCOS, GOOS,
Frequent Closed Pattern Search By Row and Feature Enumeration
Groom-gliders data-management workshop Brest, December 2012 Groom gliders data management n In 2012 : vertical profiles from 26 platforms.
An OPeNDAP server for CORIOLIS data centre Context and functions Architecture and technical description Planned improvements T. Loubrieu, A. Forest (IFREMER)
Preparing CMOR for CMIP6 and other WCRP Projects
Pierre Jaccard1 MyOcean Quality Control for Ferryboxes MyOcean Tutorial, NERSC, Bergen.
C van Ingen, D Agarwal, M Goode, J Gupchup, J Hunt, R Leonardson, M Rodriguez, N Li Berkeley Water Center John Hopkins University Lawrence Berkeley Laboratory.
Argo Real-time Quality Control Process NOAA/AOML: Y.-H. DANESHZADEH, R. MOLINARI, R. SABINA, C. SCHMID CIMAS/UM: E. FORTEZA, X. XIA, H. YANG.
Operational Quality Control in Helsinki Testbed Mesoscale Atmospheric Network Workshop University of Helsinki, 13 February 2007 Hannu Lahtela & Heikki.
Scalable Information-Driven Sensor Querying and Routing for ad hoc Heterogeneous Sensor Networks Maurice Chu, Horst Haussecker and Feng Zhao Xerox Palo.
Abstract In this project we expand our previous work entitled "Design of a Robotic Platform and Algorithms for Adaptive Control of Sensing Parameters".
10th Argo data management 2009 Toulouse User’s manual comment Here is a list of comments received on User’s manual version 2.2 since August 28 th. Jean-Philippe.
Reiner Schlitzer Alfred Wegener Institute for Polar and Marine Research Ocean Data View - Available Data Collections and Data Model.
World Renewable Energy Forum May 15-17, 2012 Dr. James Hall.
Ocean glider data management: Argo concepts, GROOM, OGC sensor observation services Justin Buck + many collaborators British.
1 Validated Stage 1 Science Maturity Review for {JPSS Algorithm} Presented by Date.
AON Data Questionnaire Results 21 Respondents Last Updated 27 March 2007 First AON PI Meeting Scot Loehrer, Jim Moore.
Mike Botts – August Supporting QA/QC for Ocean Observations using Sensor Web Enablement (SWE) and SensorML August 2008 Mike Botts (UAH), Tony Cook.
DM_PPT_NP_v01 SESIP_0715_AJ HDF Product Designer Aleksandar Jelenak, H. Joe Lee, Ted Habermann Gerd Heber, John Readey, Joel Plutchak The HDF Group HDF.
Winrunner Usage - Best Practices S.A.Christopher.
Soft Sensor for Faulty Measurements Detection and Reconstruction in Urban Traffic Department of Adaptive systems, Institute of Information Theory and Automation,
Editing RT QC flag in delayed mode ? Virginie Thierry DMQC 4 Toulouse, 28 septembre 2009.
EGO – COST - Groom-gliders Kiel meeting Kiel, June 2014 Data-management activity n Within Groom and with COST support, the glider data- management group.
Status report from the Lead Centre for Surface Processes and Assimilation E. Rodríguez-Camino (INM) and S. Gollvik (SMHI)
Prescription Mode DDI Supporting Use Cases Joe W. Tevis.
Documentation from NcML to ISO Ted Habermann, NOAA NESDIS NGDC.
IOOS National Glider Data Assembly Center
Configuring NVODS Datasets in LAS Joseph McLean TMAP/PMEL/NOAA and JISAO/UW OAR WebShop 2002.
NPOESS Enhanced Description Tool - “ned” Richard E. Ullman NASA/GSFC/NPP NOAA/NESDIS/IPO Data / Information Architecture Algorithm / System Engineering.
Annual Meeting, June , Istanbul, Turkey WP2: Data aggregation and products generation in the Atlantic Sea: results and criticalities EMODnet.
1 Quality Attributes of Requirements Documents Lecture # 25.
Summary of the SMIng WG The Next Generation Structure of Management Information.
CERN IT Department t LHCb Software Distribution Roberto Santinelli CERN IT/GS.
1 NODC Quality Control : Automatic Checks - reveal systematic errors in incoming data and metadata - eliminate most non-representative data from consideration.
General Objective: Conduct R&D activities to improve the quality of SST products used by MERSEA modeling and assimilation centers and produce global, Atlantic.
Monte-Carlo based Expertise A powerful Tool for System Evaluation & Optimization  Introduction  Features  System Performance.
Written by Changhyun, SON Chapter 5. Introduction to Design Optimization - 1 PART II Design Optimization.
10 th Argo data management 2009 Toulouse What is new at GDACs ?
QARTOD in Practice Luke Campbell, Software Engineer, RPS ASA.
1 09/2003 Processing Library Update CF Checker – Script made available as a web based form on the BADC site -
Andrea Kaiser-Weiss, Melbourne Joint GHRSST Workshop, 6 th March 2012 Experiences with SST profiles from near-surface Argo measurements A. Kaiser-Weiss.
Norwegian Meteorological Institute met.no QC2 Status
CTD Data Processing Current BIO Procedure. Current Processing Software Matlab Migrating to R & Python Code Version Control SVN Migrating to GitHub.
SeaDataNet Technical Task Group meeting JRA1 Standards Development Task 1.2 Common Data Management Protocol (for dissemination to all NODCs and JRA3) Data.
Assumptions Limitations
Overview Modern chip designs have multiple IP components with different process, voltage, temperature sensitivities Optimizing mix to different customer.
Inside a typical astrochemical model
Managed Objects for Packet Sampling
ROM SAF PTM18 IEEC, Barcelona
Visualization Shading
Argo Delayed-Mode Salinity Data
EMODnet Chemistry themes: EMODnet Chemistry objectives:
Parametric calibration of speed–density relationships in mesoscopic traffic simulator with data mining Adviser: Yu-Chiang Li Speaker: Gung-Shian Lin Date:2009/10/20.
Using Model Builder in ArcGIS
Data Management: Documentation & Metadata
CdmCL, a Specific Textual Constraint Language for Common Data Model
Michael Schäfer, Mark van der Loo & Olav ten Bosch
Ying shen Sse, tongji university Sep. 2016
National Center for Atmospheric Research
Generalizations of Markov model to characterize biological sequences
Graphic File Format Skill Area
Data aggregation and products generation
Discussion on product interoperability: from CARD4L to MRI
Traceability of the GIRO to the ROLO: Definition of a benchmark
Scientific Methods Notes
Emulator of Cosmological Simulation for Initial Parameters Study
Generation of data products in the Norwegian and Barents Seas
VERY DEEP CONVOLUTIONAL NETWORKS FOR LARGE-SCALE IMAGE RECOGNITION
A handbook on validation methodology. Metrics.
Presentation transcript:

Outline RTQC goals to achieve Description of current proposal Tests Tool Output

RTQC Goal Define a set of criteria that glider datasets should meet to be considered as usable According to glider observing capabilities Implement a tool that validates datasets against the defined set of criteria. Do not modify data, just flag it. Other tools can make use of it to apply corrections if required

RTQC Tests Tests to be applied on glider data have to be defined (what criteria should the data meet?). Initial guess made up now is a translation and adaptation of some of the tests for ARGO profilers (Thanks to Justin) Still some specific tests are missing, like the freezing point test.

RTQC Tests But tests will have to be adapted to glider observing capabilities: Resolution, both temporal and spatial are typically higher than ARGO profilers. Sensors specificities pose problems less visible on other platforms. Platform movement allows to observe different processes.

RTQC tool A Matlab tool has been developed to this end. Given a dataset and a QC configuration Applies defined QC Tests over the data. Outputs the result on a NetCDF file, containing the original information plus the ancillary information about data quality.

RTQC tool The tool is configurable by means of a JSON file. The configuration contains a list of parameters to be tested. Each parameter contains an ordered list of tests to be applied on the data.

RTQC tool Each test contains: Type of test to be applied. Parameters (thresholds, windows, etc.). Constraints on data that should pass the test. Flag to apply if test fails (suspicious, bad, etc.).

RTQC tool By now, 4 test types are implemented: Valid range Gradient Spike Stationary

RTQC tool Generalizations to previous QC tests definitions are introduced here, adding a reference variable: Range test is sample independent. Gradient, spike and stationary tests depend on sampling rate, which can by varying. A reference variable is added at this point. Reference variable can be time, depth, etc.

RTQC tool Constraints on the test allow to restrict applicability of a test to only certain conditions: Regions Depth ranges Time (for seasonality, still to be implemented)

RTQC tool Examples (separatedly)

RTQC tool Density inversion test: Build folding profiles: From surface to deepest, at each point compute the maximum density value. From deepest point to surface, at each point compute the minimum density value. Given the two folding profiles, wherever the difference exceeds a defined threshold, it is flagged as a density inversion.

RTQC Tool output The output of the RTQC process is dumped on a NetCDF file. It has to be adequately documented and compliant with conventions.

RTQC Tool output Scientific variable needs to have documented in it that there is QC information available. Attribute “QC_indicator” can exist and have one value for the whole variable Attribute “ancillary_variables” can exist and have a space-separated list of variable names

RTQC Tool output QC variable needs to be documented as well: Attribute standard_name is built with the standard name of the variable it refers to plus a modifier: “status_flag”. Attributes “flag_values” and “flag_meanings” document the table of possible values for this variable.

Conclusions A RTQC tool has been developed. Basic tests are implemented. Easy configuration. Need advise of scientists with regional expertise and glider technicians to fine tune the tests to be applied.