1 Implementing QA/QC Standards for In Situ Ocean Sensors Using OGC-Sensor Web Enablement a.k.a. QARTOD to OGC a.k.a. Q2O Project Status Brief to NOAA IOOS Program January 29, 2009

2 Once upon a time ….

3

4 Community activities were addressing individual parts…

5 Bringing together community members as Q2O Team Janet Fredericks - WHOI, MMI, MVCO, QARTOD Mike Botts/Tony Cook - UAH, OGC SWE Julie Bosch - NOAA, MMI, IOOS DMAC, QARTOD Harvey Seim/Sara Haines - SECOORA, NCCOOS Philip Bogden/Eric Bridger - GoMOOS, IOOS DMAC, SURA, MMI, OOSTethys Luis Bermudez - SURA, OOSTethys, OpenIOOS, MMI, OIE

6 What is Q2O? Funded by NOAA CSC/IOOS (January 2008 – December 2010) Deliverables: Implement the QARTOD recommendations into the OGC Sensor Web Enablement framework providing SensorML profiles for QARTOD tests and Documenting results by providing a tutorial and Test the deliverables by implementing services at participating data centers Methodology: Bring together IT specialists with domain experts (for waves, in situ currents, CTD observations and Dissolved Oxygen) Partner with community building projects such as OOSTethys and MMI

7 What does that mean? What do we have (know) to start with? A sensor (wave buoy or ADCP) with certain characteristics A sensor history QA info associated with a sensor Deployment characteristics Methods to process the data QC Tests to apply to the data … What information can we provide to data users via systems (OOSTethys, OpenIOOS)? What sensors we have available as a service Description of the sensor Description of where / how / when it is deployed List of the processing methods used on the data List of the QC tests applied The criteria used in the QC tests The results of the QC tests The data … How do we convey that information in SOS? Get Capabilities Lists available data offerings Returns SensorML Describe Sensor -Provides sensor and deployment characteristics and processing methods -Returns SensorML Get Observation -Provides the data, test results and points to file with processing/test info -Returns O&M

8 What had/has to be done by Q2O? QARTOD - WAVES - QC RECOMMENDED TESTS Engage QARTOD experts Gather QARTOD information – identify recommendations Define Processes – input / output / criteria Develop vocabularies Convey as SWE instances Test implementation Develop Guidance Applied by OOS implementers

9 QARTOD is a multi-organization effort to address the Quality Assurance and Quality Control issues of the Integrated Ocean Observing System (IOOS). Motivation Expected increase in the number of observations, observing systems, and users Need simple, accurate and consistent quality descriptions for the real-time data to convey expected level of data quality to users Engaging the community to develop standards

10 Activity initiated by NOAA NDBC and CO-OPS in 2003 Series of Workshops Workshop format with plenary and breakout sessions Approach to tackling the QA, QC, and metadata issues evolved From separate breakout sessions for QA, QC, and metadata for all observations combined To breakout sessions by observation focus group (covering QA, QC, and metadata) Each observation focus group addresses the same guidance questions Participation Approximately 80 participants per workshop Primarily observing system operators and data managers Representation from federal agencies, oceanographic institutions, universities, and instrument manufacturers

11 QARTOD I: December 3-5, 2003 NDBC, Stennis Space Center, MS Task to develop minimum standards for QA/QC methods and metadata Considering existing observing systems and existing practices QARTOD II: February 28-March 2, 2005 CO-OPS, Norfolk, VA Focus on wave, in situ current and remote current (HF Radar) measurements Identify unique calibration, metadata and QA/QC needs QARTOD III: November 2-4, 2005 SIO, La Jolla, CA Continued waves, in situ currents and remote currents work Added CTD focus group Primarily addressed QC with some metadata emphasis QARTOD IV: June 21-23, 2006 WHOI, Woods Hole, MA Continued wave, in situ current and CTD work Added DO focus group Primarily addressed QA; increased emphasis on capturing metadata requirements International considerations The QARTOD Workshops

12 Some activities to date … Kick-off Meeting - Feb 08 Q2O, QARTOD Leads, OGC SWE and OIE Waves and Currents Experts Meeting - Jun 08 Potential Waves Implementers and Vocabulary Experts Meeting - Aug 08 OOSSI Workshop - Waves and Currents Vocabulary and SWE Review - Nov 08 Waves and Currents Hands-on SWE Implementation - Feb 09 Initial Draft Q2O Waves Vocabulary Initial SensorML Demonstration SWE Concept Demonstrations / Use Case Scenarios Engaged waves implementers (COMPS, VIMS, CDIP, NDBC) in vocabulary and SWE development Draft Q2O vocabulary registered and Lessons learned provided to implementers; Refining SWE Content and test processes Practical Guidance to implementers Initial SensorML Instance & SOS implementation MVCO Live SensorML & SOS offerings

13 Take QARTOD tests and flags and code them into OGC/SWE instances. Each test is defined as a process that is described with input and output E.g., RangeTest has inputs (data) and parameters (min, max) and outputs (data, QC flag) The details…

14 The details… defining processes

15 The details… developing vocabularies

16 The details… on the Q2O Project website materials available to public - account access (working materials)

17 Status… Draft general QA/QC vocabulary (including waves tests) registered at MMI Draft MVCO waves implementation nearing completion Lessons learned –vocabulary development –generation and registration of vocabulary/ontology –SWE process chains, parameters, components,… Beginning of Guidance document and example files Waves implementers signed on (COMPS and VIMS) Gaps identified for QARTOD community to address

18 What's next? Q2O – develop generic waves profile from Q2O instances; get cookbooks online; complete tutorial; continue with QARTOD currents, CTD and DO. Demonstrate computation of waves parameters from triplet processing QARTOD – demonstrate Q2O project to QARTOD community and bring back better defined tests and algorithms (discussion of gaps for implementation) IOOS DIF Coordination Ingest SWE observations with QA/QC descriptions and flags; evaluate and apply additional tests; document in SWE offerings? OOSTethys into OpenIOOS - utilize QA/QC flags SUGGESTIONS?? (and Thank you!!)