Challenges in Centralized Aggregated Data Quality Control Richard L. Crout, PhD NOAA NWS National Data Buoy Center Stennis Space Center, MS 10 March 2006 Richard.Crout@noaa.gov

Overview Process Overview System Challenges Data Volume Challenges Data Diversity Challenges Data Quality Assurance Challenges Data Quality Control Challenges Metadata Implementation Challenges Summary

Data Volume 2005 2011 10,000 obs/day Equal contributions NDBC/non-NDBC ~4M obs/year total 2011 30,000 obs/day More NDBC obs ~11 M obs/year NDBC is sizing its assets for the future.

Data Diversity Eighty (88) met and ocean buoys Fifty-six (56) met C-MAN stations. Seven (7) Deep-ocean Assessment and Reporting of Tsunamis (DART) buoys Five (5) more are leaving Fort Pierce, FL for Atlantic and Caribbean. Eight hundred plus (800+) VOS ships Fifty-five (55) TAO moorings Fifty-six (56) oil platforms and drilling rigs Ocean profiling floats and drifters Regional observatory data NOS/NWLON stations Will host an IOOS National HF Radar Server

Data Diversity Wind Speed Wind Direction Peak Wind Air Temperature Atmospheric Pressure Dew Point Temperature Solar Radiation (LW/SW) Precipitation Rate Sea Surface Temperature Conductivity Salinity Significant Wave Height Wave Period Wave Spectra Ocean Currents Sea Level Water Column Height Chlorophyll Turbidity pCO2 pH Eh

Data Quality Each platform has seasonal and regional min/max values associated with each sensor. Complete record of what is at the station. Operational 24/7 Data Assembly Center (DAC) Comparison to climatology, imagery and model output (when available, NCEP reanalysis of MET data now) Ex. MMS ADCP

Initial MMS ADCP QA/QC Operators, MMS, Teledyne RDI and NDBC Teleconference 38 kHz and 75 kHz Acoustic Doppler Current Profilers (ADCP) only, external Power Single-profile based Two-Tier Approach Thresholds at Each Bin (Tier 1) Overall Profile (Tier 2) Internal Built-In Test (BIT)

Initial QA/QC: Bin Thresholds (Tier 1) >2.5 m/sec 1.25-2.5 m/sec <1.25 m/sec Horizontal Velocity (u & v) >50 cm/sec 30-50 cm/sec < 30 cm/sec Vertical Velocity (each bin) 38N: 1 beam >110 38W: 1 beam >190 75: 1 beam > 64 38N: 2 beams >110 38W: 2 beams >190 75: 2 beams > 64 38N: 3 beams > 110 38W: 3 beams > 190 75: 3 beams > 64 Correlation Magnitude (counts) 38: <22% 75: <5% 38: 22-25% 75: 5-10% 38: >25% 75: >10% Percent Good >30 cm/sec 15-30 cm/sec < 15 cm/sec Error Velocity Hard Flag (1) SoftFlag (2) Pass (3)

Initial QA/QC: Overall Profile (Tier 2) Echo Intensity (After 15th bin) No adjacent bin differences >30 counts for any beam Adjacent bin differences >30 on 1 beam Adjacent bin differences >30 on 2+ beams Overall Profile Evaluation Pass: All bins passed all Tier 1 tests Soft Flag: Fewer than half bins flagged Hard Flag: More than half bins flagged NDBC Flag Implementation Flag and release all data via web Do not release hard-flagged data to GTS in TESAC code No plots of hard-flagged data

MMS Data Quality Flags 192121113 200 208 48 42 47 MM 1157 35 1 50 18 200 208 48 42 47 MM 1157 35 1 50 18 27 2 2006 198 223 40 49 1125 34 192123333 39 202 243 46 52 19 -0.1 4.9 171 1093 33 393333333 212 225 239 211 58 51 99 -1.6 -2.1 4.6 231 1061 32 219 235 222 53 68 84 0.1 -1.9 4.5 230 1029 31 218 229 79 82 54 -0.3 -0.5 7 300 997 30 224 221 232 66 70 62 59 100 0.9 6 293 965 29 228 71 57 64 0.8 -1.1 8 313 933 28 241 110 105 107 -0.8 3.1 9.2 216 645 393333330 145 139 140 141 11.1 246 453 13 240 152 154 137 -0.2 12.3 421 12 242 126 151 124 11.5 254 389 11 236 122 138 142 120 12.5 238 357 10 119 127 130 115 1.2 217 325 9 123 118 -0.4 0.7 16.4 125 129 135 0.4 22.3 214 261 245 234 156 147 149 157 -1 20.5 207 237 163 164 160 -2.2 16.2 210 197 5 166 -1.5 12.1 165 4 169 2.4 11.7 133 3 168 173 15.3 101 233 227 194 190 191 -2.6 19.7 69 Flags CM4 CM3 CM2 CM1 EI4 EI3 EI2 EI1 % %Good4 VerVl ErrVl Speed Dir Dept Bin I mm hh DD M YYYY

QA/QC Comparison

Follow-on QC Efforts Adapt strategy to higher frequency (and battery powered) current profilers and all current profiler data processed by NDBC Large surface current and current profiler expansion underway at NDBC Time continuity and other processing dependent on previous report Spatial Checks Model Comparisons (In-House only) Training of NDBC DAC Staff for Man/machine QA/QC QARTOD/COTS Demonstration Compliant

Implementation Challenges Contractor tasked Code is written Code is tested by IT Code is tested by “tester” Code is tested within the processing stream on the backup system Code is implemented on the operational system Implementation takes time (and resources)

Metadata Strong proponent Participate in DMAC workshops, etc. Generally collected with first data Credit given on webpage

Summary Large volumes Diverse platforms, sensors, formats, sources Quality control techniques must be shared Implementation of QC is costly, unless you have slave labor NDBC has experience to share (failures as well as successes) Focusing NDBC on being the National DATA Buoy Center

Questions? Contact Info: Richard L. Crout, Ph.D. Oceanographer National Data Buoy Center Stennis Space Center, MS 39529 Richard.Crout@noaa.gov 228-688-1021

5-Day Current Plots Ocean Baroness and Front Runner

QA/QC Comparison