1. Challenges in Centralized Aggregated Data Quality Control
Richard L. Crout, PhD
NOAA NWS National Data Buoy Center
Stennis Space Center, MS
10 March 2006

2. Overview Process Overview System Challenges Data Volume Challenges
Data Diversity Challenges
Data Quality Assurance Challenges
Data Quality Control Challenges
Metadata
Implementation Challenges
Summary

4. Data Volume 2005 2011 10,000 obs/day Equal contributions NDBC/non-NDBC
~4M obs/year total
2011
30,000 obs/day
More NDBC obs
~11 M obs/year
NDBC is sizing its assets for the future.

5. Data Diversity Eighty (88) met and ocean buoys
Fifty-six (56) met C-MAN stations.
Seven (7) Deep-ocean Assessment and Reporting of Tsunamis (DART) buoys
Five (5) more are leaving Fort Pierce, FL for Atlantic and Caribbean.
Eight hundred plus (800+) VOS ships
Fifty-five (55) TAO moorings
Fifty-six (56) oil platforms and drilling rigs
Ocean profiling floats and drifters
Regional observatory data
NOS/NWLON stations
Will host an IOOS National HF Radar Server

6. Data Diversity Wind Speed Wind Direction Peak Wind Air Temperature
Atmospheric Pressure
Dew Point Temperature
Solar Radiation (LW/SW)
Precipitation Rate
Sea Surface Temperature
Conductivity
Salinity
Significant Wave Height
Wave Period
Wave Spectra
Ocean Currents
Sea Level
Water Column Height
Chlorophyll
Turbidity
pCO2 pH Eh

7. Data Quality
Each platform has seasonal and regional min/max values associated with each sensor. Complete record of what is at the station.
Operational 24/7 Data Assembly Center (DAC)
Comparison to climatology, imagery and model output (when available, NCEP reanalysis of MET data now)
Ex. MMS ADCP

8. Initial MMS ADCP QA/QC
Operators, MMS, Teledyne RDI and NDBC Teleconference
38 kHz and 75 kHz Acoustic Doppler Current Profilers (ADCP) only, external Power
Single-profile based
Two-Tier Approach
Thresholds at Each Bin (Tier 1)
Overall Profile (Tier 2)
Internal Built-In Test (BIT)

9. Initial QA/QC: Bin Thresholds (Tier 1)
>2.5 m/sec
m/sec
<1.25 m/sec
Horizontal
Velocity (u & v)
>50 cm/sec
30-50 cm/sec
< 30 cm/sec
Vertical Velocity
(each bin)
38N: 1 beam >110
38W: 1 beam >190
75: 1 beam > 64
38N: 2 beams >110
38W: 2 beams >190
75: 2 beams > 64
38N: 3 beams > 110
38W: 3 beams > 190
75: 3 beams > 64
Correlation Magnitude (counts)
38: <22%
75: <5%
38: 22-25%
75: 5-10%
38: >25%
75: >10%
Percent Good
>30 cm/sec
15-30 cm/sec
< 15 cm/sec
Error Velocity
Hard Flag (1)
SoftFlag (2)
Pass (3)

10. Initial QA/QC: Overall Profile (Tier 2)
Echo Intensity (After 15th bin)
No adjacent bin differences >30 counts for any beam
Adjacent bin differences >30 on 1 beam
Adjacent bin differences >30 on 2+ beams
Overall Profile Evaluation
Pass: All bins passed all Tier 1 tests
Soft Flag: Fewer than half bins flagged
Hard Flag: More than half bins flagged
NDBC Flag Implementation
Flag and release all data via web
Do not release hard-flagged data to GTS in TESAC code
No plots of hard-flagged data

11. MMS Data Quality Flags 192121113 200 208 48 42 47 MM 1157 35 1 50 18
200
208 48 42 47 MM
1157 35 1 50 18 27 2
2006
198
223 40 49
1125 34 39
202
243 46 52 19
-0.1
4.9
171
1093 33
212
225
239
211 58 51 99
-1.6
-2.1
4.6
231
1061 32
219
235
222 53 68 84
0.1
-1.9
4.5
230
1029 31
218
229 79 82 54
-0.3
-0.5 7
300
997 30
224
221
232 66 70 62 59
100
0.9 6
293
965 29
228 71 57 64
0.8
-1.1 8
313
933 28
241
110
105
107
-0.8
3.1
9.2
216
645
145
139
140
141
11.1
246
453 13
240
152
154
137
-0.2
12.3
421 12
242
126
151
124
11.5
254
389 11
236
122
138
142
120
12.5
238
357 10
119
127
130
115
1.2
217
325 9
123
118
-0.4
0.7
16.4
125
129
135
0.4
22.3
214
261
245
234
156
147
149
157 -1
20.5
207
237
163
164
160
-2.2
16.2
210
197 5
166
-1.5
12.1
165 4
169
2.4
11.7
133 3
168
173
15.3
101
233
227
194
190
191
-2.6
19.7 69
Flags
CM4
CM3
CM2
CM1
EI4
EI3
EI2
EI1 %
%Good4
VerVl
ErrVl
Speed
Dir
Dept
Bin I mm hh DD M
YYYY

12. QA/QC Comparison

13. Follow-on QC Efforts
Adapt strategy to higher frequency (and battery powered) current profilers and all current profiler data processed by NDBC
Large surface current and current profiler expansion underway at NDBC
Time continuity and other processing dependent on previous report
Spatial Checks
Model Comparisons (In-House only)
Training of NDBC DAC Staff for Man/machine QA/QC
QARTOD/COTS Demonstration Compliant

14. Implementation Challenges
Contractor tasked
Code is written
Code is tested by IT
Code is tested by “tester”
Code is tested within the processing stream on the backup system
Code is implemented on the operational system
Implementation takes time (and resources)

15. Metadata Strong proponent Participate in DMAC workshops, etc.
Generally collected with first data
Credit given on webpage

16. Summary Large volumes Diverse platforms, sensors, formats, sources
Quality control techniques must be shared
Implementation of QC is costly, unless you have slave labor
NDBC has experience to share (failures as well as successes)
Focusing NDBC on being the National DATA Buoy Center

17. Questions?
Contact Info:
Richard L. Crout, Ph.D.
Oceanographer
National Data Buoy Center
Stennis Space Center, MS 39529

18. 5-Day Current Plots Ocean Baroness and Front Runner

19. QA/QC Comparison