1. Note: This presentation is being recorded for training purposes. 1 AHPS River Data Quality Control Techniques and Tools OCWWS/HSD Hydrologic Support Branch December 15, 2011

2. Note: This presentation is being recorded for training purposes. 2 Outline  Introduction  Solution  Review of Quality Control (QC) Operations  Operational Discussion Identifying...Problem Data Identifying...Problem Data Reporting...Problem Data Reporting...Problem Data Correcting...Problem Data Correcting...Problem Data  Conclusion

3. Note: This presentation is being recorded for training purposes. 3 Intro: Problem  Bad observed data at water.weather.gov  Data is very visible Evident on AHPS maps/graphs/tables Evident on AHPS maps/graphs/tables Increased use of web info brings greater scrutiny Increased use of web info brings greater scrutiny Data used by public, countless partners at all levels Data used by public, countless partners at all levels  Bad data characteristics Usually involves data spikes Usually involves data spikes Focus is on observed river data Focus is on observed river data Forecast and precipitation data on web not covered Forecast and precipitation data on web not covered Current WHFS tools provide only limited support Current WHFS tools provide only limited support

4. Note: This presentation is being recorded for training purposes. 4 Intro: Goals 1) Eliminate bad data presented to public 2) Ensure reliable observations, best forecasts 3) Provide decision support at local, regional, and national levels using web info 4) Minimize burden on WFO staff - provide automated tools where possible

5. Note: This presentation is being recorded for training purposes. 5 Solution: Summary QC process modeled as three fundamental steps: 1)Identify bad data automatically - Minimizes burden of manual review 2)Report/notice detected bad data - Reported to forecaster -Automated, still requires user to take action - Noticed by forecaster - Manual review can simultaneously identify/”report”... 3)Correct reported bad data - For AHPS, manually set data to missing (not delete)

6. Note: This presentation is being recorded for training purposes. 6 Solution: Principles  Focus on data spike problems Most “bad” data spikes detectable via rate-of-change (ROC) check Most “bad” data spikes detectable via rate-of-change (ROC) check But not all ROC-failed data are actually bad or questionable But not all ROC-failed data are actually bad or questionable Focus on spikes assumes range checks locally configured Focus on spikes assumes range checks locally configured  Maintain separation of QC and Alert-Alarm (AA) functions ROC thresholds exist for both QC and AA ROC thresholds exist for both QC and AA QC judges data quality, AA judges threat QC judges data quality, AA judges threat Difficulty in trusting precise thresholds for ROC complicates automated QC process Difficulty in trusting precise thresholds for ROC complicates automated QC process

7. Note: This presentation is being recorded for training purposes. 7 Solution: Details  Identify using rate-of-change (ROC) check Configure WHFS ROC_checker Configure WHFS ROC_checker Ensure thresholds are appropriate –Ensure thresholds are appropriate – Done by local office Set token so ROC-failed data set as questionable, not bad (qc_questorbad) – HSD Support will doSet token so ROC-failed data set as questionable, not bad (qc_questorbad) – HSD Support will do  Report using new QC report utility Complements Alert-Alarm report utility Complements Alert-Alarm report utility Configure new utility operations Configure new utility operations Set up new utility in existing cron - HSD Support will doSet up new utility in existing cron - HSD Support will do Set up trigger in textdb operations -Set up trigger in textdb operations - Done by local office Define QC product id in report script -Define QC product id in report script - Done by local office  Correct using WHFS TimeSeries application

8. Note: This presentation is being recorded for training purposes. 8 QC Review: Info Available  Web QC part of general hydro data QC and Alert-Alarm operations  Previous WHFS QC/Alert-Alarm webinar Given October 6, 2009 Given October 6, 2009 Powerpoint and recording available at: Powerpoint and recording available at: http://www.nwstc.noaa.gov/Hydrology/hydromain/webinars/index.html  Available WHFS documentation https://ocwws.weather.gov/intranet/whfs

9. Note: This presentation is being recorded for training purposes. 9 QC Review: QC Checks  All value set to one of three QC levels: Good, Questionable, Bad (G, Q, B) Good, Questionable, Bad (G, Q, B)  QC Checks in WHFS Rate-of-change Rate-of-change Only one ROC thresholdOnly one ROC threshold If exceeded, set to Q or B (token controlled)If exceeded, set to Q or B (token controlled) Range Range Separate thresholds for Q and BSeparate thresholds for Q and B Obs-Forecast Difference - set to Q Obs-Forecast Difference - set to Q Station Consistency Station Consistency New report utility on LADNew report utility on LAD Compares data from multiple sources at same stationCompares data from multiple sources at same station Does not set QC flagDoes not set QC flag Assume already setup ok New focus

10. Note: This presentation is being recorded for training purposes. 10 QC Review: Processing  Hydro data organized by type (obs, fcst, other) and physical element (PE)  River Data Access for AHPS Height / FcstHeight tables (stage) Height / FcstHeight tables (stage) Discharge / FcstDischarge tables Discharge / FcstDischarge tables RejectedData table *not* read RejectedData table *not* read Written by SHEFdecode for out-of-range data (per token shef_post_baddata)Written by SHEFdecode for out-of-range data (per token shef_post_baddata) ROC Bad data not moved there! (regardless of token)ROC Bad data not moved there! (regardless of token)  HydroGen Data Products Tables read by genXML function within HydroGen operations Tables read by genXML function within HydroGen operations HML product file generated and sent every 15 minutes! HML product file generated and sent every 15 minutes! HydroGen does NOT filter Bad data [say what!? change???] HydroGen does NOT filter Bad data [say what!? change???]

11. Note: This presentation is being recorded for training purposes. 11 QC Review: HydroBase Controls  QC thresholds defined as: Default values for physical element Default values for physical element Location-specific values override defaults Location-specific values override defaults  ROC thresholds ROC QC threshold should be set ROC QC threshold should be set ROC Alert-Alarm thresholds are separate ROC Alert-Alarm thresholds are separate Help identify hydromet threats, not bad dataHelp identify hydromet threats, not bad data  If sensor continually sends bad data, consider disabling the station or specific data elements  If sensor continually sends bad data, consider disabling the station or specific data elements Location->Modify Location-> Additional Info: disable station for AHPS and WHFS Location->Modify Location-> Additional Info: disable station for AHPS and WHFS Data Ingest->IngestFilter: disable specific element for AHPS and WHFS Data Ingest->IngestFilter: disable specific element for AHPS and WHFS Setup->HydroGen Configuration: disable station Setup->HydroGen Configuration: disable station Can also use AHPS CMS to... Can also use AHPS CMS to... Define annotated hydro note to describe data QC problemDefine annotated hydro note to describe data QC problem Disable station able using “Sensor in Service” optionDisable station able using “Sensor in Service” option

12. Note: This presentation is being recorded for training purposes. 12 Operational Discussion Hydro Database ---- Good Questionable Bad ROC_checker report_hydroQC TimeSeries HydroGen QC report product HML product Web Text Database Identify: Report: Correct: report_alarm Alert/Alarm report product NEW trigger

13. Note: This presentation is being recorded for training purposes. 13 Identify Bad Data: ROC Checks  Automated ROC checks Detects data spikes (QC) or quick changes indicating threat (AA) Detects data spikes (QC) or quick changes indicating threat (AA) Compares successive time series values Compares successive time series values Only second value in value pair being checked is flagged Only second value in value pair being checked is flagged Ignores B data, ignores Q data if program argument set Ignores B data, ignores Q data if program argument set  Data exceeding ROC threshold  Data exceeding QC ROC threshold Flagged as Q or B based on token qc_questorbad Flagged as Q or B based on token qc_questorbad Default behavior is to set to B Default behavior is to set to B Even if B, not moved to the RejectedData table so AHPS will still use it since AHPS does not filter bad data!!! Even if B, not moved to the RejectedData table so AHPS will still use it since AHPS does not filter bad data!!! HSD Support will change token to set to Q HSD Support will change token to set to Q

14. Note: This presentation is being recorded for training purposes. 14 Report Bad Data: Options Options for becoming aware of data QC problems: 1) Internal text reports using textdb notification: -New QC summary product -Existing AA summary product -Could set up dedicated report for only report ROC alert-alarms -Use –fRMT in run_roc_checker script -Must consider this since some “bad” data may actually only be flagged for alert-alarm purposes 2) Manual review using WHFS applications 3) Manual review using AHPS web pages themselves 4) Manual review using local CRH app 5) Other ideas?

15. Note: This presentation is being recorded for training purposes. 15 Report Bad Data: New QC Report  Name: report_hydroQC  Arguments: Table – Height, Discharge, or both Table – Height, Discharge, or both Lookback time – number of minutes (MM) Lookback time – number of minutes (MM) Location filter – all, HSA only, HSA AHPS only Location filter – all, HSA only, HSA AHPS only Other switches/filters to be added per request Other switches/filters to be added per request  Function: Reads database table(s) Reads database table(s) Considers all questionable or bad valuesConsiders all questionable or bad values Filters on location filterFilters on location filter Filters on time period ending at current time and beginning MM minutes earlierFilters on time period ending at current time and beginning MM minutes earlier Formats text product Formats text product Sends text product to textdb if Q/B values found Sends text product to textdb if Q/B values found

16. Note: This presentation is being recorded for training purposes. 16 Report Bad Data: New QC Report ** Report of Questionable/Bad Hydro Obs Data ** Period (Z): Tue 12-13 06:04:09 to Tue Dec 13 18:04:09 BOGW1 HG RP Dec 13 10:30 [ROC 1/3] (Bogachiel River near La Push) CENW1 HG RP Dec 13 12:00 [RNG 1/2] (Chehalis River at Centralia) CRNW1 HG RP Dec 13 10:30 [ROC 1/1] (Snoqualmie River near Carnation) UIL HZ RZ Dec 13 12:00 [SHF 0/1] (Quillayute) ALRW1 QR RZ Dec 13 15:00 [ROC 1/0] (Alder Dam) Created: 12-13-2011 18:04:09 GMT Reading Height, Discharge data. Shows latest obs time [QCreason questionable/bad value count] Sample QC Report:

17. Note: This presentation is being recorded for training purposes. 17 Notice Bad Data: WHFS Applications  HydroView LiveData-> Questionable/Bad window Questionable/Bad window Rejected Data (‘trash”) window Rejected Data (‘trash”) window  TimeSeries Tabular/graphical display Tabular/graphical display  River Monitor Automatically updates Automatically updates

18. Note: This presentation is being recorded for training purposes. 18 Sample HydroView Questionable/Bad Window

19. Note: This presentation is being recorded for training purposes. 19 Sample RiverMonitor Main Window “A” means alert/alarm In effect

20. Note: This presentation is being recorded for training purposes. 20 Sample RiverMon AlertAlarm Window

21. Note: This presentation is being recorded for training purposes. 21 Notice Bad Data: AHPS Pages  AHPS Page features “Area HydroGraph” feature “Area HydroGraph” feature Under “Additional Resources” headingUnder “Additional Resources” heading Allows quick, multi-station reviewAllows quick, multi-station review Monitor AHPS map on screen in office Monitor AHPS map on screen in office Snapshots only - must refresh Snapshots only - must refresh  Local CRH app is compact, auto- refresh of “area hydrograph: feature http://intra.crh.noaa.gov/metdat/ DDC/AHPS/ahps_qc.html http://intra.crh.noaa.gov/metdat/ DDC/AHPS/ahps_qc.html http://intra.crh.noaa.gov/metdat/ DDC/AHPS/ahps_qc.html http://intra.crh.noaa.gov/metdat/ DDC/AHPS/ahps_qc.html On METDAT server, under DDC/AHPS/ On METDAT server, under DDC/AHPS/

22. Note: This presentation is being recorded for training purposes. 22 Correcting Bad Data:TimeSeries  General functions  General functions Primary tool to edit data Primary tool to edit data Also very helpful for reviewing data Also very helpful for reviewing data Using the "pre-defined groups" mode, can be pre- configured to show select station data Using the "pre-defined groups" mode, can be pre- configured to show select station data  Correcting of data Data already sent to AHPS must be set to missing to correct it at the web farm database Data already sent to AHPS must be set to missing to correct it at the web farm database Next HydroGen run will retransmit data to web Next HydroGen run will retransmit data to web Presentation, titled "Data QC Procedural Change for WHFS", describing this: Presentation, titled "Data QC Procedural Change for WHFS", describing this:

23. Note: This presentation is being recorded for training purposes. 23 Sample TimeSeries Tabular Window

24. Note: This presentation is being recorded for training purposes. 24 Conclusion: HQ To-Do  HSD Support To-Do Will update token Will update token Will update script to create hydroQC product Will update script to create hydroQC product Will provide instructions for local changes Will provide instructions for local changes May do this as early as next week May do this as early as next week  Future Changes Will push for future AWIPS work to improve displays and tools for QC and situational awareness Will push for future AWIPS work to improve displays and tools for QC and situational awareness

25. Note: This presentation is being recorded for training purposes. 25 Conclusion: WFO To-Do  Configure Review QC/AA thresholds in HydroBase Review QC/AA thresholds in HydroBase Define triggers in textDB Define triggers in textDB Define product id in report_hydroQC script Define product id in report_hydroQC script  Review data regularly RiverMon/PrecipMon, AA reports, HydroView, TimeSeries… RiverMon/PrecipMon, AA reports, HydroView, TimeSeries…  Correct Set AHPS data to missing...don’t delete Set AHPS data to missing...don’t delete

26. Note: This presentation is being recorded for training purposes. 26 Conclusion  Questions?  Contact: whfs@noaa.gov  Thanks