Real-time Verification of Operational Precipitation Forecasts using Hourly Gauge Data Andrew Loughe Judy Henderson Jennifer MahoneyEdward Tollerud Real-time Verification System (RTVS) of NOAA / FSL Boulder, Colorado USA

Outline Some approaches to objective verification How we perform automated precipitation verification What we mean by "real-time" Forecasts + Obs --> Results disseminated over the web. (The steps involved) QC, Model comparisons, Statistical displays Future direction

If you don't have objective data you are just another person with an opinion

Our Approach? Basically, we're gross! No really, we are... We process 4,500 gauge measurements each hour of every day. On average we retain 2,800 "good" reports. That's 67,000 observations per day, 200,000 per month, and over 6 million per season.

The Real-time Verification System An independent, real-time, automated data ingest and management system Gauge observations received each hour of every day (~4500) Gross error check on observations is performed Model forecasts interpolated to the observation points Results stored in 2 x 2 contingency tables of forecast / observation pairs (YY, YN, NY, NN) Graphics, skill scores and contingency information disseminated over the WorldWide Web

Alternative Approaches (Should be objective) Grid-to-grid verification We're game... but not yet! More fair to the modelers Less fair to the end-users of the forecast? More representative of the areal coverage of precipitation Can do pattern matching and partitioning of the error (Ebert, et al.) or studies of representativeness error (Foufoula et al.)

What about Case Studies? Do you fish with a pole or do you fish with a net? We fish with a net Case studies are insufficient for evaluating national-scale forecast systems Subjective analyses often focus on where forecasts work well, and not on where they work poorly There exists a need to assess variability on many time and space scales (from daily to seasonal) Timely and objective information is needed for decision making

Realtime or Near-Realtime? Realtime processing... Monthly and Seasonal dissemination of results (for now) Gauge data stored in hourly bins Model data interpolated once the observations catch up (Models initialized as late as 18Z, and then 24h forecasts are made) Data collected over numerous accumulation periods

Go with the flow... I. Obtain gauge data and collect it into hourly bins / Match data with list of "good" stations (QC'd list) II. Interpolate model data to "good" observation points III. Accumulate precipitation over 3, 6, 12, 24 hours IV. Compute contingency pairs (YY, YN, NY, NN) V. Process these contingency data to create plots of ESS and Bias for Eta and RUC2 VI. Make these displays and the associated statistical information available through the web

A Point-Specific Approach (Eta at 40 km)

Gauge Data Checked for Accuracy ¢ Hourly gauge data are checked for accuracy vs. radar, 24h totals, nearest neighbor ¢ Further data are included through in-house QC efforts

Forecast / Observation Comparisons Comparisons made at numerous thresholds from 0.1 to 5.0 inches Comparisons made over 3, 6, 12, and 24h accumulation periods

2x2 Contingency Tables

Results Available over the Web www-ad.fsl.noaa.gov/afra/rtvs/precip ¢ Specify parameters... obtain graphical result ¢ View contingency tables stored on disk

The Future! Access and Displays via Database (Model Icing Forecasts) ¢ Specify parameters ¢ Display results (gnuplot) via database query (MySQL)

Are these methods sufficient? Trade off between dealing with the specifics and dealing with the general (rifle vs. shotgun) Method is not discretized by region or event Density of observations is not smooth Although method is straightforward, there still is a lack of understanding for what the skill scores represent May tell you which forecast system is "better", but not why

Future Plans Add more models to this point-specific approach, and provide a measure of confidence Perform verification using a gridded, analyzed precipitation field (Stage IV Precipitation) Verify the probabilistic forecasts of ensembles Move verification data into the relational database and compute results on-the-fly Relate verification results geographically Access verification results as soon as the forecast period ends (timeliness)

Contd,... Test and extend QC of the observations Currently we are: Assessing skill using East-only and West-only hourly station data Assessing skill using full RFC and the in-house QC methods Assessing skill using no QC methods whatsoever Comparing these four experimental results

Problem Not Reporting "Zero" Precipitation?

The Affect on Precipitation Verification