Verification in NCEP/HPC Using VSDB-fvs Keith F. Brill November 2007

2 Overview Introduce VSDB-fvs VSDB component fvs component Examples of fvs output

3 VSDB-fvs Major Components VSDB = Verification Statistics DataBase fvs = forecast verification system VSDB-fvs refers to a collection of software 1. Scripted programs to create VSDB records Grid (GEMPAK) to grid comparison (“perfect obs” assumption) Polygon (VG) to grid comparison Grid to point (GEMPAK) comparison Image overlay visual comparison as a by-product 2. fvs ( command line or scripted ) Define VSDB search conditions Search VSDB for matching data Compute and display (GEMPLT) performance measures

4 Brief History of VSDB-fvs ~1995: EMC begins to specify VSDB format 1997: I began coding fvs in the EMC 1998: I delivered first version of fvs 1999: I accepted a position in HPC 1999 – present: I continued the following...  Support fvs for EMC use  Develop HPC oriented VSDB components  Support fvs for HPC use  Add new capabilities to fvs as time permits

5 VSDB Records Space delimited text, single-line records, multiple records per file 9 identifier fields “=” count_value data_value1... data_valueN Number of data values depends on statistic type in ID field 7 data_values (summary statistics) are sums scaled by count_value  these may be combined arbitrarily by fvs  count_value is number of ob-forecast pairs used for data Example record (split into two lines): V01 WWD/ CSQ G212 PBS_WWD/4 SF SFC = vrsn frcst_source fhr valid_date-time obs_src region statistic_type parm level count Defect: “Perfect obs” assumption.

6 Partial List of Statistic Types and Corresponding Data L1L2 – means of forecasts, observations, their squares, their products, and absolute error. FHO – fractions of events Forecast, Hit, and Observed for dichotomous forecasts PHSE – east-west phase error: phase error (km), amplitude error, variances, analysis phase angle and amplitude, more variances PBS – partitioned Brier Score: fractions forecast in probability categories followed by fractions both forecast and observed in categories

7 HPC Verification Regions Defect: No stratification by elevation in West.

8 fvs – the tool for using VSDB data “fvs u” lets user set search conditions  specify statistic type  specify lists of identifier strings for data combination (forecast hours, time, areas, etc.) “fvs s” executes the search through VSDB  applies event equalization consistency checking “fvs p”  computes user requested performance measures  displays graphics under user control  performs statistical significance test for paired data fvs can be run in “batch” mode to populate web pages

9 Examples of Graphs Produced Using VSDB-fvs

10 Example Visual Comparison For HPC Basic Wx Forecasts of Broken Precipitation Purple is QPE based estimate Broken Precip area forecast

11

12 Significance Test: Does HPC beat the 00Z ENS mean? STATISTICAL SIGNIFICANCE ANALYSIS FOR PAIRED DATA FROM TWO TRACES STAT=SSAL1L2 PARAM=SLP V_ANL=HPC/SFC V_RGN=MRDG VHHMM= TO TRACE # 1 MODEL=MEDR/ FHOUR=120 SCALR_CORR TRACE # 2 MODEL=ENSMN FHOUR=132 SCALR_CORR Null Hypothesis (trace1 = trace2): The mean of the differences of the paired values from the two traces is zero. Alternative Hypothesis (trace1 > trace2): The mean of the paired differences exceeds zero. Gaussian probability of wrongly rejecting null hypothesis: Test Level Decision (one-sided Gaussian) REJECT null hypothesis REJECT null hypothesis REJECT null hypothesis REJECT null hypothesis REJECT null hypothesis ZRAW = E+00 ZTST = E+00 N = 90 Lag1 Corr =

13 Significance Test: Does HPC beat the 12Z ENS mean? STATISTICAL SIGNIFICANCE ANALYSIS FOR PAIRED DATA FROM TWO TRACES STAT=SSAL1L2 PARAM=SLP FHOUR=120 V_ANL=HPC/SFC V_RGN=MRDG VHHMM= TO TRACE # 1 MODEL=MEDR/ SCALR_CORR TRACE # 2 MODEL=ENSMN SCALR_CORR Null Hypothesis (trace1 = trace2): The mean of the differences of the paired values from the two traces is zero. Alternative Hypothesis (trace1 > trace2): The mean of the paired differences exceeds zero. Gaussian probability of wrongly rejecting null hypothesis: Test Level Decision (one-sided Gaussian) ACCEPT null hypothesis ACCEPT null hypothesis ACCEPT null hypothesis ACCEPT null hypothesis ACCEPT null hypothesis ZRAW = E-01 ZTST = E-01 N = 88 Lag1 Corr =

14

15

16

17

18

Brier Score Decompositon Attribute Diagram Perfect Reliability Zero Resolution Histogram shows forecast use frequency NO SKILL Brier Score Brier Skill Score NO SKILL SKILL NO SKILL SKILL Sample Climatological Frequency

20 SUMMARY VSDB-fvs is an end-to-end system for verifying forecast objects against analyses or points VSDB is text data base of summary statistics VSDB data are combinable in a variety of ways fvs searches for VSDB data and combines it under user control fvs computes performance metrics and displays them in a variety of ways