1. Development of an Uncertainty Tool to Assess Model Forecast Parameters
Taylor Mandelbaum1, Brian Colle1, Trevor Alcott2
1Stony Brook University School of Marine and Atmospheric Sciences, Stony Brook, NY
2Earth Systems Research Laboratory, Boulder, CO
* This work is supported by NOAA-CSTAR

2. Background
The Ensemble Situational Ensemble Table (ESAT) plots anomalies in multiple formats (using NAEFS and GEFS).
The goal of ESAT is to provide a tool to assess anomalies in ensemble forecasts.
DSS and forecasters can determine how anomalous a forecast is relative to previous events.

3. Terminology

4. The Motivation
The ensemble mean is often regarded as the “best guess” forecast
Conventional spread can provide uncertainty information to a forecaster
There is a lot of information that is compressed in ensemble spread
The North American Ensemble Forecast System (NAEFS, GEFS + CMC) 96 hour mean, valid February 11, 2010 at 00z.

5. The Motivation
R-Climate and M-Climate metrics on the ESAT show the anomaly of the forecast (in both reanalysis and model relative formats) but not the anomaly of uncertainty.
The spread of a forecast for a similar anomaly can be compared to M-Climate to assess anomaly.
If the anomaly is less than “normal”, it should translate to less uncertainty.
ERA Interim analysis and mean absolute error, valid February 11, 2010 at 00z.

6. The Motivation
The goal is to create a “standardized spread anomaly” which can be used operationally but…
The reliability of spread in relation to the skill or error of a forecast is key to moving forward.
In order for spread to have a value to forecasters, it must provide a snapshot of the likely magnitude and/or location of forecast error. Otherwise, what use is it?
Previous studies (Barker 1991, Whitaker & Loughe 1998, Hopson Scherrer et al. 2004) utilized either “perfect” ensemble models (no IC errors) or did not approach the problem from a purely operational standpoint. The results are mixed.
Other studies have been done using 500mb heights (Toth et al. 2001) based on R-Climate.

7. The Questions
Is there a relationship between spread and mean absolute error?
Does the NAEFS offer an improvement over single model systems for spread/error relationship?
Does the subset of spread based on similar forecasts (M-Climate) provide context to the uncertainty of an event?
Is the tool useful – can it be applied to operations similar to ESAT?

8. The Data M-Climate Verification Data Ensemble Data
GEFS Reforecast 1 degree x 1 degree for DJF, 10 members
MSLP ensemble mean, spread
Verification Data
ERA-Interim 1 degree x 1 degree
Ensemble Data
DJF cyclone storm tracks (Korfe, personal correspondence) for storms with >=8 verifying ensemble hr 96
GEFS and CMC MSLP perturbed members (20 members each) DJF

9. Verification 2007-2014 Eastern US Cyclones
A verified cyclone must have passed through the region within 72W – 68W, 34N – 46N
Each forecast must contain >= 8 ensemble members with cyclones identified
96 hour forecast minimum low pressures are ID’d in GEFS data (1x1 grid)
312 identified 96 hour forecasts
Storm tracks for all storms identified by Nathan Korfe’s tracker data (blue, green, purple lines denote month) and 96 hour ID’d ensemble mean cyclone (blue points)

10. Cyclone Relative Error and Spread
Low pressure centers are identified within bounding box and centered within a 10 degree by 10 degree grid (~500km radius)
Spread, ensemble mean SLP, and mean absolute error are averaged and plotted for GEFS, CMC, and NAEFS.

11. Cyclone Relative Error and Spread Results
Error, MSLP (hPa)
GEFS (61)
CMC (54)
NAEFS (56)
Spread, MSLP (hPa)

12. Cyclone Relative Error – Spread (Difference) GEFSn=61
Difference between MAE and spread for each set of cyclones. Red hatching denotes statistical significance at 95% confidence (t-test)
CMC
n=54
NAEFS n=56

13. Cyclone Relative Error - Spread, Separated by Spread Magnitude
GEFS
Separating the GEFS and NAEFS by ranked mean spread of each event…
GEFS shows greater error near the center of the low for the bottom half of spread cases; lows have slightly less magnitude (top left).
NAEFS handles lower spread storms better (bottom left) but has higher spread than error along the east, center, and southern flank (bottom right).
n=30,31
NAEFS
n=28,28

14. Cyclone Mean Error GEFS NAEFS n=30,31 n=28,28
Separating the events in half by ranked spread magnitude…
Slight dipole between center of storm and north of storm for GEFS lower half
Dipole pattern for the GEFS upper half
Slight positive error in NAEFS lower half
Larger positive error north and slightly west of NAEFS upper half
GEFS
n=30,31
NAEFS
n=28,28

15. GEFS NAEFS Error, MSLP (hPa) Spread, MSLP (hPa)
valid December 28, 2012 x
Spread, MSLP (hPa)
More than one way to look at similarities between spread and error, ideally the error and spread correlate well with a linear pattern.
When looking at the MAE and Spread in a joint plot, correlations appear low in both GEFS and NAEFS. There’s more to do – there is value to looking at spread in the context of both position and error, but as always it can’t be used as the end-all product.

16. Standardized Spread Anomaly
Method
21-day M-Climate data loaded into code
Find standardized anomaly for all mean M-Climate points
Calculate standardized anomaly for mean forecast points
Find mean M-Climate values within -2 to 1 std devs of forecast at each gridpoint
Take spread values of valid M-Climate datapoints, forming new spread climatology
Calculate, output spread standardized anomaly

17. Standardized Spread Anomaly
Sample Case: February 10-11, 2010
Synopsis
A strong mid-latitude low brought blizzard conditions to the Mid-Atlantic and Northeast, where snowfall amounts topped 12 inches into much of Pennsylvania, New Jersey, and into the New York Metro region.
Results
The SSA indicates a 3-4 standard deviation anomaly (non-normal distribution) offshore Cape Cod during the latter stages of the event, coinciding w/ the hPa spread visible in overlapping locations.

18. Standardized Spread Anomaly
Sample Case: February 10-11, 2010
Synopsis
A strong mid-latitude low brought blizzard conditions to the Mid-Atlantic and Northeast, where snowfall amounts topped 12 inches into much of Pennsylvania, New Jersey, and into the New York Metro region.
Results
The SSA indicates a 3-4 standard deviation anomaly (non-normal distribution) offshore Cape Cod during the latter stages of the event, coinciding w/ the hPa spread visible in overlapping locations.

19. Standardized Spread Anomaly
Sample Case: February 10-11, 2010
Synopsis
A strong mid-latitude low brought blizzard conditions to the Mid-Atlantic and Northeast, where snowfall amounts topped 12 inches into much of Pennsylvania, New Jersey, and into the New York Metro region.
Results
The SSA indicates a 3-4 standard deviation anomaly (non-normal distribution) offshore Cape Cod during the latter stages of the event, coinciding w/ the hPa spread visible in overlapping locations.

20. Standardized Spread Anomaly
Sample Case: February 9-10, 2013
Synopsis
A significant cyclone caused snowfall totals of over 2 feet in Long Island and New England, with some locations in Long Island and Connecticut reaching inches.
Results
The storm on the GEFS at 96 hours was widely dispersed but spread only peaking at 8-10 hPa near the Gulf of Maine. Neither the location nor magnitude was apparent. The SSA indicates at least a larger anomaly wrt magnitude.

21. Standardized Spread Anomaly
Sample Case: February 9-10, 2013
Synopsis
A significant cyclone caused snowfall totals of over 2 feet in Long Island and New England, with some locations in Long Island and Connecticut reaching inches.
Results
The storm on the GEFS at 96 hours was widely dispersed but spread only peaking at 8-10 hPa near the Gulf of Maine. Neither the location nor magnitude was apparent. The SSA indicates at least a larger anomaly wrt magnitude.

22. Standardized Spread Anomaly
Sample Case: February 9-10, 2013
Synopsis
A significant cyclone caused snowfall totals of over 2 feet in Long Island and New England, with some locations in Long Island and Connecticut reaching inches.
Results
The storm on the GEFS at 96 hours was widely dispersed but spread only peaking at 8-10 hPa near the Gulf of Maine. Neither the location nor magnitude was apparent. The SSA indicates at least a larger anomaly wrt magnitude.

23. Conclusions Verification Spread Anomaly
GEFS has no statistically significant difference (95%) between mean absolute error and spread with n=61
NAEFS (and CMC) has statistically significant differences on the southern (and south, center, east) flank of cyclones with n=56 (and n=54)
Using mean absolute error as a first order verification metric is promising for cyclone relative SLP, other variables should be possible (QPF, Z, 850T, etc)
Spread Anomaly
The tool shows promise in terms of identifying magnitude of spread anomaly but is only as good as the model its looking at.
Standardized anomalies when looking at a non-normal distribution is less than optimal; percentiles “wash out” the events and don’t specify location as well. Return intervals are a potential alternative.

24. Future Work Website Variables Verification of Spread Anomaly
A website is currently being built for spread anomaly and verification for real time usage this winter
Verification via backtesting the previous year using plots similar to the ESAT verification (Spread vs MAE) will be tested for GEFS and NAEFS
Variables
Other variables will be used including QPF and 850mb temperature similar to ESAT
Verification of Spread Anomaly
The verification procedure looking at spread/error will also be used for SSA.
GEFS Reforecast will be tested for similarity w/ the operational GEFS
Utilizing probability matched mean to account for magnitude errors in error/spread calculations

25. Email: Taylor.Mandelbaum@stonybrook.edu
References
Barker, T. W. (1991). The Relationship between Spread and Forecast Error in Extended-range Forecasts. Journal of Climate.
Buizza, R. (1997). Potential Forecast Skill of Ensemble Prediction and Spread and Skill Distributions of the ECMWF Ensemble Prediction System.
Graham, R. A., & Grumm, R. H. (2010). Utilizing Normalized Anomalies to Assess Synoptic- Scale Weather Events in the Western United States. Weather and Forecasting, 25(2), 428–
Hamill, T. M., G.T. Bates, J. S. Whitaker, D. R. Murray, M. Fiorino, T. J. Galarneau, Y. Zhu, and W. Lapenta. (2013). NOAA’s Second­ Generation Global Medium­ Range Ensemble Forecast Dataset. Bull. Amer. Meteor. Soc., 94​, 1553­1565.
Hopson, T. M. (2001). Assessing the Ensemble Spread–Error Relationship.
Scherrer, S. C., Appenzeller, C., Eckert, P., & Cattani, D. (2004). Analysis of the Spread–Skill Relations Using the ECMWF Ensemble Prediction System over Europe.
Toth, Z., Zhu, Y., & Marchok, T. (2001). The Use of Ensembles to Identify Forecasts with Small and Large Uncertainty. Weather and Forecasting, 16(4), 463–
Whitaker, J. S., & Loughe, A. F. (1998). The Relationship between Ensemble Spread and Ensemble Mean Skill.
Questions?