East Coast vs. West Coast: A Documentation of Model Forecast Failures for Eta, NAM, GFS, GEM, and ECMWF Garrett Wedam Lynn McMurdie, Cliff Mass

72-hour NAM Forecast: Ridging 1024 mb contour 72-hr NAM forecast with Obs verification Valid Dec 24,

72-hour Reality: Low Center 00-hr NAM analysis and obs verification Dec 24, mb contour

72-hour forecast errors –Eta/NAM: 24.1 mb –CMC-GEM: 14.4 mb –ECMWF: 10.0 mb –GFS: 3.2 mb 48-hour forecast errors –Eta/NAM: 13.7 mb –CMC-GEM: 9.1 mb –ECMWF: 7.6 mb –GFS: 1.5 mb 24-hour forecast errors –Eta/NAM: 4.7 mb –CMC-GEM: 2.4 mb –ECMWF: 2.7 mb –GFS: -0.2 mb GFS 72-hour forecast verifying Verifying buoy

72-hour Forecast: Low Center 72-hr NAM forecast with Obs verification Valid Dec 24, mb contour

72-hour Reality: Ridging 00-hr NAM Analysis and Obs verification Dec 24, mb contour

Method Directly compare observations to interpolated model forecasts Emphasize East and West Coasts –Buoys eliminate terrain effects Emphasize Sea Level Pressure comparisons –SLP is good indicator of model performance (i.e., storm strength and location can be defined and compared using SLP) –Insufficient off-coast upper-level observations

Method Useful Tools for Comparing Forecast Skill –Mean Absolute Error (MAE) –Mean Error, or Bias (ME) –Frequency of Errors which exceed an arbitrary “Large Error” criterion (different for 24, 48, 72-hr forecasts) –Timelines of said “Large Errors” to cross-reference failure dates between models –Error Histograms to compare distributions and sizes of largest errors

Matching Variance in Sea Level Pressure:

Eta to WRF-NAM operational switchover > 7 mb > 5 mb > 3 mb West Coast: Number of Large Errors by model and month

East Coast: Number of Large Errors by model and month CMC - GEM major model update. Included: increase in vertical and horizontal resolution, new physics scheme, decreased time step, data assimilation changes

Number of Large Errors: West Coast *minus* East Coast

Mean Absolute Error: West Coast *minus* East Coast For reference, typical MAE values: GFS West Coast average for winter 2006/07: 24-hr: 1mb; 48-hr: 1.4 mb; 72-hr: 2.0 mb

Comparing models: East Coast Mean Absolute Error *Standardized* to ECMWF CMC – GEM update

Comparing models: West Coast Mean Absolute Error *Standardized* to ECMWF

Results Comparing models: ECMWF generally outperforms and NAM underperforms others. There are indications that a CMC-GEM model update significantly improved some forecasts, and that GFS has not performed as well this season as the past season. More “forecast bust” events occur on the West than East Coast for 24, 48, and 72 hour forecasts The East Coast is better forecast in terms of MAE in sea level pressure

Dates of Large Forecast Errors (by model) Errors greater than 5 mb On the Horizon

Forecast Failures can be a result of initialization errors, insufficient “realism” of the model, and other model inadequacies. Knowing the most common causes of failure for high-impact forecast failures can help determine policy: do we invest more in improving models or in a new observation buoy? One cause of failure is High Model Sensitivity…

On the Horizon Courtesy of Ryan Torn

On the Horizon

Thirty-four of 420 forecasts showed the highest sensitivity Nineteen of the 34 “sensitive” cases (55%) resulted in multiple models meeting “large error criterion” Compare to: 10-20% of all forecasts result in multiple models meeting “large error criterion”  motivates a more formal investigation!

On the Horizon Picture from fijaciones.org Are Two Winters Representative? Currently adding data from 2004/05 and remainder of 2006/07 to further examine year-to-year variability in forecast skill What About The Rest of the Country? And Beyond? Will examine open-ocean buoys, and land sites; over land will use 850 mb observations instead of SLP What Are the Features of “Busts”: Forecast failures with high impacts on people still occur. What are their features? And why do some models fail when others don’t?

East Coast vs. West Coast: A Documentation of Model Forecast Failures for Eta, NAM, GFS, GEM, and ECMWF Garrett Wedam Lynn McMurdie, Cliff Mass Thank You!