National Hurricane Center 2009 Forecast Verification James L. Franklin Branch Chief, Hurricane Specialist Unit National Hurricane Center 2009 NOAA Hurricane Conference James L. Franklin Branch Chief, Hurricane Specialist Unit National Hurricane Center 2009 NOAA Hurricane Conference 1

Verification Rules  Verification rules unchanged for Results presented here in both basins are preliminary.  System must be a tropical or subtropical cyclone at both forecast initial time and verification time. All verifications include depression stage except for GPRA track goal verification.  Special advisories ignored (original advisory is verified.  Skill baselines are recomputed after the season from operational compute data. Decay- SHIFOR5 is the intensity skill benchmark.

2009 Atlantic Verification Values in green exceed all- time records. 48 h track error for TS and H only (GPRA goal) was 69.9 n mi, well below previous record of Sample is very small (last year 346 forecasts, with 149 verifying at 5 days). Five-day sample is smallest ever. VT NT TRACK INT (h) (n mi) (kt) ============================ Four- and five day track error was almost exclusively along-track (slow).

Atlantic Track Errors vs. 5-Year Mean Official forecast was better than the 5-year mean, even though the season’s storms were “harder” than normal. OFCL errors were mostly below the 5-yr means, even though CLIPER5 errors were above their 5-yr means.

Atlantic Track Error Trends Errors have been cut in half over the past 15 years was best year ever. Smaller samples give more erratic trends at days 4-5.

Atlantic Track Skill Trends 2009 set skill records at h. Sharp increase over past two years due to greater availability of the ECMWF?

2009 Track Guidance Official forecast performance was very close to the consensus models. Good year for FSSE. First year of availability for CMCI. A good start indeed. Best dynamical models were ECMWF and GFS. UKMET and NOGAPS appear to be serious drags on the consensus. Will need to reevaluate TCON/TVCN members for BAMD performed poorly (strong shear).

Atlantic Intensity Errors vs. 5-Year Mean OFCL errors in 2009 were mostly at or above the 5-yr means, but the 2009 Decay-SHIFOR errors were above their 5-yr means, indicating storms with unusual behaviors.

Atlantic Intensity Error Trends No progress with intensity.

Atlantic Intensity Skill Trends Little net change in skill over the past several years.

2009 Intensity Guidance Best model at every time period was statistical. Very good year for LGEM, which handles changes in the environment better than SHIPS.

2009 East Pacific Verification VT NT TRACK INT (h) (n mi) (kt) ============================ Values in green tied or exceeded all-time lows.

EPAC Track Error Trends Since 1990, track errors have decreased by 30%-50%.

EPAC Track Skill Trends Although errors were higher in 2009, skill was mixed.

EPAC Intensity Error Trends Errors look pretty flat.

EPAC Intensity Skill Trends Skill also seems flat in this decade.

Genesis Forecast Verification Relatively steady increase of verifying percentage with forecast percentage indicates good reliability. Forecasters can distinguish likelihood of genesis in 10% increments. NHC will provide public genesis forecasts to the nearest 10% in Fairly systematic low bias in the east Pacific.

Experimental Rapid Intensification Forecast Verification In house probabilistic forecast of the chance of 30 kt or more of intensification over the next 24 hours (climatology = 5%). EPAC curve (where low probs have high bias while high probs have a low bias) reflects underconfidence on the part of the forecaster (i.e., we know more than we think we do).

Summary  Very small Atlantic sample. OFCL track errors set records for accuracy from 24 to 72 h. Errors continue their downward trends, skill was also up.  Good years for FSSE, GFSI, EMXI, CMCI (first year of availability) in the Atlantic.  Not much new with intensity. Best model was statistical (LGEM).  Genesis forecasts show good reliability. NHC will provide these forecasts in 10% increments in 2010.

Forecast Error by Parameter Radii errors look good (especially 64 kt radii), but this is misleading...

Forecast Skill by Parameter Track forecast skill is very high (and higher at 5 days than it is at 24 h). Intensity skill is low but fairly steady with lead time. Wind radii skill falls off significantly with lead time (to near zero by 72 h). 30% skill of the 36 h hurricane radii forecasts represents accuracy improvement of 3-4 n mi over DRCL. This is meaningless in the context of the ~75 n mi mean track error.

Forecast Change Variance Explained at Longest Lead Much more information in the intensity forecasts. Intensity forecasts have a very large “dynamic range”, while the vast majority of hurricane radii forecasts are within 20 n mi of initial radii (such that the forecast radii change is much smaller than typical track error.)

Forecast Parameter Scale Analysis Tropical Cyclone Forecast Parameter Scale Analysis (Based on Atlantic Basin statistics, ) Parameter Analysis Uncertainty Verification Uncertainty (at 36 h) Forecast Skill at longest lead Variance Explained at longest lead Track <1% 30° ± 0.3° 20% 75 nm ± 15 nm 45%Very high Intensity 10% 100 kt ± 10 kt ~50% 13 kt ± 7 kt* 10%60% Wind Radii 25-40% 100 nm ± 25 nm 25 nm ± 10 nm ~100% 30 nm ± 25 nm 10 nm ± 10 nm 0% (34kt/72h) 30% (64kt/36h) 17-34% (34kt) 8-12% (64 kt) * Assumes 10% error applied to a 70 kt “typical” TC.

Upper bound estimate of the impact of radii forecast uncertainty on WS probabilities (Francis 2004). Control (Radii-CLIPER perturbations)Radii assumed perfect Conclusion: Knowing the error characteristics of the radii forecasts is of limited practical use (because the probabilities are dominated by the track uncertainty).

Considerations for Forecast Extension  Track (location) is very easy to determine, easy to verify, forecasts have high skill and high information content. Forecasts have high utility. Excellent candidate to extend lead time.  Intensity is readily determined, and can be meaningfully verified most of the time. Forecasts have little skill (relative to climatology and persistence), but have high information content. Forecasts have high utility. Decent candidate for extension.  Wind radii are difficult to determine, virtually impossible to verify quantitatively; 34 kt radii have no skill at extended leads and low information content. Forecasts have low utility due to the much larger track errors. Lousy candidate for extension.