1. Bob Livezey and Marina Timofeyeva
Diagnosis of Skill Variability of CPC Long-Lead Seasonal Forecasts: Implications for Production and Use
Bob Livezey and Marina Timofeyeva
NOAA/NWS/OCWWS/Climate Services Division
Climate Diagnostics and Prediction Workshop
October 21, 2004
Madison WI

2. Outline Introduction Skill Stratifications Results
Conclusions and Lessons

3. Introduction
Users should only care about the performance of forecasts that can potentially benefit their decision process
Livezey (1990): There are non-random subsets of seasonal forecasts that were skillful enough to be useful

4. Seasonal Temperature Forecast Skill 1960s to 80s
All Seasons 8.3
Winter
Spring
Summer
Fall
Seasonal Temperature Forecast Skill 1960s to 80s
All Seasons 8.3
Winter
Spring
Summer
Fall

5. Introduction (Cont.) This talk will make the point :
That was made by Livezey (1990)
That there are many non-random subsets of forecasts that do not have useful skill
That it is critical for this information to be shared with potential users
That skill analyses with different stratifications are highly informative
That seasonal forecast production should be dominantly objective
That long-lead seasonal forecasts of non-trend signal should only be done on appropriate opportunities, not routinely

6. Displays and Stratifications
CPC Seasonal Forecasts
For 3-equally probable temperature and precipitation classes at 102 Climate Divisions
Made every month from 1995 to present for 0.5-, 1.5-, …, 12.5 month leads
Skill Measure: Modified Heidke Skill Score of Categorized Forecasts
1/3 EC’s scored as ‘hits’
Displays and Stratifications
Summed over all forecasts for each lead for three overlapping seasons at a time (DJF to FMA, FMA to AMJ, etc.)
Stratified further by strong ENSO years vs. other years
Mapped for appropriate combinations of leads

7. Temperature

8. Temperature

9. Temperature

10. Results Seasonal Temperature Die-Aways:
Moderate to high national-scale skill confined to Fall/Winter strong ENSO years at short to medium leads
Otherwise skill is dominantly modest and level with lead (derived from biased climatologies, ie long-term trend)
Worst forecasts are for
Fall/Winter at short to medium leads for non strong-ENSO years: Forecasters subtracting value
Summer/Fall at medium to long leads for strong ENSOs: No remedy except science advances
Short-lead forecasts are better now than for the 1960s-80s, ~14 vs ~8 overall, ~20 vs ~13 for the winter

11. Temperature

12. Temperature

13. Temperature

14. Temperature

15. Results Seasonal Temperature Maps:
Without a strong ENSO, justification for OND-DJF forecasts is questionable
For certain regions/seasons/situations skill is unambiguously useful even for undisciplined, occasional users

16. Precipitation

17. Precipitation

18. Results Seasonal Precipitation Die-Aways:
Barely useable national-scale skill entirely confined to Fall/Winter strong ENSO years in short to medium leads
Otherwise skill is statistically indistinguishable from zero
Short-lead forecasts overall seem to be no better now than for the 1960s-80s (but at least are now made two-weeks earlier)

19. Precipitation

20. Precipitation

21. Precipitation

22. Precipitation

23. Results Seasonal Precipitation Maps:
Justification for long-lead forecasts is questionable for at least three of six seasonal groups examined
For certain regions/seasons/situations skill is unambiguously useful even for undisciplined, occasional users

24. Conclusions and Recommendations
There are non-random subsets of seasonal forecasts that are skillful enough even for undisciplined, occasional users
These skills exclusively reflect ENSO and trend signals
There are many non-random subsets of forecasts that do not have useful skill
We must share this information with potential users
Skill analyses with different stratifications are highly informative
Seasonal forecast production should dominantly rely on objective exploitation of ENSO and trend with subjective modification only after rigorous criteria are met
Routine production of certain long-lead forecasts should be terminated (can be accommodated by transition to separate trend and high frequency forecasts with latter issued on forecast of opportunity basis only)