A Superior Alternative to the Modified Heidke Skill Score for Verification of Categorical Versions of CPC Outlooks Bob Livezey Climate Services Division/OCWWS/NWS 28 th Climate Diagnostics and Prediction Workshop Reno, October 20, 2003

Outline 1.Introduction 2. Contingency Tables & Notation 3.Common Scores & Score Attributes 4.Gandin & Murphy Equitable Scores 5.Gerrity Scores 6.Recommendations

Contingency Tables and Notation p ij: Joint relative frequencies p i: : Observed relative frequencies q i : Forecast relative frequencies p i * : Prescribed relative observed frequencies (climatology) Table 4.2. Contingency table giving pij in percent (total sample size n=788) for U.S. mean temperature forecasts for June through August

Some Simple Categorical Skill Scores: Heidke, CPC Heidke, and Pierce

Score InequitableHSSPSSGS FMA JJA Skill scores for U.S. mean temperature forecasts in three categories for February through April and June through August

Desirable Attributes of Scores Equitable; –Equitable without dependence on the forecast distribution; Rewards for correct forecasts inversely proportional to their event frequencies; Penalties for incorrect forecasts directly proportional to their event frequencies; –Penalties for incorrect ordinal forecasts with equal event frequencies proportional to degree of miss; Consistent with an underlying linear association and insensitive to type or number of categories used. Note: 2 &3 imply that all information in the contingency table is taken into account.

Figure 4.1

Gandin and Murphy Equitable Scores

Gerrity Scores

Figure 4.4

Recommendations CPC use the Gerrity score for ordinal multi- categorical verification –Forecast history is digitized so skill history can be constructed –Clueless audience remains clueless –Score now equitably accounts for all facets of forecast performance CPC use actual frequencies CPC routinely determine confidence limits of scores Reference Jolliffe and Stephenson (2003; Wiley)