1. Verification of IRI Forecasts 1997 - 2008 Tony Barnston and Shuhua Li

2. IRI’s Forecast System IRI has been using a 2-tiered prediction system to probabilistically predict global temperature and precipitation with respect to terciles of the historical climatological distribution. Within the 2-tiered system IRI uses 4 SST prediction scenarios, and combines the ensemble predictions from 7 AGCMs. Predictions from the 7 AGCMs are merged into a single one using multi-model ensembling.

3. 30 12 30 24 12 24 10 24 10 FORECAST SST TROP. PACIFIC: THREE (multi-models, from 2 dyn and 1 statistical model) TROP. ATL, INDIAN Similar to trop. Pacific EXTRATROPICAL (damped persistence) GLOBAL ATMOSPHERIC MODELS ECPC(Scripps) ECHAM4.5(MPI) CCM3.6(NCAR) NCEP(MRF9) NSIPP(NASA) COLA2 GFDL Forecast SST Ensembles 3/6 Mo. lead Persisted SST Ensembles 3 Mo. lead IRI DYNAMICAL CLIMATE FORECAST SYSTEM POST PROCESSING MULTIMODEL ENSEMBLING PERSISTED GLOBAL SST ANOMALY 2-tiered OCEAN ATMOSPHERE 30 model weighting

4. FORECAST SST TROP. PACIFIC: THREE scenarios: 1) Average of predictions of CFS (NCEP), LDEO5 (Lamont) and Constructed Analogue (statistical; NCEP/CPC) 2) same as 1), plus an uncertainty pattern 3) same as 1), minus an uncertainty pattern TROP. ATL, and INDIAN oceans Same as Pacific, without LDEO5 and with statistical (CCA) forecast for Indian Oc. EXTRATROPICAL: damped persistence MULTIPLE GLOBAL ATMOSPHERIC MODELS ECPC(Scripps) ECHAM4.5(MPI) CCM3.6(NCAR) NCEP(MRF9) NSIPP(NASA) COLA2 GFDL AM2p12b IRI DYNAMICAL CLIMATE FORECAST SYSTEM PERSISTED GLOBAL SST ANOMALY 2-tiered OCEAN ATMOSPHERE

5. 25N CFS CFS CFS CFS LDEO CA CCA LDEO CA CA CA 25S Contributors to SST Predictions for Forcing IRI AGCMs damped persistence damped persistence

6. YEAR AGCM 9899000102030405060708 ECHAM 3.6 4.5 CCM3 NCEP MRF-9 NASA COLA Scripps GFDL History of AGCMs used in IRI Seasonal Climate Forecasts

7. Merging of Forecasts of 7 AGCMs The merging of 7 predictions into a single one uses two multi-model ensemble systems: Bayesian and canonical variate. These give somewhat differing solutions, and are presently given equal weight.

8. NOV | Dec-Jan-Feb 15 Jan-Feb-Mar Feb-Mar-Apr Mar-Apr-May IRI’s monthly issued probability forecasts of seasonal global precipitation and temperature Forecasts are issued at four lead times. For example: Forecast models are run 7 months into future. Observed data are available through the end of the previous month (end of October in example above). Probabilities are given for the three tercile-based categories of the climatological distribution.

10. Ranked Probability Skill Score (RPSS) “probability” for the obs is 1 for the observed category, 0 for the others RPSS = 1 - (RPS fcst / RPS clim ) RPS clim is RPS of forecasts of climatology (33%,33%,33%)

21. Earlier Real-time Forecast Skill from Goddard et al. 2003, BAMS, p1773

22. Some other verification measures: Likelihood score ROC area generalized to all categories Temporal correlation (standard) Temporal correlation (sample mean not removed)

23. The likelihood score The likelihood score is the nth root of the product of the probabilities given for the event that was later observed. For example, using terciles, suppose 5 forecasts were given as follows, and the category in red was observed: 45 35 20 The likelihood score 33 33 33 disregards what prob- 40 33 27 abilities were forecast 15 30 55 for categories that did 20 40 40 not occur. The likelihood score for this example would then be = 0.40= This score could then be scaled such that 0.333 would be 0%, and 1 would be 100%. A score of 0.40 would translate linearly to (0.40 - 0.333) / (1.00 - 0.333) = 10.0%. A nonlinear translation between 0.333 and 1 might also be selected.

24. GROC = -------------------------------------------------- 0 if P lj (k) < P ki (k) Where 0.5 if P lj (k) = P ki (k) (tie) 1 if P lj (k) > P ki (k) Generalized ROC area (integrated over categories): Each pair of forecasts having differing observations is examined to see if the forecast for the higher obs result was higher than that for the lower observational result. Number of unique forecast pairs having differing obs results

25. Comparison of verification measures

40. Earlier Real-time Forecast Skill from Goddard et al. 2003, BAMS, p1765

41. Comparison of verification measures

46. | | | | | | | | | | | | 1998 | 1999 | 2000 | 2001 | 2002 | 2003 | 2004 | 2005 | 2006 | 2007 | 2008 OBS Lead 1 Lead 4 | 1998 | 1999 | 2000 | 2001 | 2002 | 2003 | 2004 | 2005 | 2006 | 2007 | 2008 | | | | | | | | | | | 1998-08 2002-08 Lead 1.92.91 Lead 4.80.78 Correlations   How did we do in predicting ENSO fluctuations?

53. absolute value r =.56 r =.25 r=.35 r=.43 | | El Nino neutral or LaNina

54. 3-month lead r =.74 r=.66 r =.63 r=.65 3 months earlier La Nina El Nino

55. Reliability: Degree and nature of relationship between forecast probabilities and subsequent relative frequency of occurrence of corresponding obser- vations. Large sample of cases for any probability level is needed. 20 cases of 60% for “above”  what % of time did “above” occur?

56. Frequency of issuance OND 1997 to MJJ 2008: first lead time

57. Frequency of issuance JFM 1998 to MJJ 2008: first lead time

58. Conclusions Real-time predictive skill of 11 years of IRI seasonal climate forecasts ranges from nonexistent to moderate, with “modest” being the most common skill level. Predictions at such levels are worthwhile and economically beneficial over time. We do not know what proportion of potential predictability are realized in IRI’s skills. Precipitation is predicted with generally lower skill than temperature, due to its smaller scale, and skill is more seasonally/regionally specific. ENSO is by far the largest source of seasonal climate prediction skill for precipitation in most regions. For temperature, climate change generally equals, or overshadows, ENSO as a cause of anomalies. But this would change if we defined anomalies with respect to a more current (shorter) recent period, or or even conservatively incorporated forecast data.