Misleading estimates of forecast quality: quantifying skill with sequential forecasts Alex Jarman and Leonard A. Smith EGU 2012 Conference – Session NP rd April 2012 The support of Munich Re is gratefully acknowledged Centre for the Analysis of Time Series, London School of Economics www2.lse.ac.uk/CATS

Serial Dependence in Data and Statistical Inference Background Geophysical forecasting applications are numerous and have potentially significant beneficial economic impacts... Sources: NASA, interestingengineering.com, SOHO/EIT, Torsten Blackwood/AFP/Getty...hence, the potentially adverse impacts of misleading forecast skill estimation are also significant!

Serial Dependence in Data and Statistical Inference Background Serial dependence is common in geophysical data time series – even on long-term timescales; a well documented and well understood problem for statistical inference Source: Wilks, Statistical Methods for the Atmospheric Sciences, 2011 However, the effect on forecast skill has not been so well documented...

Serial Dependence in Data and Statistical Inference Statistical Inference of Forecast Skill Source: Wilks, Sampling Distributions of the Brier Score and Brier Skill Score under serial dependence, QJRMS, 2010 Serial correlation can lead to sample variance inflation and overconfidence in skill, BUT not always…

Overview of Research 3 Cases of Serial Correlation Effect/Non-effect on Forecast Skill Inference 1.Linear serial correlation in data 2.Linear serial correlation in data 3.Non-linear serial correlation in data misleading estimate of forecast skill non-misleading estimate of forecast skill misleading estimate of forecast skill

Linear Serial Correlation in Data Misleading Estimate of Skill Lorenz63 System: Univariate Time Series Exhibits chaotic behaviour but has strong degree of serial correlation: r 1 ~ 0.95

Lorenz63 System: Skill Score Statistics Skill score serial correlation… r 1 ~ 0.55 Linear Serial Correlation in Data Misleading Estimate of Skill

Lorenz63 System: Skill Score Statistics Estimates of forecast skill are unreliable… …but is dependent on location on the attractor and lead time

Overview of Research 3 Cases of Serial Correlation Effect/Non-effect on Forecast Skill Inference 1.Linear serial correlation in data 2.Linear serial correlation in data 3.Non-linear serial correlation in data misleading estimate of forecast skill (e.g. Lorenz63) non-misleading estimate of forecast skill misleading estimate of forecast skill

Linear Serial Correlation in Data Non-misleading Estimate of Skill AR(1) Process: Time Series Highly autocorrelated time series (φ=0.9)... … but is the serial correlation propagated into the skill score statistics??

Linear Serial Correlation in Data Non-misleading estimate of skill AR(1) Process: Skill Score Statistics No serial correlation in skill score: r 1 ~ 0 …so forecast skill estimates are reliable, even though the data is serially correlated

Linear Serial Correlation in DataNon-misleading Estimate of Skill AR(1) Process: Skill Score Statistics …but the skill estimate is unreliable with a perfect climatological forecast

Overview of Research 3 Cases of Serial Correlation Effect/Non-effect on Forecast Skill Inference 1.Linear serial correlation in data 2.Linear serial correlation in data 3.Non-linear serial correlation in data misleading estimate of forecast skill (e.g. Lorenz63) non-misleading estimate of forecast skill (e.g. AR(1) Process) misleading estimate of forecast skill

Non-linear Serial Correlation in Data Misleading Estimate of Skill Chaotic Map: Delay Plots CAUTION FOR DECISION-MAKERS: sample size may be important for accurate skill inference Truth (Data)Skill Score

Overview of Research 3 Cases of Serial Correlation Effect/Non-effect on Forecast Skill Inference 1.Linear serial correlation in data 2.Linear serial correlation in data 3.Non-linear serial correlation in data misleading estimate of forecast skill (e.g. Lorenz63) non-misleading estimate of forecast skill (e.g. AR(1) Process) misleading estimate of forecast skill (e.g. Chaotic Map)

Misleading Estimates of Forecast Quality: Quantifying Skill with Sequential Forecasts Summary 1.Serial dependence in verification data can lead to erroneous estimates of forecast skill (in some cases) Effect of serial correlation on forecast skill inference outlined by Wilks (2010), who showed: 1.the effect is increased for skilful forecasts and rarer events 2.sample size becomes important 2.Relationship between serial dependence in data and serial dependence in forecast skill is varied 3 identified cases of serial correlation in data, and effect or non-effect on statistical inference of skill e.g. a linearly correlated AR(1) process can result in un-correlated forecast skill The degree of serial correlation can be dependent on phase and forecast lead time in dynamical systems e.g. Lorenz63 attractor Wilks’s findings are not applicable in all cases e.g an unskilful forecast can lead to skill score sample variance inflation 3.Decision-makers should be aware of the complexities of the relationship, and recognise when the effects of serial dependence may lead to unrealistic expectations of forecast skill!

Additional Information Contact: Alex Jarman London School of Economics EGU Poster: “Distinguishing between Skill and Value in Hurricane Forecasting” Poster Programme: HS4.3 Location: Hall A, #A252