© Crown copyright Met Office Standard Verification System for Long-range Forecasts (SVSLRF) Richard Graham, Met Office Hadley Centre. With acknowledgements to all GPCs, LC- SVSLRF and members of the WMO Expert Team on Extended and Long-range Forecasting (ET-ELRF) WMO/THORPEX/WCRP sub-seasonal to seasonal implementation planning meeting, WMO HQ, Geneva, 2-3 December 2011

Lead Centre for Standard Verification of Long-range forecasts (LC-SVSLRF) repository of global SVSLRF scores for GPC forecasts Core of SVS: ROC maps and diagrams, reliability diagrams, MSSS documentation on verification scores (Attachment II-8 of the WMO manual on the GDPFS) observation datasets for verification, and code for calculating the SVSLRF scores Jointly operated by Bureau of Met. BoM) & Met. Service of Canada (MSC) David Jones, Bertrand Denis

SVSLRF: skill measures Probability forecasts (for tercile categories): Relative Operating Characteristics (ROC): diagrams and scores (area under ROC curve) Reliability diagrams and sharpness diagrams Deterministic forecasts (ensemble mean): Mean Square Skill Score (MSSS) and decomposition terms (correlation, variance, bias) – climatology as reference Variables: T2m, precipitation, SST, Nino3.4 index

Level 1: ‘bulk’ scores and diagrams for regions/indices (for admin.) ROC curves and scores; reliability diagrams and freq histograms; MSSS Variables/indices: T2m, precip, Nino3.4 Regions: Tropics (20N to 20S), N. Extratropics (20-90N); S. Extratropics (20-90S) Seasons: MAM, JJA, SON, DJF (Nino3.4: calendar months) Lead-times: all available (but not > 4 months for T2m and precip; not > 6 months for Nino3.4)

Level 2: scores at grid points for generating skill maps (for users) ROC scores; MSSS (and decomposition) Variables/indices: T2m, precip, SST Regions: global, each defined grid point Seasons: 12 rolling 3-month seasons Lead-times: all available (but not > 4 months) Stratification: Where sufficient hindcast years, stratify into seasons with El Niño active and seasons with La Niña active

Lead Centre SVSLRF ROC curves ROC score maps Reliability/sharpness diagrams Courtesy: David Jones (BoM)

Sample verification: ROC scores for SON precipitation, 1-month lead Toulouse WashingtonTokyoMelbourne above below 0.6

Some issues Each GPC calculates own scores and submits to LC-SVSLRF. Centralised verification would help standardisation (e.g. in category definition, verification datasets used) and speed processing; Good practice = all forecast products have corresponding verification products – big task, we are still catching up to a degree; Only GPC minimum variable set are required to be verified (T2m, precip, SST, Nino3.4) – but forecasts for other variables (e.g. Z500) are available No verification yet of individual month forecasts (available on LC-LRFMME website) No verification yet of the LC-LRFMME multi-model combinations

Thank you! Any questions?