1. Page 1© Crown copyright 2005 Using metrics to assess ocean and sea ice simulations Helene Banks, Cath Senior, Jonathan Gregory Alison McLaren, Michael Vellinga + input from many Hadley Centre colleagues WGOMD August 2007

2. Page 2© Crown copyright 2005 Outline  History of model assessment at the Hadley Centre  representative of other centres?  Proposed way forward  consistent with other community initiatives?

3. Page 3© Crown copyright 2005 HadCM3: ‘Ad-hoc’ analysis  Main focus on top of the atmosphere balance, SST drifts and heat transports  Analysis was ad-hoc  Model is still being used for lots of applications SST drifts: years 81-120 Gordon et al., 2000

4. Page 4© Crown copyright 2005 HadGEM1: More formal acceptance criteria  Acceptance criteria introduced  Scientific Credibility  Eg, Conservation of mass, energy and water  Scientific benchmarks relevant to ocean:  Net TOA flux in balance in control run to better than 0.5 W/m 2  Surface air temperature and SST drifts comparable to HadCM3  Global mean SST error less than 0.5 K, local SST errors less than 2 K except in regions of sharp gradients, and overall SST and SSS errors superior to HadCM3.  Oceanic circulation stable, with accurate THC strength (NATHC = 20 +/- 5 Sv).  Oceanic poleward heat transport within 20% of observed estimates.  Wintertime sea ice extents within 20% of observed estimates in both hemispheres.  Oceanic water mass (T and S) drifts better than HadCM3.  Introduction of the Climate Prediction Index (CPI)

5. Page 5© Crown copyright 2005 CPI used for HadGEM1 (Johns et al., 2006)

6. Page 6© Crown copyright 2005 HadGEM2: Beginning to use metrics more formally as part of model development  Focussed on improving ENSO  Acceptance criteria:  ENSO criteria  Measures of the tropical mean basic atmosphere and ocean state and ENSO performance to place the model within the pack of leading (IPCC AR4) models in most if not all respects (judged relative to observations)  ENSO simulation judged to be competitive with HadCM3 and GloSea including its skill in idealised predictability experiments (as judged by the seasonal forecasters)  CPI criteria  No CPI elements judged to be significantly worse than HadGEM1 (i.e. key scientific improvements identified in HadGEM1 over HadCM3 and captured within the CPI should be preserved)  Overall CPI judged to be as good as or better than HadGEM1  Other scientific criteria  TOA radiative imbalance, energy/mass/freshwater budgets, and magnitude of resultant coupled drifts in the control run judged to be no worse than HadGEM1  Any other key scientific improvements identifiable in HadGEM1 over HadCM3 judged to be preserved [e.g. MJO]  Model to be judged at least equally suitable for general climate variability studies as HadCM3  Substantially increased rainfall over the Indian subcontinent in the summer monsoon compared to HadGEM1 (in the mean) is desirable

7. Page 7© Crown copyright 2005 ‘Guilyardi’ plot Sarah Ineson

8. Page 8© Crown copyright 2005 HadGEM3: Requirements  HadGEM3 is being developed now  Take a top down approach to assessment  Requirements for the model defined  These requirements are translated into assessment areas: 1. Conservation 2. Global circulation Atmosphere Radiative balance Hydrological cycle Ocean Sea ice Land surface Stratosphere 3. Regional variability Regional predictions Monsoon ENSO MJO NAO Extremes 4. Seasonal to decadal

9. Page 9© Crown copyright 2005 Assessment criteria  In each assessment area, we are defining metrics to assess the simulations and underlying processes (i.e., is it the right answer for the right reasons)  As far as possible, use ‘community’ metrics  The metrics are generally not ‘new’  This approach provides a framework for objectively defining the assessment  Move towards a common presentation-summarise each area in (for example) a bar chart to allow a full assessment of the model

10. Page 10© Crown copyright 2005 What is good enough?  Assess against best observations (or leading models in absence of observations)  No pass/fail level  ‘Comfort zone’ defined  Fuzzy level based on uncertainties in observations/pack of leading models Comfort zone

11. Page 11© Crown copyright 2005 HadGEM1 sea ice spin up

12. Page 12© Crown copyright 2005 Proposed sea ice variables to assess  March/Sept NH/SH ice extent & area  Month of maximum/minimum NH/SH ice extent  Seasonal amplitude of NH/SH ice extent  RMS difference of winter ice concentration  Ice extent in selected regions  RMS of central Arctic ice thickness  Gradient of Arctic ice thickness  Maximum ice thickness in NH/SH  RMS ice speed in NH/SH winter  Transport across selected Straits  Annual mean northwards ice transport in SH Summary in a bar chart cf McLaren et al., 2006

13. Page 13© Crown copyright 2005 Proposed ocean variables to assess  Temperature (SST, X-secns/collocations)  Salinity (SSS, X-secns/collocations)  Mixed layer  Currents (EUC, ACC, Arctic transports)  Upwelling (Equatorial, Basin upwelling)  Ocean transports (heat, freshwater)  Water masses (T-S, formation)  Budgets (conservation, surface fluxes)  MOC (overflows, transports, etc)  SSH (mean, anomaly)  Mesoscale features (eddy ke, TIWs, Gulf Stream separation, Agulhas)

14. Page 14© Crown copyright 2005 Issues with ocean assessment  Drift is an issue especially for assessing T-S properties of water masses  Runs of different lengths  Can we make an ‘educated guess’ on whether there will be a long-term impact of drifts  The use of neutral densities in observational estimates  Not easily applied to models  How to combine a large number of criteria into something ‘digestable’-possible use of skill scores (under discussion-Vellinga, Williams, Sexton)

15. Page 15© Crown copyright 2005 Other community efforts  WGNE workshop on metrics-San Francisco early 2007  PCMDI effort???  GODAE-Eric  GSOP-Detlef

16. Page 16© Crown copyright 2005 Ideas that might be useful for WGOMD/ SOPHOCLES  Who are the users and what do they need the model to get right?  Eg, Carbon uptake in Southern Ocean  What are the appropriate metrics?  Metrics should assess processes as well as answer  What are the levels of acceptability?  How to present assessment in an objective manner?