Earth System Model Evaluation Tool (ESMValTool) Axel Lauer1, Yoko Tsushima2, Mattia Righi1, Veronika Eyring1, Alexander Löw3, Ulrika Willén4, and Mark Ringer2 1 Deutsches Zentrum für Luft- und Raumfahrt e.V. (DLR), Oberpfaffenhofen, Germany 2 Met Office, Exeter, United Kingdom 3 Department of Geography, University of Munich (LMU), Germany 4 Swedish Meteorological and Hydrological Institute (SMHI), Norrköping, Sweden 27 May 2015

Motivation – Development of an Earth System Model Evaluation Tool Facilitate the evaluation of complex Earth System Models, e.g. quick look at standard diagnostic plots & output diagnostic variables easy comparison of new simulations (e.g. sensitivity runs or runs with new model versions) with existing runs and with observations Raise the standard for model evaluation include additional diagnostics of ongoing evaluation activities so that we do not have to start from scratch each time implement more observations, account for observational uncertainties Quick assessment of where we stand with a new set of model simulations via standard namelists that reproduce specific papers, reports, etc. Traceability and reproducibility Easy expendability synergies with ongoing projects to expand the tool (e.g. NCAR CVDP) useful for model groups & those analyzing models useful for model development

Technical Improvements Schematic Overview of the ESMValTool Structure Introduction to the ClimValDiaglTool 3

Development of an Earth System Model Evaluation Tool Within EMBRACE: DLR, SMHI & EMBRACE partners in collaboration with NCAR, PCMDI, GFDL Open Source: Python Script that calls NCL (NCAR Command Language) and other languages (e.g. R, Python) Input: CF compliant netCDF model output (CMIP standards) Observations: can be easily added Extensible: easy to (a) read models (b) process output [diagnostic] with observations and (c) use standard plot types (e.g. lat-lon map) Current developments include for example Essential Climate Variables, e.g. Sea-Ice Temperatures & water vapor Radiation CO2 Ozone Tropical variability (incl. monsoon, ENSO, MJO) Southern ocean Continental dry biases and soil-hydrology-climate interactions (e.g. standardized precipitation index) Atmospheric CO2 and NO2 budget Aerosol, clouds Stratospheric ozone Goal: Standard namelists to reproduce certain reports or papers (e.g., IPCC AR5 Chapter 9, Massonnet et al., 2012; Anav et al., 2012; Wenzel et al. 2014; Eyring et al., 2013) Release as open-source software planned for later this year.

Running along-side the ESGF ESMValTool: A community diagnostic and performance metrics tool for routine benchmarking and process evaluation of Earth System Models in CMIP Similar to Figure 9.7 of AR5 Monsoon Precipitation Intensity and Domain CMIP5 MMM Running along-side the ESGF CMIP5 MMM - OBS Similar to Figure 9.7 of AR5 Link to projections Similar to Figure 9.5 of AR5 Similar to Figure 9.24 of AR5 Similar to Figure 9.24 of AR5 Introduction to the ClimValDiaglTool 5

Considering Observational Uncertainty in the ESMValTool Observational uncertainty needs to be considered in the assessment of model performance and should be an integral part of the evaluation tools The ESMValTool is quite flexible and can consider observational uncertainty in many ways, e.g.: using more than one observational dataset to evaluate the models showing the differences between the observational datasets in addition to the difference between the model and the reference dataset easily usage of an observed uncertainty ensemble spanning the observed uncertainty space, consistent with statistics Example: HadCRUT4 surface temperatures, 5 (out of 100) realizations

WCRP Grand Challenges: (1) Clouds, circulation and climate sensitivity, (2) Changes in cryosphere, (3) Climate extremes, (4) Regional climate information, (5) Regional sea-level rise, and (6) Water availability, plus an additional theme on “biospheric forcings and feedbacks” Goal ESMValTool as one of the CMIP documentation functions to routinely assess the performance of CMIP DECK and CMIP6 simulations running alongside the ESGF Meehl et al., EOS, 2014

Work as part of ESA CCI CMUG Model benchmarking initiatives have become increasingly important to evaluate the quality of coupled Earth System Models (ESMs) and to support the model development process. However, ESA data is currently not used in the context of routine model evaluation. Model benchmarking is also important for the Climate Research Groups (CRGs) within the CCI and various CMUG activities. CMUG will therefore establish a standardized model benchmarking approach for the CCI in close collaboration with the CMIP and WGNE/WGCM Climate Model Metrics Panel, and will contribute to the development of a community-wide Earth System Model Evaluation Tool (ESMValTool) that is currently developed by different partners in different projects – coordinated by DLR. The goal is to work towards a standardized community based benchmarking toolkit that includes ESA CCI data and that could be used operationally for CMIP6 analysis in the forthcoming years. It is expected that the preparation of the individual ESA CCI datasets in CMIP compliant format for obs4MIPs as well as the corresponding technical notes for obs4MIPs are created and submitted by the individual ESA CCI teams.

WP 5.1 – Subtasks Contribute to the community-wide development of an Earth System Model Evaluation Tool (ESMValTool) by developing code and integrating diagnostics and performance metrics into the tool Ensure integration of ESA CCI datasets as well as other ESA datasets (in particular GlobAlbedo or other Glob-products) in the ESMValTool for routine model evaluation and benchmarking. Provide comparison to other observations that are routinely used in model evaluation, in particular those that are contributed to obs4MIPs (http://obs4mips.llnl.gov/). Provide recommendations of standardized benchmarking to the CMIP and WGNE/WGCM climate model metrics panels. Visualize the results through a website to enhance visibility of the ESA CCI data. The ESMValTool will be extended with capabilities to visualize the resulting plots on a website. Support the CCI Climate Research Groups in the use of the ESMValTool by providing a user guide with the goal that the CCI CRGs can use the tool in their own evaluation work.

Goal: Performance Metrics calculated with ESA CCI Data Relative error measures of CMIP5 model performance, based on the global seasonal-cycle climatology (1980–2005) computed from the historical CMIP5 experiments. Figure 9.8 of IPCC AR5 (Flato et al., 2013). A similar figure will be produced for selected ESA CCI ECVs using ESA CCI as the reference data set and if available an alternate observational data set for comparison.

Baseline Proposal and Batch Options 1 and 2 Atmospheric ECVs (aerosols) [DLR] Terrestrial ECVs (soil moisture, fire, land cover) [MPI-M] Ocean ECVs (SST, sea-ice) [DLR, MPI-M] Batch Options 1 – O5.1 Atmosphere: Ozone, GHG (CO2) [DLR], Clouds [Met Office] Ocean: ocean colour [MPI-M] Atmosphere, land and ocean [SMHI] Batch Options 1 – O5.2 Evaluation of a selected set of CMIP5 [DLR, MPI, Met Office] and CCI4MIP simulations [MPI-M] Using the results from the ESMValTool, contribute to a report that includes an assessment of: how well state-of-the-art Earth system models simulate climatological mean, variability and trends of selected ECVs, how the new observational time series of the ESA CCI change the model evaluation and benchmarking assessment of the models in comparison to observations provided by obs4MIPs, whether observational uncertainty is still a limiting factor in benchmarking models. Batch Options 2 – O5.1 Atmosphere: GHG (CH4) [DLR]

DLR contributions What has been done so far 1. Maintenance and documentation (Eyring/Lauer/Righi) (WP5.1) A subversion controlled repository has been made available to allow the development by multiple users. In addition, a Mantis bug tracking system and a wiki page is provided to facilitate communication and documentation of the tool. A user guide is in preparation that will support the ESA CCI teams in the use of the tool Work on ESMValTool documentation paper started Technical improvements are on-going First release planned for fall 2015 2. Development and integration of ESA CCI data into the ESMValTool (WP3.6, WP3.7, WP5.1, O3.2, O5.1, O5.2) Atmospheric ECVs: aerosols, Righi/Lauer), ozone and GHG/CO2 (Wenzel), clouds (Lauer/Righi) Ocean: sea ice (Senftleben) Work with new ESA’s new 17 year ATSR climate data record (AOD) started Will be continued in year 2, performance metrics plots will be provided for all CMIP5 models, AOD will be added (also WP3.7) Cloud products (CLT, LWP) from non-ESA sources (also WP3.7) O3, CO2, sea ice (also for WP3.6): work with ESA CCI data will be started in year 2

Technical ESMValTool workshop 3-4 March 2015, Munich, Germany Present status and future perspectives of the ESMValTool Specific questions addressed What technical improvements can be made that would facilitate broad community development and usage? How to simplify ESMValTool integration with existing data repositories and the ESGF infrastructure? What are necessary steps towards a roadmap for ESMValTool enhancement for CMIP6 and beyond? How to prepare the software release and what is an appropriate license? → Technical improvements of the ESMValTool are supported also by ESA CMUG

Technical ESMValTool workshop 3-4 March 2015, Munich, Germany Timeline for submission of first ESMValTool documentation paper The workshop was of particular importance for the technical development of diagnostics of the ESA CCI datasets and their integration into the ESMValTool, and for technical requirements to run the tool on new CMIP data alongside the ESGF. Defining the further mid-term roadmap for ESMValTool development on the international level. A first release of the ESMValTool as open-source software is planned this year as a deliverable of EMBRACE. This release will form an important basis for the work on the ESMValTool as part of ESA CMUG and will allow to include the ESA CCI data in an efficient manner. From ESA CMUG, we are planning to include some selected diagnostics using the Aerosol CCI 17 year ATSR climate data record already in this first release. → Technical improvements of the ESMValTool are supported also by ESA CMUG

Assessment of CMIP5 models with new ESA’s 17 year ATSR climate data record As in Fig. 9.29 of IPCC Chapter 9. AOD integrated over the oceans for CMIP5 models, MODIS and ESA-CCI

Clouds 20-yr average LWP (1986-2005) CMIP5 in comparison with UWisc satellite climatology

LMU contributions (Alexander Löw) Technical work LMU has implemented a flexible python interface into ESMValTool that allows for integration of new diagnostics. LMU has developed a prototype for flexible automated testing of diagnostics within ESMValTool. This will maximize the quality of diagnostics across all ECVs. LMU has started to implement a soil moisture diagnostic to integrate ESA CCI soil moisture data (Loew et al., 2013) – to be finalized for first CMUG release of ESMValTool. Administration LMU has so far not signed a sub-contract; draft of sub-contract was submitted by Met Office and is currently with LMU administration. Until signature and hiring of staff only minor progress can be expected.

SMHI contributions (Ulrika Willén) First tests using ESA-CCI SST with the ESMValTool Example plot

Metrics of Clouds and Radiation Yoko Tsushima ESA CMUG integration meeting May 2015 © Crown copyright Met Office

Auto-assess: Aims: Primary aim is to produce a comprehensive set of metrics & diagnostics to inform model development. Current status: Scientific development currently frozen whilst code is re-written in Python using iris library. Aim to produce a first external release to UM user community in 2015. Currently assumes native UM file format (pp). © Crown copyright Met Office

Current & planned assessment areas Global tropospheric circulation Conservation Clouds & radiation Hydrological cycle Asian monsoon systems MJO ENSO Processes over Africa Tropical cyclones Mid-latitude storm tracks and blocking Stratosphere Land surface Aerosols Ocean Sea Ice © Crown copyright Met Office

Auto-assess metric plot © Crown copyright Met Office

Cloud Regime Error Metric for present day (CREMpd) (Williams and Webb 2009, Tsushima et al.,2013) Tsushima et al 2013 RMSE metric based on cloud clustering of daily ISCCP simulator data The error can be broken down into different cloud regimes, contributions of frequency of occurrence component and in-regime cloud radiative effects Code provides as one of CFMIP diagnostics © Crown copyright Met Office

Improvements are seen in mainly in the radiative properties of optically thicker low and high topped cloud regimes rather than frequency of occurrence. Shallow Mid-level Thin Low level Convective Deep Stratocu Cumulus Topped Cirrus Transition Anvil Conv in in-cloud Net CRE error in regime Net CRE RMSE CFMIP1 -> CMIP5 Change Change between CFMIP-1 and CMIP5 of in-cloud Net Cloud Radiative Effect (Net CRE) Error in Net CRE RMSE within daily ISCCP simulator cloud regimes in the Tropics Tsushima et al 2013 © Crown copyright Met Office

Cloud Feedback Model Intercomparison Project (CFMIP) CFMIP diagnostics code repository Klein et al., 2012 Tsushima et al.,2013, Williams and Webb, 2009 Cloud Regime Error Metric for present day (CREMpd) Qu et al., 2012 Nam et al., 2012 Sherwood et al., 2014 CFMIP encourages climate modelling and diagnostic groups to both use the existing metrics and provide additional relevant codes .

Summary and Outlook An ESM Evaluation Tool is available that facilitates the complex evaluation of ESMs and their simulations (e.g. submitted to international Model Intercomparison Projects such as CMIP, C4MIP, CCMI) The tool is developed under a subversion controlled repository Allows multiple developers from different institutions to join the development team Broad community-wide and international participation in the development is envisaged Collaboration with the WGNE/WGCM metrics panel Current extensions Atmospheric dynamics, biogeochemical processes, cryosphere and ocean Need for a wide range of observational data to be used with the tool Observations should ideally be provided with uncertainty and a technical documentation (e.g. similar to those from obs4MIPs) and in CF compliant netCDF Improving statistical comparison and visualization Regular releases to the wider international community Further develop and share the diagnostic tool and routinely run it on CMIP DECK output and according observations (obs4MIPs) alongside the Earth System Grid Federation (ESGF) Release of ESMVal tool to the public → will contribute to metrics panel code repository Work towards a full community-tool developed by multiple institutions / countries

Thank you!