IS-ENES2: Contributions to HPC for climate modelling HPC Summit Week ENES/ESIWACE Workshop, 16/05/2017, Barcelone IS-ENES2: Infrastructure for the European Network for Earth System Modelling Phase 2 Contributions to HPC for climate modelling Sylvie Joussaume CNRS, IPSL, Coordinator IS-ENES2 With contributions from Giovanni Aloisio, Joachim Biercamp,Mick Carter, Marie Alice Foujols, Uwe Fladrich, Eric Maisonnave, Yann Meurdesoif, Sophie Valcke

Infrastructure for ENES European Network for Earth System modelling http://enes.org/ http://is.enes.org/ A network of European groups in climate/Earth system modelling Launched in 2001 (MOU) Ca 50 groups from academic, public and industrial world Main focus : discuss strategy to accelerate progress in climate/Earth system modelling and understanding IS-ENES Infrastructure for ENES FP7 European projects IS-ENES 2009-2013 IS-ENES2 2013-2017 Infrastructure Models & their environment Model data (ESGF-ESDOC) Interface with HPC ecosystem Users : Climate modelling community (Global & regional) Impact studies Several EU projects ENSEMBLES, COMBINE, EUCLIPSE, EMBRACE, SPECS, PRIMAVERA, CRESCENDO PRISM, METAFOR, IS-ENES (1& 2), ESIWACE

Earth’s climate system models Understand and predict climate variability and change WCRP international coordinated experiments CMIP & CORDEX HPC needs: Resolution Duration Complexity Ensembles Capability & Capacity CEA CMIP5: 7 European models CMIP6: 9 models CORDEX Euro-cordex Africa Cordex Evaluate / Understand / Predict

IS-ENES : Infrastructure for ENES FP7 project « Integrating Activities » 1rst phase: March 2009- Feb 2013 (7.6 M€), 18 partners 2nd phase: Apr 2013- March 2017 (8 M€), 23 partners http://is.enes.org/ Better understand and predict climate variability & changes Foster: The integration of the European ESM community The development of ESMs and their environment High-end simulations The application of ESM simulations for climate change impacts

Measure real sustained performance of climate models: a comprehensive set of metrics Definition of a set of metrics (“FLOPS” is not enough) to better estimate and reduce effect of complexity, memory bloat, coupling, workflow and other costs that matter for climate modelling Measurement of IS-ENES2 high resolution models performance Computational performance oriented CMIP6 (CPMIP): ESDOC adaptation and associated tools for reporting IS-ENES-2 HR IS-ENES-2 ESM Balaji et al., GMD, 2017

Improve model performance: Coupler Benchmarking coupling technology OASIS3-MCT Benchmarking coupling technology Test of coupler OASIS 3 versus OASIS3-MCT Time of exchange between two toy models (25 km resolution) CURIE IS-ENES benchmark available on ENES portal : http://enes.org 4 stand alone components /4 different grids Coupled systems for regular lat-lon grids with OASIS, ESMF, OpenPALM, MCT & YAC First results available for specific coupled test case (Bullx Occigen, MetOffice Cray, Marconi Broadwell) OASIS 3 OASIS3-MCT Sources: S. Valcke Warning: work still needed to understand differences

Improve functionalities Parallel IOs 2 approaches: XIOS (netcdf) / CDI-pio (grib) Example XIOS: IS-ENES1: developed for NEMO Used in coupled models: IPSLCM, ARPEGE-NEMO, planned for EC-Earth, Hadley UM IS-ENES2 Improve functionalities Parallel in situ workflow: Online post-processing Interpolation / remapping CMIP6 workflow for model output Next steps in ESIWACE : openMP IO server/coupler integration Y. Meurdesoif

Support few developments Configuration management guide: Workflow Codes Configurations Execution Initialisation Execution Raw data Post-processing Share best practices on tools : configuration management, workflow tools Support few developments Configuration management guide: 4 levels: codes / configurations / simulations / post-processing Levels 1 & 2 : common Also extend to 3 & 4 Configuration management (CM) is defined as a “process for establishing and maintaining consistency of a product's performance, functional and physical attributes with its requirements, design and operational information throughout its life M. Carter CYLC meta-scheduler Construct and control suites of forecasts And data processing UKMet AUTOSUBMIT: Submission and monitoring of multi member runs One executable BSC

Learning from NEMO and ICON Technology tracking Learning from NEMO and ICON Source: G. Aloisio & J. Biercamp NEMO Nucleus for European Modeling of the Ocean ICON Icosahedral non-hydrostatic general circulation model ~10x10 ~40x40 Reduce the data movement overhead and increase memory scaling Analyze and improve the computation per byte of accessed data NEMO currently only scales to local domain sizes of ~40x40. The target for the future should be 10x10. Adopt hybrid parallel approaches to increase the parallelism (also for time) Decouple the computational aspects from the numerics

Coupled benchmarks Objective: portable, easy-to-use, and well documented climate model benchmarks for more efficient collaboration with hardware/software vendors Available Benchmarks on 4 Earth System Models (ESMs) MPI-ESM1 IPSLCM CMCC-CESM-NEMO EC-EARTH https://portal.enes.org/computing/benchmarks 1 component Model ICON 2 Kernels NEMO tracer advection kernel ICON decomposition and halo communication kernel Presented at a series of IS-ENES2 workshops with HPC vendors Coupled model benchmark: multi-executables with different MIP/OpenMP Issue of freezing a version – difficult to follow model evolution To qualify a new system: coupled benchmark or at least coupled toy benchmark & tests of coupled model with pre-system

Conclusions Fostering high-end experiments : Started with IS-ENES, now extended through the ESIWACE Center of Excellence on weather and climate: prepare for future architectures IS-ENES next phase : support to international WCRP experiments