1. IS-ENES2: Contributions to HPC for climate modelling
HPC Summit Week
ENES/ESIWACE Workshop, 16/05/2017, Barcelone
IS-ENES2:
Infrastructure for the European Network for Earth System Modelling
Phase 2
Contributions to HPC for climate modelling
Sylvie Joussaume
CNRS, IPSL, Coordinator IS-ENES2
With contributions from
Giovanni Aloisio, Joachim Biercamp,Mick Carter, Marie Alice Foujols, Uwe Fladrich, Eric Maisonnave, Yann Meurdesoif, Sophie Valcke

2. Infrastructure for ENES
European Network for Earth System modelling
A network of European groups in climate/Earth system modelling
Launched in 2001 (MOU)
Ca 50 groups from academic, public and industrial world
Main focus :
discuss strategy
to accelerate progress in climate/Earth system modelling and understanding
IS-ENES
Infrastructure for ENES
FP7 European projects
IS-ENES
IS-ENES
Infrastructure
Models & their environment
Model data (ESGF-ESDOC)
Interface with HPC ecosystem
Users :
Climate modelling community
(Global & regional)
Impact studies
Several EU projects
ENSEMBLES, COMBINE, EUCLIPSE,
EMBRACE, SPECS, PRIMAVERA, CRESCENDO
PRISM, METAFOR, IS-ENES (1& 2), ESIWACE

3. Earth’s climate system models
Understand and predict climate variability and change
WCRP international coordinated experiments CMIP & CORDEX
HPC needs:
Resolution
Duration
Complexity
Ensembles
Capability
& Capacity
CEA
CMIP5:
7 European models
CMIP6: 9 models
CORDEX
Euro-cordex
Africa Cordex
Evaluate / Understand / Predict

4. IS-ENES : Infrastructure for ENES
FP7 project « Integrating Activities »
1rst phase: March Feb 2013 (7.6 M€), 18 partners
2nd phase: Apr March (8 M€), 23 partners
Better understand and predict climate variability & changes
Foster:
The integration of the European ESM community
The development of ESMs and their environment
High-end simulations
The application of ESM simulations for climate change impacts

5. Measure real sustained performance of climate models:
a comprehensive set of metrics
Definition of a set of metrics (“FLOPS” is not enough) to better estimate and reduce effect of complexity, memory bloat, coupling, workflow and other costs that matter for climate modelling
Measurement of IS-ENES2 high resolution models performance
Computational performance oriented CMIP6 (CPMIP):
ESDOC adaptation and associated tools for reporting
IS-ENES-2 HR
IS-ENES-2 ESM
Balaji et al., GMD, 2017

6. Improve model performance: Coupler Benchmarking coupling technology
OASIS3-MCT
Benchmarking coupling technology
Test of coupler
OASIS 3 versus OASIS3-MCT
Time of exchange between two toy models (25 km resolution)
CURIE
IS-ENES benchmark available on ENES portal :
4 stand alone components /4 different grids
Coupled systems for regular lat-lon grids with OASIS, ESMF, OpenPALM, MCT & YAC
First results available for specific coupled test case (Bullx Occigen, MetOffice Cray, Marconi Broadwell)
OASIS 3
OASIS3-MCT
Sources: S. Valcke
Warning: work still needed to understand differences

7. Improve functionalities
Parallel IOs
2 approaches: XIOS (netcdf) / CDI-pio (grib)
Example XIOS:
IS-ENES1: developed for NEMO
Used in coupled models:
IPSLCM, ARPEGE-NEMO,
planned for EC-Earth, Hadley UM
IS-ENES2
Improve functionalities
Parallel in situ workflow:
Online post-processing
Interpolation / remapping
CMIP6 workflow for model output
Next steps in ESIWACE : openMP
IO server/coupler integration
Y. Meurdesoif

8. Support few developments Configuration management guide:
Workflow
Codes
Configurations
Execution
Initialisation
Execution
Raw data
Post-processing
Share best practices on tools : configuration management, workflow tools
Support few developments
Configuration management guide:
4 levels: codes / configurations / simulations / post-processing
Levels 1 & 2 : common
Also extend to 3 & 4
Configuration management (CM) is defined as a “process for establishing and maintaining consistency of a product's performance, functional and physical attributes with its requirements, design and operational information throughout its life
M. Carter
CYLC meta-scheduler
Construct and control suites of forecasts
And data processing
UKMet
AUTOSUBMIT:
Submission and monitoring of multi member runs
One executable
BSC

9. Learning from NEMO and ICON
Technology tracking
Learning from NEMO and ICON
Source: G. Aloisio & J. Biercamp
NEMO
Nucleus for European
Modeling of the Ocean
ICON
Icosahedral non-hydrostatic general circulation model
~10x10
~40x40
Reduce the data movement overhead and increase memory scaling
Analyze and improve the computation per byte of accessed data
NEMO currently only scales to local domain sizes of ~40x40.
The target for the future should be 10x10.
Adopt hybrid parallel approaches to increase the parallelism (also for time)
Decouple the computational aspects from the numerics

10. Coupled benchmarks
Objective: portable, easy-to-use, and well documented climate model benchmarks for more efficient collaboration with hardware/software vendors
Available Benchmarks on
4 Earth System Models (ESMs)
MPI-ESM1
IPSLCM
CMCC-CESM-NEMO
EC-EARTH
1 component Model ICON
2 Kernels
NEMO tracer advection kernel
ICON decomposition and halo communication kernel
Presented at a series of IS-ENES2 workshops with HPC vendors
Coupled model benchmark: multi-executables with different MIP/OpenMP
Issue of freezing a version – difficult to follow model evolution
To qualify a new system:
coupled benchmark
or at least coupled toy benchmark & tests of coupled model with pre-system

11. Conclusions Fostering high-end experiments :
Started with IS-ENES, now extended through the
ESIWACE Center of Excellence on weather and climate:
prepare for future architectures
IS-ENES next phase : support to international WCRP experiments