Energy efficient SCalable Algorithms for weather Prediction at Exascale www.hpc-escape.eu Peter Bauer

ESCAPE key objectives Define fundamental algorithm building blocks (“Weather & Climate Dwarfs”) to co-design, advance, benchmark and efficiently run the next generation of NWP and climate models on energy-efficient, heterogeneous HPC architectures. Combine frontier research on algorithm development and extreme-scale, high-performance computing applications with novel hardware technology, to create a flexible and sustainable weather and climate prediction system. Foster the future design of Earth-system models and commercialisation of weather-dependent innovative products and services in Europe through enabling open-source technology. Pairing world-leading NWP with innovative HPC solutions.

ESCAPE European impact map 34 countries 7 countries 11 countries 16 countries

ESCAPE partners & expertise Global operational NWP Petaflop data centre Numerical methods, parallel computing Regional operational NWP Numerical methods Key technology development Parallel computing Regional operational NWP Teraflop data centre Physical processes Numerical methods Teraflop data centre Parallel computing Teraflop data centre Parallel computing Performance modelling Teraflop data centre Parallel computing Global/regional operational NWP Teraflop data centre Numerical methods Global/regional operational NWP Petaflop data centre Numerical methods Key technology development Parallel computing Key technology development Parallel computing Regional operational NWP Petaflop data centre Parallel computing

Traditional science workflow [Schulthess 2015]

Future science workflow Energy efficient SCalable Algorithms for weather Prediction at Exascale  www.hpc-escape.eu  science specific code  generic code [Schulthess 2015]

Energy efficiency .. aiming at minimizing Watts per forecast

Model development: ESCAPE Energy efficient SCalable Algorithms for weather Prediction at Exascale (hpc-escape.eu) Disassemble global … extract, redesign … optimize for energy … reassemble global NWP model… key components = dwarfs… efficiency on new hardware… NWP model

Computational cost Global, spectral Regional, spectral Regional, Eulerian

What’s wrong with weather and climate codes? Advection: halo-communication, scalability Transforms: communication bandwidth, memory Physics: expensive calculations, scalable Ocean waves: expensive calculations, load balancing 3D-solver: iterations, memory, communication bandwidth No 1: sequential with time steps (10d x 24h x 3600s / 450s = 1920 @ 9km Dwarfs: Spectral transforms (FT/LT and bi-FT) ………….. very memory and communication bandwidth intensive, possibly limited scalability 2 & 3-dimensional elliptic solver ……………….... compute and communication latency intensive, possibly limited scalability Semi-Lagrangian advection …………………….... communication intensive, possibly limited scalability Flux-form finite-volume advection ……………….. local communication, latency intensive, limited scalability Cloud physics parameterization …………………. expensive computations, independent vertical columns, scalable Radiation parameterization ………………………. expensive computations, spatial, temporal and wavenumber space, scalable

Dwarfs & adaptation

Separation of concerns: Atlas

Atlas support of grids Grid object is collection of grid points (structured or unstructured), Global or Regional model set-up: with hierarchical interpretations:

Atals & GridTools GridTools library and interfaces: Atlas data structures & functionalities:

Algorithmic flexibility: path and control volume

Algorithmic flexibility: spectral and grid-point

Dwarf performance more fields

Porting of LAITRI dwarf LAITRI performs interpolations for semi-Lagrangian advection scheme The settings to achieve this: correct use of OpenMP (e.g. ensuring correct ‘first touch’ of data), suitable compiler data alignment directives to good vectorisation performance, and prudent setting of runtime variables such as KMP AFFINITY and KMP HW SUBSET. Aim: integrate these settings in Atlas, away from science code!

Weather & Climate in H2020 PantaRhei Reference application Dissemination across community Showcase of new technology Reference application Co-design hardware & programming models Co-design hardware & programming models Ingestion of dwarfs in full-scale ESM Dissemination platform for dwarfs Reference application Reference application Evaluation against alternative programming models High-impact application demonstrator Performance assessment and optimization tools Feedback on tool applicability and value PantaRhei Finite-volume fully compressible core Global NWP implementation

ESCAPE Staff

The 10-year production challenge Data acquisition Single model run Product generation Dissemination RMDCN Internet Web services Archive Data Handling System Scenarios Multi-model run Climate Data Store Observations shared worldwide O(GB) Dissemination To 300 destinations National met services and commercial customers O(TB) No.1 goal: Global 1 km simulations at >1 SYPD Computing and data challenges grow by factor 100-1000

The 10-year technical&programmatic challenge Top-3 computing bottlenecks: Memory bandwidth, Memory Size/Layout, Network latency… but not FLOPS! [G. Kalbe, 2017, European HPC, Time line overview]

The 10-year impact challenge & Climate Observations Satellites Radar Ground stations Nowcasting Weather forecasting Climate prediction Reanalyses + Computing Direct data links Internet TV, radio Other Threat Impact Probability Reliability Warning Protection Buying/selling Investing [J. Lazo 2016, the NWP value chain]