PT Evaluation of the Dycore Parallel Phase (EDP2) X. Lapillonne, M. Baldauf, P. Spörri, O. Fuhrer, C. Barbu, C. Osuna, U. Schättler, A. Walser

Main goal Two implementations of the RK dynamical core coexist in COSMO: Fortran C++ based on STELLA library (required to run on GPU) C++ version will soon be in the offical version Evaluate cost and give recommendation to STC regarding the parallel phase where the 2 implementations coexist Stop short presentation at this slide

Cost evaluation The total time maintenance: 0.5 FTE # Type Cost/Change Cost 20 Minor Changes Adapting configuration variables, defaults or changing a line in the computation. 0.005 FTE 0.1 FTE 5 Medium Changes Changes that require new stencils but fit into the existing context. 0.02 FTE 1 Major Changes E.g. port a new Fast Waves Solver 0.2 FTE General maintenance Keeping up with recent compilers, investigation of performance issues. The total time maintenance: 0.5 FTE 0.3 FTE for basic maintenance and small to medium changes 0.2 FTE for integrating major developments Based on previous years, may vary now that the focus switch to ICON and the C++ code is distributed (more small changes, less large one)

Impact and evaluation Impact (on users/developers) : need for additional communication between the main dycore developers and the C++ dycore maintainer Performance on latest architecture of the C++/STELLA as compared to the Fortran dynamics x3 faster on GPU as compare to the original code on CPU   9x Haswell Sockets 96x Compute, 4x I/O Nodes 1x Haswell, 8x K80 Sockets 8x Compute, 2x I/O Nodes Setup GCC 4.9 F90 Dycore C++ Dycore GCC 4.9, CUDA 7.0 Double Precision 316 s 214 s 101 s Single Precision 169 s 152 s 65 s Consequences and experience of using the C++ dynamics for operational weather prediction : No adverse consequences in terms of maintenance, stability or complexity for operations

Developer experience (using STELLA) C++ Dycore Source Code Maintainer (P. Spörri, MCH) : nice and efficient to write single source performance portable HPC code C++ Dycore Users: Experiences at DWD: very different than higher programming language (like Fortran, C++, Python, ...), need permanent support for development. => Seen as a large obstacle for model development Experience at IMGW (port of EULAG with GridTools): while this is early stage, progress is good and currently there are no major obstacles to port the Fortran code to C++. => Divergent opinions !

Impact on support and installation of the COSMO code NMA involved in support activity, and will provides the second level support Impact of using a DSL for the dynamical core Faster and easier to maintain code when targeting multiple architectures Comparison with an OpenACC GPU implementation of some dycore component shows the STELLA implementation is 1.4x to 1.8x faster on

Possible development workflow Working in C++ only (no Fortran dynamics) reference implementation directly using STELLA or plain C++ code (with explicit loops) can be written directly inside the existing C++ dycore dycore maintainer integrates and optimizes the reference code in STELLA

Consequence of discontinuing the Fortran dynamics Training and change of workflow of the main développer (several weeks of investment) Additional training for developer at other universities would be required New training also needed for the user community : compilation of the c++ + Fortran code requires some additional knowhow. Consequence of discontinuing the C++ dynamics Issue for members and universities using the C++ dycore for production (MCH, ETH, EMPA) Would loose the ability to run on GPU architectures which is a strength of the COSMO model

Recommendation Extend the parallel phase of maintaining both the C++ and Fortran implementations Evaluate again after a period of at least two years Cost of having 2 dynamics: 0.5 FTE/Year => Extension accepted by STC