1. Priority Project Performance On Massively Parallel Architectures (POMPA) Nice to meet you! COSMO GM10, Moscow

2. Overview Motivation COSMO code (as seen by computer engineer) Important Bottlenecks Memory bandwidth Scaling I/O POMPA overview

3. Motivation What can you do with more computational power? # EPS members (x 2) Resolution (x 1.25) Lead time (x 2) Model complexity (x 2)

4. Motivation How to increase computational power? Efficiency Algorithm Computer POMPA

5. Motivation Moore’s law has held since 1970’s and will probably continue to hold Up to now we didn’t need to worry too much about adapting our codes, why should we worry now? ?

6. Current HPC Platforms Research system: Cray XT5 – “Rosa” 3688 AMD hexa-core Opteron @ 2.4 GHz (212 TF) 28.8 TB DDR2 RAM 9.6 GB/s interconnect bandwidth Operational system: Cray XT4 – “Buin” 264 AMD quad-core Opteron @ 2.6 GHz (4.6 TF) 2.1 TB DDR RAM 7.6 GB/s interconnect bandwidth Old system: Cray XT3 – “Palu” 416 AMD dual-core Opteron @ 2.6 GHz (5.7 TF) 0.83 TB DDR RAM 7.6 GB/s interconnect bandwidth Source: CSCS

7. The Thermal Wall Power ~ Voltage 2 × Frequency ~ Frequency 3 Clock frequency will not follow Moore’s Law! Source: Intel

8. Moore’s Law Reinterpreted Number of cores doubles every year while clock speed decreases (not increases) Source: Wikipedia

9. What are transistors used for? AMD Opteron (single-core) Source: Advanced Micro Devices Inc. memory (latency avoidance) load/store/control (latency tolerance) memory and I/O interface

10. The Memory Gap Memory speed only doubles every 6 years! Source: Hennessy and Patterson, 2006

11. “Brutal Facts of HPC” Massive concurrency – increase in number of cores, stagnant or decreasing clock frequency Less and “slower” memory per thread – memory bandwidth per insruction/second and thread will decrease, more complex memory hierarchies Only slow improvements of inter-processor and inter-thread communication – interconnect bandwidth will improve only slowly Stagnant I/O sub-systems – technology for long-term data storage will stagnate compared to compute performance Resilience and fault tolerance – mean time to failure of massively parallel system may be short as compared to time to solution of simulation, need fault tolerant software layers We will have to adapt our codes to exploit the power of future HPC architectures!  Source: HP2C

12. Why a new Priority Project? Efficient codes may enable new science and save money for operations We need to adapt our codes to efficiently run on current / future massively parallel architectures! Great opportunity to profit from the momentum and knowhow generated by the HP2C or G8 projects and use synergies (e.g. ICON). Consistent with goals of the COSMO Science Plan and similar activities in other consortia.

13. COSMO Code How would a computer engineer look at the COSMO code?

14. COSMO Code 227’389 lines of Fortran 90 code % Code Lines% Runtime (C-2 forecast) active

15. Dynamics

16. Key Algorithmic Motifs Stencil computations do k=1,ie do j=1,je do i=1,ie a(i,j,k) = w1 * b(i+1,j,k) + w2 * b(i,j,k) + w3 * b(i-1,j,k) end do end do end do Tridiagonal solver (vertical, Thomas alogrithm) do j=1,je ! Modify coefficients do k=2,ke do i=1,ie c(i,j,k) = 1.0 / ( b(i,j,k) – c(i,j,k-1) * a(i,j,k) ) d(i,j,k) = ( d(i,j,k) – d(i,j,k-1) * a(i,j,k) ) * c(i,j,k) end do end do ! Back substitution do k=n-1,1,-1 do i=1,ie x(i,j,k) = d(i,j,k) – c(i,j,k) * x(i,j,k+1) end do end do end do

17. field(ie,je,ke,nt) [in Fortran first is fastest varying] Optimized for minimal computation (pre calculations) Optimized for vector machine Often repeatedly sweeps over the complete grid (bad cache usage) A lot of copy paste for handling different configurations (difficult to maintain) Metric terms and different averaging positions make code complex Code / Data Structures

18. Parallelization Strategy How do distribute work onto O(1000) cores? 2D-domain decomposition using MPI library calls Example: operational COSMO-2 Total: 520 x 350 x 60 gridpointsPer core: 24 x 16 x 60 gridpoints Exchange information with MPI halo/comp = 0.75

19. Bottlenecks? What are/will be the main bottlenecks of the COSMO code on current/future massively parallel architectures? Memory bandwidth Scalability I/O

20. Memory scaling Problem size 102 x 102 x 60 gridpoints (60 cores, similar to COSMO- 2) Keep number of cores constant, vary number of cores/node used Relative Runtime (4 cores = 100%)

21. HP2C: Feasibility Study Goal: Investigate how COSMO would have to be implemented in order to reach optimal performance on modern processors Tasks understand the code performance model prototype software new software design proposal Company http://www.scs.ch/http://www.scs.ch/ Duration 4 months (3 months of work)

22. Focus only on dynamical core (fast wave solver) as it… dominates profiles (30% time) contains the key algorithmic motifs (stencils, tridiagonal solver) is manageable size (14’000 lines) can be run stand-alone in a meaningful way correctness of prototype can be verified Feasibility Study: Idea

23. Feasibility Study: Results Prototype vs. Original

24. Key Ingredients Reduce number of memory accesses (less precalculation) Change index order from (i,j,k) to (2,k,i/2,j) or (2,k,j/2,i,) cache efficiency in tridiagonal solver don’t load halo into cache Use iterators instead of on the fly array position computations Merge loops in order to reduce the number of sweeps over full domain Vectorize as much as possible of code

25. GPUs have O(10) higher bandwidth! Source: Prof. Aoki, Tokio Tech

26. Bottlenecks? What are the main bottlenecks of the COSMO code on current/future massively parallel architectures? Memory bandwidth Scalability I/O

27. “Weak” scaling Problem size 1142 x 765 x 90 gridpoints (dt = 8s) “COSMO-2” Matt Cordery, CSCS

28. Strong scaling (small problem) Problem size 102 x 102 x 60 gridpoints (dt = 20s) “COSMO-2”

29. Improve Scalability? Several approaches can be followed... Improve MPI parallelization Hybrid parallelization (loop level) Hybrid parallelization (restructure code)...

30. Hybrid Motivation NUMA = Non-Uniform Memory Access Nodes views… Reality

31. Hybrid Pros / Cons Pros Eliminates domain decomposition at node Automatic memory coherency at node Lower (memory) latency and faster data movement within node Can synchronize on memory instead of barrier Easier on-node load balancing Cons Benefit for memory bound codes questionable Can be hard to maintain

32. Hybrid: First Results OpenMP on loop level (> 600 directives) Matt Cordery, CSCS linear speedup

33. Bottlenecks? What are the main bottlenecks of the COSMO code on current/future massively parallel architectures? Memory bandwidth Scalability I/O

34. The I/O Bottleneck NetCDF I/O is serial and synchronous grib1 output is asynchronous (and probably not in an ideal way) No parallel output exists! Example: Operational COSMO-2 run REF (s)NO OUTPUT (s) DIFF (s) TOTAL18891676-212 (-11%) MPI571387-184 USER13171289-28 MPI_gather1781-177 cal_conv_ind220-22 organize_output30-3 tautsp2d10

35. PP-POMPA Performance On Massively Parallel Architectures Goal Prepare COSMO code for emerging massively parallel architectures Timeframe 3 years (Sep. 2010 – Sep. 2013) Status Draft of project plan has been sent around. STC has approved the project. Next step Kickoff meeting and detailed planning of activities with all participants.

36. Tasks ① Performance analysis ② Redesign memory layout ③ Improving scalability (MPI, hybrid) ④ Massively parallel I/O ⑤ Adapt physical parametrizations ⑥ Redesign dynamical core ⑦ Explore GPU acceleration ⑧ Update documentation Current COSMO code base New code and programming models  See project plan!

37. Who is POMPA? DWD (Ulrich Schättler, …) ARPA-SIMC, USAM & CASPUR (Davide Cesari, Stefano Zampini, David Palella, Piero Lancura, Alessandro Cheloni, Pier Francesco Coppola, …) MeteoSwiss, CSCS & SCS (Oliver Fuhrer, Will Sawyer, Thomas Schulthess, Matt Cordery, Xavier Lapillonne, Neil Stringfellow, Tobias Gysi, …) And you?

38. Questions? Coming to a supercomputer near your soon!