1. Solution of the Implicit Formulation of High Order Diffusion for the Canadian Atmospheric GEM Model “High Performance Computing and Simulation Symposium 2008” Ottawa, Canada, April 14-16, 2008 Abdessamad Qaddouri & Vivian Lee Atmospheric Science & Technology

2. Ottawa, Canada, April 14-16, 2008 2 Outline Introduction of GEM Model High order Diffusion equation and solution Parallelization of the solution Numerical performance Tests Conclusion

3. Ottawa, Canada, April 14-16, 2008 3 Numerical Weather Prediction (NWP) Physics Applied Mathematics Real-time applications Computers at Canadian Meteorological centre (CMC)

4. Ottawa, Canada, April 14-16, 2008 4 051030 365 deterministic forecasts probabilistic forecasts (days) 90 2 Statistical (4 times per year) 1 empirical forecasts 2.5 km resolution(once per day) 15 km resolution (twice per day) 35 km resolution(once per day) 100 km resolution (once per day) 250 km resolution (twice per month) 250-400 km resolution (4 times per year) Forecast lead time

5. Ottawa, Canada, April 14-16, 2008 5 Variable Uniform Rotated Limited Area 15km= 574x641x58 35km=800x600x58 2.5km=672x494x58

6. Ottawa, Canada, April 14-16, 2008 6 Hydrostatic Model Horizontal motion (momentum) Thermodynamics, hydrostatic and state Continuity and boundary conditions

7. Ottawa, Canada, April 14-16, 2008 7 Schematic for Semi lagrangian implicit Method used for the integration of GEM Model DiscretizationTrajectory Nonlinear Iterations Diffusion on specific fields

8. Ottawa, Canada, April 14-16, 2008 8 Horizontal High order Diffusion Horizontal prognostic field Damping rate Wave-length Damping rate

9. Ottawa, Canada, April 14-16, 2008 9 Horizontal High order Diffusion… Horizontal prognostic field Implicit Discretization

10. Ottawa, Canada, April 14-16, 2008 10 Horizontal High order Diffusion … Del 4 Horizontal Diffusion Spatial Discretization

11. Ottawa, Canada, April 14-16, 2008 11 Spatial disretization

12. Ottawa, Canada, April 14-16, 2008 12 Horizontal High order Diffusion … Fast Direct Solution Projection

13. Ottawa, Canada, April 14-16, 2008 13 Horizontal High order Diffusion … Direct Solution Matrix Form

14. Ottawa, Canada, April 14-16, 2008 14 Horizontal High order Diffusion … Block Tri-diagonal problem solution Solution

15. Ottawa, Canada, April 14-16, 2008 15 Summary of the algorithm Analysis of the right hand side (FFT or MMM) Solution of (Nk*Ni) tri-diagonal Problems Synthesis of the solution (FFT or MMM)

16. Ottawa, Canada, April 14-16, 2008 16 A Parallel algorithm Global Transposition (Ni/P,Nj/Q,Nk) (Nj/Q,Nk/P,Ni) Analysis of the right hand side Global Transposition (Nj/Q,Nk/P,Ni) (Nk/P,Ni/Q,Nj) Solution of the block tridiagonal problems Global Transposition (Nk/P,Ni/Q,Nj) (Nj/Q,Nk/P,Ni) Synthesis of the solution Global Transposition (Nj/Q,Nk/P,Ni) (Ni/P,Nj/Q,Nk)

17. Ottawa, Canada, April 14-16, 2008 17 35km mesoglobal run At 72hr forecast U component without diffusion U component with DEL 6 diffusion

18. Ottawa, Canada, April 14-16, 2008 18 Table 1. Breakdown of timings in the major components of the Canadian 35Km mesoglobal operational model for an integration of 72 hours on 12 nodes (2 x 24 x 4) ComponentsTime(sec)Percentage Rhs14.081.48 Adv247.7126.01 Prep14.241.49 Nli33.113.48 Sol71.067.46 Bac13.41.41 Phy435.1945.7 Hzd82.868.7 vspng82.862.14 output10.381.09 Others9.911.04 Total952.31100

19. Ottawa, Canada, April 14-16, 2008 19 Table 2. MPI test runs for 35km mesoglobal (OpenMP=1); the number of calls to the diffusion is 964 times Setup P x Q Number of PEs NodesDiffusion Time(sec) Relative Ideal Speedup Relative Speedup 1x16 16 1 596.4611 2x16 32 2 320.4621.86 2x24 48 3 222.3432.68 4x16 64 4 170.1243.51

20. Ottawa, Canada, April 14-16, 2008 20 Table 3. MPI test runs for 17 Km mesoglobal (OpenMP=1); the number of calls to the diffusion is 964 times. Setup P x Q Number of PEs NodesDiffusion Time(sec) Relative Ideal Speedup Relative Speedup 2x1632 2 1769.48 1 1 2x2448 3 1206.01 1.5 1.47 4x1664 4 915.83 2 1.93 4x2080 5 764.13 2.5 2.32 4x2496 6 646.64 3 2.74 7x16112 7 620.98 3.5 2.85 8x16128 8 595.77 4 2.97

21. Ottawa, Canada, April 14-16, 2008 21 MPI Relative Speedup 35km Mesoglobal FFT 17km Mesoglobal FFT

22. Ottawa, Canada, April 14-16, 2008 22 Table 4. OpenMP test runs for 35Km mesoglobal configured (1 x 16 x OpenMP) using FFT: the number of calls to the diffusion is 964 times. OpenMP NodesDiffusion Time(sec) Relative Ideal Speedup Relative Speedup 1 1 596.46 11 4 4 186.41 43.2 8 8 132.27 84.51

23. Ottawa, Canada, April 14-16, 2008 23 Table 5. OpenMP test runs for 35Km mesoglobal configured (1 x 16 x OpenMP) using Matrix multiplication: the number of calls to the diffusion is 1084 times. OpenMP Nodes Diffusion Time(sec)Relative Ideal Speedup Relative Speedup 1 1 2129.93 11 4 4 588.08 43.62 8 8 348.44 86.11

24. Ottawa, Canada, April 14-16, 2008 24 OpenMP relative Speedup 35km Mesoglobal FFT 35km Mesoglobal MXM

25. Ottawa, Canada, April 14-16, 2008 25 Conclusion An efficient implementation of the parallel Fast Direct Solution for the implicit formulation of horizontal diffusion problem Comparison with iterative methods like preconditioned Krylov methods.

26. Ottawa, Canada, April 14-16, 2008 26 Thank You! Merci!