WWOSC 2014 Montreal, Canada Running operational Canadian NWP models on next-generation supercomputers Michel Desgagné, Abdessamad Qaddouri, Janusz Pudykiewicz, Stéphane Gaudreault, Michel Valin, Martin Charron R echerche en P révisions N umériques (RPN – Dorval,Canada)

Plan G lobal E nvironmental M ultiscale NWP model Current large Gem configurations at CMC Scaling up scenarios From Global Lat-Lon to Yin-Yang Investigating new numerics Investigating new horizontal geometry Conclusions

Global Deterministic Prediction System (GDPS) ENVar Data Assimilation 25 km horizontal resolution, 80 levels 00, 12Z runs up to 10-days 1280 IBM P7 cores Current NWP Systems(1) Regional Deterministic Prediction System (RDPS) ENVar Regional Data Assimilation 10 km horizontal resolution, 80 levels 48-hour forecast, 4x per day 1024 IBM P7 cores

Current NWP Systems(2) High-res Deterministic Prediction System (HRDPS): Downscaling from the RDPS 10 km 2.5 km horizontal resolution, 65 levels 48-hour forecast, 4x per day, 2976 IBM P7 cores

Scaling up those NWP Systems GDPS RDPS HRDPS Yin-Yang 10 km, 120 levels 12K “IBM P7” cores (4.5 X current cluster size at 33%) 2.5 km, 80 levels With ENVar & larger domain 5-9K “IBM P7” cores Experimental system Urban scale 250 m Strategic interests Special contracts Yin-Yang ~7 km ?? 36K “IBM P7” cores (3-fold) Yin-Yang ~2.5 km 800K “IBM P7” cores (22-fold) ~1.0 km, 120 levels 200K “IBM P7” cores

The GEM model 1.Grid point lat/lon model 2.Finite differences on an Arakawa-C grid 3.Semi-Lagrangian (poles are an issue) 4.Implicit time discretization 1.Direct solver (Nk 2D horizontal elliptic problems) 2.Full 3D iterative solver based on FGMRES 5.Global Lat-Lon, Yin-Yang and LAM configurations 6.Hybrid MPI/OpenMP 1.Halo exchanges 2.Array transposes for elliptic problems 7.PE block partitioning for I/O Global Lat-Lon grid: 1) a challenge for DM implementation 2) many more elliptic problems to solve due to implicit horizontal diffusion (transposes) 3) semi-Lagrangian near the poles 4) current DM implementation will not scale

Yin-Yang grid configuration Implemented as 2 LAMs communicating at boundaries Optimized Schwarz iterative method for solving the elliptic problem. Scales a whole lot better Operational implementation due in spring 2015 Communications are an issue Exchanging a Global Lat-Lon scalability problem (poles) by another scalability problem Abdessamad Qaddouri, Vivian Lee (2011) The Canadian Global Environmental Multiscale model on the Yin-Yang grid system, Q. J. R. Meteorol. Soc. 137: 1913–1926

H960H3200H5056H1920 Y3200Y8192Y16384Y30968 Y5056 H= CMC/hadar: IBM Power7 Y= NCAR/yellowstone: IBM iDataPlex 16x30x1 198x36 79x32x1 40x34 79x49x4 40x22 32x32x8 99x34 32x32x4 99x34 32x30x1 99x36 40x40x1 79x27 Yin-Yang 10 km scalability Ni=3160, Nj=1078, Nk=158 Y x49x4 20x22

Yin-Yang 10 km scalability Dynamics components semi-Lagrangian transpose Yin-Yang exchanges fft-solver

The future of GEM Yin-Yang 2km on order 100K cores is already feasible on P7 processors or similar Solver and Yin-Yang exchanges will need work Using GPUs capabilities is on the table Improve Omp scalability Reshaping main memory for better cache usage Export SL interpolations to reduce halo size Processor mapping to reduce the need to communicate through the switch Partition NK MIMD approach for I/O: I/O server

Revisiting Time Discretization: Exponential integration methods for meteorological models Jacobi operator Introducing the integrating factor yields Clancy C., Pudykiewicz J. (2013) On the use of exponential time integration methods in atmospheric models, Tellus A, vol. 65

And integrating over followed by multiplication by Depending on the quadrature used we will obtain different versions of the exponential integration schemes. Their common property is that the desired solution could be expressed as weighted sum of “phi functions” Tokman (2006) Efficient integration of large stiff systems of ODEs with exponential propagation iterative methods, J. Comp. Phys. (213),

Exploring exponential integrators with Eulerian finite-volume schemes in GEM The main difficulty of implementing exponential integrators is the evaluation of the φ functions. Recent advance in computational linear algebra is now allowing efficient computation:  Fast, matrix-free, Krylov method  Only the action of the matrix on a vector (matvec product) is need  Completely remove the need to evaluate φ functions  Reduced-size problem : Reduce the large system matrix to a smaller Hessenberg matrix on which calculating the exponential is easy  inherent local and global conservation  non-oscillatory advection (monotonic, positivity preservation)  geometric flexibility (any type of grid system)  stable even with large Courrant numbers

Major Findings Exponential methods can resolve high frequencies to the required level of tolerance without severe time step restriction of the explicit numerical schemes Contrast to usual implicit integration used with large time steps, which either damp high frequencies or map them to one and the same frequency (or nearly so) in the discretization scheme is free of noise even when the low order spatial discretization is used Compute efficienty is very good on linear problems and looks real promising for full model Expected to scale very well because of data locality

Investigating scalability and accuracy on an icosahedral geodesic grid Spatial discretization: finite volume method on icosahedral geodesic grid Time discretization: exponential integration methods which resolve high frequencies to the required level of tolerance without severe time step restriction Shallow water implementation already shows great scalability Vertical coordinates: Generalized quasi-Lagrangian with conservative monotonic remapping Pudykiewicz J. (2011), J. Comp. Phys., 230, pp Qaddouri A., J. et al. (2012), Q. J. Roy. Met. Soc., 138, pp

Conclusions Yin-Yang 2km on order 100K cores is already feasible on P7 processors or similar No real worries for next 4-6 years Ready to address architecture changes: –GPUs –Larger vectors –Larger # of cores per node Many developments items on the agenda Keep investigating new numerics: –that will enhance data locality and limit communications –that will be better suited for upcoming architectures

END