1. M. Baldauf (DWD)1 29 th WGNE meeting 10-14 March 2014, Melbourne M. Baldauf (DWD) Recent developments in Numerical Methods - with a report from the ECMWF annual seminar on ‚Recent developments in numerical methods for atmosphere and ocean modelling 2013‘

2. M. Baldauf (DWD)2 ECMWF annual seminar 2013 on ‚Recent developments in numerical methods for atmosphere and ocean modelling‘ 2-5 Sept. 2013, ECMWF, Reading, UK organizer: Nils Wedi (ECMWF) invited talks also from many globally modelling centers http://www.ecmwf.int/newsevents/meetings/annual_seminar/2013/index.html (last annual seminars on numerical methods: 2004, 1998, 1991, …) horizontal grid: N. Wood, G. Zängl, B. Skamarock, H. Tomita, J. Thuburn vertical discret.: G. Zängl, M. Hortal, M. Baldauf Finite element based: C. Cotter, F. Giraldo, M. Baldauf time integration:P. Benard, R. Klein, S.-J. Lock unstructured meshes:J. Szmelter reduced equation systems: P. Smolarkiewicz, R. Klein Physics-dynamics-coupling: S. Malardel

3. M. Baldauf (DWD)3 Classical requirements for a dynamical core: Stability Accuracy (up to a certain order; at least 2) Efficiency Today, additional requirements: conservation of certain variables „mimetic“ properties (discretization reproduces some exact analytical properties) well-balancing scalability efficient on (quite) different computer architectures (CPU, GPU,…)

4. M. Baldauf (DWD)4 Development and projection of HPC over the years (source: www.top500.org) factor 1.8 per year = factor 2 per 14 months estimated ~10 8 cores  code scalability is crucial #1 #500 sum over all 500 1993 2003 2013 1 GFlops 1 TFlops 1 PFlops 1 EFlops 3*10 6 cores ~10 4 cores

5. M. Baldauf (DWD)5 Since the clock rate of supercomputers does not increase  increase of computer power only by parallelisation  scalability of the code is very important Many current global models (still) use a lat-lon grid  clustering of grid points around the pole  get rid of the strong time step restriction by combination of Semi-Lagrangian-advection and semi-implicit time integration  SL for high Courant numbers requires intensive communication  detrimental for scalability  look for horizontally more uniform grids (lat-lon is not a problem for limited area models)

6. M. Baldauf (DWD)6 Horizontal grids review of horizontal grids on the sphere: Staniforth, Thuburn (2012) QJRMS Why bothering with all this horizontal grid stuff? more uniform grids (see above) grid staggering  degrees of freedom (DoFs) per cell and/or variable see also a general overview about ‚computational modes‘ by J. Thuburn

7. Linear 1D wave equation as a prototype for hyperbolic equations continuous: unstaggered staggered: e.g. D. R. Durran: Numerical methods … M. Baldauf (DWD) 7 ‚modified‘ wavenumber wave ansatz: Why grid staggering?

8. Frequency  Phase-, group-velocity unstaggered staggered grid Dispersion relation of the linear 1D wave equation v ph v gr v ph v gr k xk x k xk x k xk x k xk x  x/c x/c  x/c x/c  =0 for 2  x waves  t stagg = ½  t unstagg negative group velocity M. Baldauf (DWD) 8 Why grid staggering?

9. M. Baldauf (DWD)9 N. Wood (UK MetOffice)

10. M. Baldauf (DWD)10 N. Wood (UK MetOffice)

11. M. Baldauf (DWD)11 N. Wood (UK MetOffice) unfortunately low accuracy of discretizations

12. M. Baldauf (DWD)12 B. Skamarock (NCAR)

13. M. Baldauf (DWD)13 Currently used grids for operational models UK MetOffice: currently lat-lon grid (‚New Dynamics‘, ‚ENDGame‘) GungHo: more uniform grids (example see above) NCAR: MPAS uses icosahedron  dual grid  hexagonal C-grid DWD: currently (GME): icosahedron (however, regular grid on 10 diamonds) new: ICON uses icosahedron with triangle refinement, triangular C-grid Canada: currently lat-lon plans: Yin-Yang grid in 2015 China (CMA): FD on regular lat-lon grid  SISL scheme new: (multi-moment) FD; Yin-Yang grid or cubed sphere. …

14. M. Baldauf (DWD)14 B. Skamarock (NCAR) this is still a conformal grid (i.e. no hanging nodes)

15. M. Baldauf (DWD)15 ECMWF: horizontally spectral (vertically FE) MeteoFrance: currently spectral models (share dyn. core with ECMWF) however, vision for FE discretization horizontally JMA: GSM uses spectral approach Brazil: spectral / lat-lon  SI integration … alternatively: hide most of the problems with the horizontal grid behind a spectral representation 

16. 16 The Integrated Forecasting System (IFS) technology applied at ECMWF for the last 30 years … A spectral transform, semi-Lagrangian, semi-implicit (compressible) (non-)hydrostatic model Big obstacle could be removed: Legendre transform is O(N^3), fast Legendre Transform O(N^2 log N) (Wedi et al. (2013), Tygert (2008, 2010) ) Similar statements hold for Aladin (biperiodic Fourier series) N. Wedi (ECMWF)

17. M. Baldauf (DWD)17 N. Wedi (ECMWF)

18. M. Baldauf (DWD)18 by G. Mozdzynski

19. M. Baldauf (DWD)19 Vertical grids issues: vertical finite element representation proper vertical averages in vertically staggered grid for FD are necessary

20. Improvement of the vertical discretization Arithmetic average from half levels to main level: Weighted average from main levels to half level Derivatives always by centered differences (appropriate average used before) M. Baldauf (DWD)20 Vertical (Lorenz)-staggering in COSMO (and a few other models): Half levels ( w -positions) are defined by a stretching function z k = f (  k ); Main levels ( p ‘, T ‘-pos.) lie in the middle of two half levels The above mentioned staggering requires proper averaging; example …

21. Divergence with weighted average of u (and v ) to the half levels: Divergence with only arithmetic average of u (and v ) to the half levels: Divergence – grid stretching variant B not a consistent discretization if s  1 ! 1/s ·dz dz s · dz Truncation error in stretched grids M. Baldauf (DWD)21

22. M. Baldauf (DWD)22 Time integration Meteo France (P. Benard): 3-time level Semi-implicit Leapfrog based integration + iteration step (ICI-scheme) (on spectral model) DWD (ICON): 2-time level Predictor-Corrector –scheme (HEVI in each step) COSMO: 2-timelevel Runge-Kutta (split explicit) NCAR: 2-timelevel Runge-Kutta (split explicit) for WRF and MPAS UK MetOffice: currently SI-SL scheme outlook: IMEX Runge-Kutta under consideration JMA: GSM uses SI-SL scheme (for the spectral model) …

23. M. Baldauf (DWD)23

24. M. Baldauf (DWD)24

25. M. Baldauf (DWD)25

26. M. Baldauf (DWD)26 Equation systems nearly all centers will use the non-hydrostatic, fully compressible equations for global applications (UK MO, DWD, RIKEN (Japan), NCAR, …)

27. M. Baldauf (DWD)27 Advection schemes For tracer advection beyond sufficient accuracy, two ingredients are seen as important: conservation  finite volume discretization consistency with continuity equation (=the advection equation for  q should reproduce those of  if q=1) easy, if the same advection scheme is used for  as for all tracers; however, NICAM, ICON, …use different time steps for continuity equation and tracers  use time averaged mass fluxes! (talks by H. Tomita, G. Zängl)

28. M. Baldauf (DWD)28 Semi-Lagrangian advection (overview given by M. Diamantakis) as used for the dynamical core in: ARPEGE(MeteoFrance)/ IFS(ECMWF)/ALADIN UM(UKMO) HIRLAM SL-AV(Russia) GEM(Environment Canada) GFS(NCEP) GSM(JMA) … For tracer advection, local conservation properties are not easy to achieve: e.g. SLICE-3D (Zerroukat, Allen (2012) QJRMS) implemented in ENDGame version of UM (but computationally expensive) on unstructured grids: conservative SL probably too expensive. (conservative) SL schemes on structured grids may be beneficial for many tracers

29. M. Baldauf (DWD)29 Finite-element based methods several Continuous Galerkin (CG) / spectral elements / Discontinuous Galerkin (DG) developments: NUMA (Naval postgraduate school, Monterey) F. Giraldo, … HOMME (NCAR) R. Nair, … CAM-SE (Sandia Nat. lab.) M. Taylor COSMO (DWD) first steps by D. Schuster, … … general remarks in talk of C. Cotter

30. M. Baldauf (DWD)30 C. Cotter (Imperial Coll.) linear 1D wave equation 

31. M. Baldauf (DWD)31 C. Cotter (Imperial Coll.)  Ladyzhenskaya/Babuska/Brezzi (LBB) condition guarantees compatibility

32. Discontinuous Galerkin (DG) methods in a nutshell From Nair et al. (2011) in ‚Numerical techniques …  weak formulation (increases solution space) Finite-element ingredient Finite-volume ingredient  ODE-system for q (k) jl Lax-Friedrichs flux M. Baldauf (DWD)32 e.g. Cockburn, Shu (1989) Math. Comput. Cockburn et al. (1989) JCP e.g. Legendre-Polynomials  talk by F. Giraldo

33. 1D wave expansion with a Discontinuous Galerkin (DG) discretization Literature: Hu, Hussaini, Rasetarinera (1999) JCP: 1D advection-, 2D wave-equation Hu, Atkins (2002) JCP: non-uniform grids  k = k (  ) Ainsworth (2004) JCP: multi-dim. advection equation M. Baldauf (DWD)33

34. DG with p=0,1,2,3 (  =c used) k xk x k xk x Re   x / c Im   x / c p max |  |  x/c 0 1 1 3.9 2 7.51 3 11.83 4 16.86 5 22.58 6 28.96 10 60.75 15113.68  max |  |  x/c  1 + 2.6 p + 0.33 p² increases slightly stronger than linear with p. Choose not too large p! M. Baldauf (DWD)34 Numerical dispersion relation for the 1D wave equation

35. M. Baldauf (DWD)35 F. Giraldo (NPS)

36. M. Baldauf (DWD)36 Other remarks „WGNE table about computer ressources 2014“ up to now, I have received contributions from Chiashi (Japan), Hoon (Korea), Xueshun (China), Ayrton (Canada), Jean-Noel (ECMWF) and DWD shall we continue the survey about dynamical cores (by Mikhail Tolstyk) ?