1. ECMWF Potsdam 2010 Slide 1 Non-hydrostatic modeling on the sphere with ultra-high resolution PDE’s on the sphere, Potsdam 2010 Nils Wedi, Mats Hamrud and George Mozdzynski Many thanks to Agathe Untch and Fernando II

2. ECMWF Potsdam 2010 Slide 2 Introduction – A history  Resolution increases of the deterministic 10-day medium-range Integrated Forecast System (IFS) over ~25 years at ECMWF:  1987: T 106 (~125km)  1991: T 213 (~63km)  1998: T L 319 (~63km)  2000: T L 511 (~39km)  2006: T L 799 (~25km)  2010: T L 1279 (~16km)  2015?: T L 2047 (~10km)  2020-???: (~1-10km) Non-hydrostatic, cloud-permitting, substan- tially different cloud-microphysics and turbulence parametrization, substantially different dynamics-physics interaction ?

3. ECMWF Potsdam 2010 Slide 3 Higher resolution influence (250hPa vorticity) Extreme events associated with vorticity filaments Variance: Skewness: Kurtosis: 125 km 1 km 12.5 km Predicted from linear stochastic models forced with non-Gaussian noise (Sardeshmukh and Surda, 2009)

4. ECMWF Potsdam 2010 Slide 4 Ultra-high resolution global IFS simulations  T L 0799 (~ 25km) >> 843,490 points per field/level  T L 1279 (~ 16km) >> 2,140,702 points per field/level  T L 2047 (~ 10km) >> 5,447,118 points per field/level  T L 3999 (~ 5km) >> 20,696,844 points per field/level (world record for spectral model ?!)

5. ECMWF Potsdam 2010 Slide 5 The Gaussian grid Full grid Reduced grid Reduction in the number of Fourier points at high latitudes is possible because the associated Legendre functions are very small near the poles for large m. About 30% reduction in number of points

6. ECMWF Potsdam 2010 Slide 6 Preparing for the future: The nonhydrostatic IFS  Developed by Météo-France and its ALADIN partners Bubnová et al., (1995); ALADIN (1997); Bénard et al. (2004,2005,2010)  Made available in IFS/Arpège by Météo-France (Yessad, 2008)  Testing of NH-IFS described in Techmemo TM594 (Wedi et al. 2009)

7. ECMWF Potsdam 2010 Slide 7

8. ECMWF Potsdam 2010 Slide 8 Orography – T1279 Alps

9. ECMWF Potsdam 2010 Slide 9 Orography T3999

10. ECMWF Potsdam 2010 Slide 10 T1279 Skandinavia 24 h total cloud cover H

11. ECMWF Potsdam 2010 Slide 11 T3999 Skandinavia 24 h total cloud cover H

12. ECMWF Potsdam 2010 Slide 12 T3999 Skandinavia 24 h total cloud cover NH

13. ECMWF Potsdam 2010 Slide 13 Kinetic Energy Spectra (10hPa) T3999 – T1279 H IFS T L 3999 T L 1279

14. ECMWF Potsdam 2010 Slide 14 Kinetic Energy Spectra (500hPa) T3999 – T1279 H IFS T L 3999 T L 1279

15. ECMWF Potsdam 2010 Slide 15 Computational Cost at T L 3999 Total cost increase for 24h forecast: H 50min vs. NH 150min

16. ECMWF Potsdam 2010 Slide 16 H T L 3999 NH T L 3999 Computational Cost at T L 3999 hydrostatic vs. non-hydrostatic IFS

17. ECMWF Potsdam 2010 Slide 17 Schematic description of the spectral transform method in the ECMWF IFS model Grid-point space -semi-Lagrangian advection -physical parametrizations -products of terms Fourier space Spectral space -horizontal gradients -semi-implicit calculations -horizontal diffusion FFT LT Inverse FFT Inverse LT Fourier space FFT: Fast Fourier Transform, LT: Legendre Transform No grid-staggering of prognostic variables

18. ECMWF Potsdam 2010 Slide 18 Horizontal discretisation of variable X (e.g. temperature) FFT (fast Fourier transform) using N F  2N+1 points (linear grid) (3N+1 if quadratic grid) “fast” algorithm available … Legendre transform by Gaussian quadrature using N L  (2N+1)/2 “Gaussian” latitudes (linear grid) ((3N+1)/2 if quadratic grid) “fast” algorithm desirable … Triangular truncation (isotropic) Spherical harmonics associated Legendre polynomials Fourier functions Triangular truncation: m n N m = -N m = N

19. ECMWF Potsdam 2010 Slide 19 Computation of the associated Legendre polynomials  Increase of error due to recurrence formulae (Belousov, 1962)  Recent changes to transform package went into cycle 35r3 that allow the computation of Legendre functions and Gaussian latitudes in double precision following (Schwarztrauber, 2002) and increased accuracy 10 -13 instead of 10 -12.  Note: the increased accuracy leads in the “Courtier and Naughton (1994) procedure for the reduced Gaussian grid” to slightly more points near the poles for all resolutions.  Note: At resolutions > T3999 above procedure needs review!

20. ECMWF Potsdam 2010 Slide 20 Spectral transform method  FFT can be computed as C*N*log(N) where C is a small positive number and N is the cut-off wave number in the triangular truncation.  Ordinary Legendre transform is O(N 2 ) but can be combined with the fields/levels such that the arising matrix-matrix multiplies make use of the highly optimized BLAS routine DGEMM.  But overall cost is O(N 3 ) for both memory and CPU time requirements. Desire to use a fast Legendre transform where the cost is proportional to C*N*log(N) with C << N and thus overall cost N 2 *log(N)

21. ECMWF Potsdam 2010 Slide 21 Fast Legendre transform  The algorithm proposed in (Tygert, 2008) suitably fits into the IFS transform library by simply replacing the single DGEMM call with 2 new steps plus more expensive pre-computations.  (1) Instead of the recursive Cuppen divide-and-conquer algorithm (Tygert, 2008) we use the so called butterfly algorithm (Tygert, 2010) based on a matrix compression technique via rank reduction with a specified accuracy to accelerate the arising matrix-vector multiplies (sub-problems still use dgemm).  (2) The arising interpolation from one set of roots of the associated Legendre polynomials to another can be accelerated by using a FMM (fast multipole method).

22. ECMWF Potsdam 2010 Slide 22

23. ECMWF Potsdam 2010 Slide 23 Summary and outlook  “Pushing the boundaries” with first T L 3999 simulations.  Nonhydrostatic IFS: Computational cost (almost 3 x at T L 3999) is a serious issue !  Even with the hydrostatic IFS at T L 3999 the spectral computations are about 50% of the total computing time.  Renewed interest in “Fast Legendre Transform” (Tygert, 2008,2010).  Renewed interest in “Non-iterative methods” and an a- priori filtering of acoustic waves (Arakawa and Konor, 2008).

24. ECMWF Potsdam 2010 Slide 24 Additional slides

25. ECMWF Potsdam 2010 Slide 25 The Athena Project: Does resolution matter for climate prediction …  IFS (latest cycle 36r1) atmosphere-only runs with prescribed lower boundary conditions (SST data from observations until 2007, 2070- A1B scenario SST forcing comes from CCSM simulation)  Four different horizontal resolutions:  T L 159L91 ( ~125 km)  T L 511L91 ( ~39 km)  T L 1279L91 ( ~16 km)  T L 2047L91 ( ~10 km)  Two different setups  Set of 13-months long integrations (1960-2007)  T L 1279 and T L 159 AMIP long runs (1960-2007 and 2070-2117)

26. ECMWF Potsdam 2010 Slide 26 Skewness and (excess) Kurtosis Predicted from linear stochastic models forced with non-Gaussian noise (Sardeshmukh and Surda, 2009) EULAG IFS Variance: Skewness: Kurtosis:

27. ECMWF Potsdam 2010 Slide 27 Cyclonic vorticity (extreme events) For example vorticity filaments are associated with high skewness and high kurtosis !

28. ECMWF Potsdam 2010 Slide 28 Kinetic Energy Spectra (500hPa) T3999 NH vs. H IFS H T L 3999 NH T L 3999 Note that the difference arises most likely from the increased horizontal diffusion necessary for stability in the NH-IFS simulation.

29. ECMWF Potsdam 2010 Slide 29 T1279 Precipitation

30. ECMWF Potsdam 2010 Slide 30 T3999 precipitation

31. ECMWF Potsdam 2010 Slide 31 Computational Cost at T L 2047 Total cost increase NH – H 106 %