Recent Developments in the NRL Spectral Element Atmospheric Model (NSEAM)* Francis X. Giraldo *Funded by the Office of Naval Research

Motivation Mission of NRL-MRY: Develop NWP models for the Navy Computing Resources: High Performance Computing has shifted towards distributed-memory architectures comprised of tens of thousands of processors (e.g., clustered systems). Current fastest system has 32,000 processors (BlueGene by IBM). Exploiting New Architectures: Need numerical discretization methods that decompose the domain naturally into many smaller subdomains Each subdomain can then be solved independently on-processor

Motivation (continued) State-of-the-Art Global Forecast Models: spectral transformBased on the spectral transform method (i.e. spherical harmonics) Spectral transform method uses global approximation functions and Nproc scales like O(T) and costs O(T^3) New Model: spectral elementNSEAM is based on the spectral element method which combines the high-order accuracy of spectral transforms with domain decomposition property of finite elements Uses local approximation functions and Nproc scales like O(T^2) and costs O(N T^2)

Domain Decomposition ST Models: Nproc = O(T) NSEAM: Nproc=O(T^2)

Spectral Transform Method (Legendre Modal Functions)

Spectral Element Method (Legendre Nodal Functions)

What We Need for a New Model To construct a model the following things are required: 1.Governing set of Partial Differential Equations along with initial and boundary conditions (PDE) 2.A Coordinate System (CS) 3.A Grid Generator in order to place solution values within the domain (GG) 4.A Spatial Discretization method in order to approximate derivatives (SD) 5.A Time-Integrator in order to advance the solution in time (TI) 6.A Very Large Computer (VLC)

Description of NSEAM Hydrostatic primitive equations (PDE) Equations formulated in Cartesian and sigma coordinates (CS) Spectral Element method in horizontal (SD) Finite Volume method in vertical (SD) 2 nd order semi-implicit in time (TI) GMRES with block Jacobi Preconditioning (TI) Message-Passing Interface (VLC) Can use all types of grids including quadrilateral and triangular adaptive grids (GG)

Hydrostatic Primitive Equations (Mass) (Momentum) (Energy) (State) (Tracers such as Moisture/Salinity)

Primitive Spherical Form: Pole problem can be avoided by: –Using Gauss quadrature (as in spherical harmonics) –Creating a hole in the domain (as in gridpoint models) –Using Cartesian coordinates (NSEAM) Why Cartesian Coordinates?

Primitive Equations: Write Primitive Equations as: Weak Problem Statement: Find –such that –where SE Method

Polynomial space for quads and tris Polynomial Approximation on the Quadrilateral: –With polynomial space Polynomial Approximation on the Triangle: –With polynomial space SE Method: Basis Functions

Accuracy of SE Method (Shallow Water on Sphere)

Icosahedral Telescoping Hexahedral Icosahedral Lat-Lon Adaptive Grids Lat-Lon Thin/Reduced

For the system: There are 2 classes of possible time-integrators: –Eulerian - fixed reference frame –Lagrangian - moving reference frame There are 2 subclasses of these methods: –Explicit – strict time-step restriction –Semi-Implicit – lenient time-step restriction –Fully Implicit – no time-step restriction Time-Integrators

Start with: Discretize by 2nd order TI –Everything on the RHS is known. Very fast to solve because only requires the evaluation of a vector of length 3*Np*Nlev (detail of this approach can be found in MWR 2004 paper) Explicit Time-Integrator

Start with: Discretize by 2 nd order TI –A global sparse NONLINEAR system must be solved. This can be quite expensive! The size of the system is (3*Np*Nlev) x (3*Np*Nlev) Fully-Implicit Time-Integrator

Start with: Rewrite as: Discretize by 2 nd order TI –A global sparse LINEAR system must be solved. Via a vertical mode decomposition we can do this by solving: Nlev systems of Np x Np sparse linear systems. This is very fast! (details of this approach can be found in QJRMS 2005 paper). Semi-Implicit Time-Integrator

Performance for T239 L30 (DT=300 seconds on IBM SP4) (Spectral Model) (SE Model)

Performance of NSEAM (216 Processors on IBM SP4) T159 L24 T239 L30 T580 L60

NOGAPS T80 L24 Rossby-Haurwitz Wave 4 (T=5 days)

NSEAM T80 L24 Rossby-Haurwitz Wave 4 (T=5 days)

ECMWF T106 L30 Held-Suarez Test Case (T=1200 days)

NSEAM T64 L20 Held-Suarez Test Case (T=1200 days) Zonal Wind (m/s) Temperature (K)

Comparison of the explicit (dashed line) and semi-implicit (solid line) NSEAM. The semi-implicit uses a time-step 8 times larger than the explicit. The difference between the two are virtually indistinguishable which is what one would hope to see. Jablonowski-Williamson Balanced Initial State (Surface Pressure Error for NSEAM T185 L26)

-90 Meridian0 Meridian The terrain field for NSEAM T239 from two different views of the globe. In this test an initially-balanced still atmosphere is integrated for 30 days. The atmosphere remained perfectly balanced for the length of the integration which is exactly what should happen. Initially-Balanced Still Atmosphere (NSEAM T239 L30 with terrain after 30 days)

No Terrain Terrain The viewpoint of these plots is from the –120 meridian. The temperature contours look similar for both but the effects of the Rockies, Andes, and Antarctica are clearly visible on the right panel. Held-Suarez Test Case (Surface Temp for NSEAM T185 L24 after 30 days)

Pressure Temperature U Velocity V Velocity Jablonowski-Williamson Baroclinic Instability (Surface Values at Day 9 for NSEAM T185 L26)

Jablonowski-Williamson Baroclinic Instability

Jablonowski-Williamson Baroclinic Instability (Surface Values for NSEAM T185 L26 during 0-30 days) Pressure Temperature

Future Work Further testing of NSEAM with terrain Physical parameterization (sub-grid scale processes) Non-hydrostatic equations (to better handle dx < 10 km) Non-hydrostatic Coastal Ocean Model (underway)