1. SCD User Forum 2005 Dynamically adaptive geophysical fluid dynamics simulation using GASpAR: Geophysics/Astrophysics Spectral-element Adaptive Refinement Duane Rosenberg, Aimé Fournier, Paul Fischer, Annick Pouquet NCAR Institute for Mathematics Applied to Geosciences Turbulence Numerics Team

2. SCD User Forum 2005 Motivation: Methodology Requirements Variable (e.g., high) order method (Patera 1984) Must be non-dissipative for DNS Must also model small scales, high Re, complex boundaries Must support variety of equations, physical models, geometries  code extensibility Leverage existing NCAR expertise in science applications, methods and performance

3. SCD User Forum 2005 GASpAR Features Designed for turbulence studies Object oriented ( C++ ) –extensive use of inheritance, polymorphism, templates Hierarchical: Elements  Fields  Eq. solvers  SEM operators  Bases  GBLAS Adaptive Mesh (currently 2D unstructured) Parallelized

4. SCD User Forum 2005

5. Spectral element numerics Based on spectral-element operators Typically, use operator-vector products  basic matrix-matrix products (m_x_m)

6. SCD User Forum 2005 Handling Nonconformity SCD User Forum 2005

7. Connectivity uses “mortars” (color matched)

8. SCD User Forum 2005 Locating neighbors and constructing mortars: VDB ( Henderson 1993 )

9. SCD User Forum 2005 ‘Mortar’ Data Exchange Mortar nodes given unique Morton keys (Warren & Salmon 1993) Bin-sort keys and create node/processor map Do DSS

10. SCD User Forum 2005 Adaptation Rules Isotropic (for now) Refinement: –At most 2 child edges (4 child faces in 3D) –Edge bisection –h-refinement only (currently) –Max. number of refinement levels Coarsening: –Cannot coarsen root –All siblings must be tagged –Must not violate Refinement rules

11. SCD User Forum 2005 Refinement & Coarsening Criteria a-posteriori spectral estimator of Mavriplis (1990) gradient 2 magnitude open question for turbulent flows (esp. in various formulations of the equations)

12. SCD User Forum 2005

13. Development/Porting Strategy Initial development under Linux using gcc Test MPI using MPICH with 1 or 2 procs Locate bottlenecks with port to Lightning (use PG compilers with -Minfo=loop,inline) --Find C m_x_m will not vectorize, but Fortran will! --Factor of 10 speedup in m_x_m! Port to IBM SPs (VAC++, f77) --Use gprof, HPM --Debug with totalview and printf’s

14. SCD User Forum 2005 Advecting Gaussian ‘Lump’

15. SCD User Forum 2005 Advecting Gaussian ‘Lump’ Mesh

16. SCD User Forum 2005 Radial N-wave Solution for Nonlinear Burgers Eqn.

17. SCD User Forum 2005 Rotated Burgers Front

18. SCD User Forum 2005 Dynamical Systems: Large Scale Passive Advection of Small Scale Field

19. SCD User Forum 2005 Conclusion 2D adaptive algorithm is working  Must include dynamic load balancing/processor distribution. Complete 3D AMR, enhance performance based on 2D; debugging difficult Better performance monitoring/reporting on the SPs?

20. SCD User Forum 2005 Nonconforming Connectivity: non-Boolean gather operator Q maps global (j=0:18) to local (i=0:26) d.o.f. (after Fischer & Kruse 1997) Q 12,2 Qi,jQi,j

21. SCD User Forum 2005