Terascale Simulation Tools and Technologies Center Jim Glimm (BNL/SB), Center Director David Brown (LLNL), Co-PI Ed D’Azevedo (ORNL), Co-PI Joe Flaherty (RPI), Co-PI Lori Freitag (ANL), Co-PI Patrick Knupp (SNL), Co-PI Mark Shephard (RPI), Co-PI Harold Trease (PNNL), Co-PI

TSTT-2 TSTT will develop interoperable meshing and discretization technology components n Meshing and Discretization Research and Development l high-quality, hybrid mesh generation for complex domains l adaptive mesh update procedures l high-order discretization techniques l algorithms for terascale computing n Software interoperability is a pervading theme l initial design will account for interoperability at all levels l encapsulate research into software components l define interfaces for plug-and-play experimentation n Application deployment and testing is paramount l SciDAC collaborations in accelerator design, fusion, climate and chemically reacting flows l existing DOE application collaborations in biology, mixing fluids, and many more

TSTT-3 Existing Tools for Mesh Generation A wide variety of tools exist for the generation of … l … structured meshes q Overture - high quality predominantly structured meshes on complex CAD geometries (LLNL) q Variational and Elliptic Grid Generators (ORNL, SNL) l … unstructured meshes q MEGA (RPI) - primarily tetrahedral meshes, boundary layer mesh generation, curved elements, AMR q CUBIT (SNL) - primarily hexahedral meshes, automatic decomposition tools, common geometry module q NWGrid (PNNL) - hybrid meshes using combined Delaunay, AMR and block structured algorithms These tools all meet particular needs, but they do not interoperate to form hybrid, composite meshes MEGA Boundary Layer Mesh (RPI) Overture Diesel Engine Mesh (LLNL)

TSTT-4 Geometric Hierarchy Required to l provide a common frame of reference for all tools l facilitate multilevel solvers l facilitate transfer of information in discretizations n Level 0: Original problem specification via high level geometric description n Level 1/2: Decomposition into subdomains and mesh components that refer back to Level 0 n Level 3: Partitioning Given Geometry Specification Domain Decomposition Mesh Components Parallel Decomposition Level 3 Level 0 Level 2 Level 1 P0 P2 P3 P1P4 P6 P7 P5 P8 Pa Pb P9 Pc Pe Pf Pd

TSTT-5 Mesh Data Hierarchy n Level A: Geometric description of the domain l Accessed via tools such as CGM (SNL) or functional interfaces to solid modeling kernels (RPI) n Level B: Full geometry hybrid meshes l mesh components l communication mechanisms that link them (key new research area) l allows structured and unstructured meshes to be combined in a single computation n Level C: Mesh Components Geometry Information (Level A) Full Geometry Meshes (Level B) Mesh Components (Level C)

TSTT-6 Access to Mesh Data Hierarchy... n … as a single object (high-level common interfaces) l TSTT will develop functions that provide, e.g., q PDE discretization operators q adaptive mesh refinement q multilevel data transfer l Prototype provided by Overture and Trellis frameworks l Enables rapid development of new mesh-based applications n … through the mesh components (low-level common interfaces) l TSTT will provide, e.g., q element-by-element access to mesh components q callable routines that return interpolation coefficients at a single point (or array of points) l Facilitates incorporation into existing applications

TSTT-7 Common Interface Specification n Initially focus on low level access to static mesh components (Level C) l Data: mesh geometry, topology, field data l Efficiency though q Access patterns appropriate for each mesh type q Caching strategies and agglomerated access l Appropriateness through working with q Application scientists q TOPS and CCA SciDAC ISICs l Application scientists program to the common interface and can than use any conforming tool without changing their code n High level interfaces l to entire grid hierarchy which allows interoperable meshing by creating a common view of geometry l mesh adaptation including error estimators and curved elements n All TSTT tools will be interface compliant

TSTT-8 Mesh Data Hierarchy Construction n Level 0 to Level 1 geometry l Leverage existing TSTT tools that provide graphical interfaces to decompose the initial geometry into subdomains q CGM (SNL), Overture (LLNL) n Level 1 mesh components l Leverage existing mesh generation tools n Level C to Level B hybrid meshes l Stitching algorithms l Overlapping meshes Start with a set of component meshes... … Cut holes... … Stitch together to form a hybrid mesh Overture Stitching Algorithm (LLNL) CUBIT Geometry Decomposition (SNL)

TSTT-9 Enhancing Mesh Generation Capabilities n Will leverage most existing TSTT technology “as is” n Provisions for l Creating interface compliant tools l Improving mesh generation capabilities on complex geometries for high order elements q Curvilinear elements q Geometry approximations l Interoperability of appropriate tools q e.g., ORNL elliptic and variational mesh generators with Overture l Mesh quality control for hybrid meshes Linear coarse elements verses high-order, curvilinear P elements in MEGA (RPI)

TSTT-10 Mesh Quality Control n Unstructured mesh quality research and development is provided by MESQUITE (SNL, ANL) l optimization-based smoothing l reconnection schemes l development of quality metrics for high order methods l a posteriori quality control using error estimators n PDE-solution based mesh optimization will be investigated for overlapping and hybrid meshes Improvedmesh 8x error reduction by selecting optimal mesh generation parameters

TSTT-11 Dynamic Mesh Evolution n Geometry evolves due to l Adaptive mesh refinement l Internally tracked interfaces (e.g., shocks) l Motion of the domain boundary MEGA Rayleigh-Taylor Simulation (RPI) Overture simulation of Hele-Shaw flow

TSTT-12 TSTT Research in Mesh Evolution n Requires evolution of both the hierarchy and the individual mesh components n TSTT will provide interfaces that allow l the mesh tools to access the changing geometry l the application programmer to access the changing mesh l local or global modifications n New techniques will address l Curvilinear geometries to preserve convergence rates of high order discretizations l abstraction of adaptive techniques to provide “plug and play” l adaptive techniques that use multiple criteria to extend applicability l automatic selection and application of optimal strategies

TSTT-13 Combining TSTT technologies will improve front tracking techniques FronTier interface representation n n Improve conservation properties and accuracy at the front by inserting a surface determined by front tracking into a volume mesh n n Results in a front-adaptive space-time discretization

TSTT-14 TSTT will ease the use of high order discretization methods n Observation: Complexities of using high-order methods on adaptively evolving grids has hampered their widespread use l Tedious low level dependence on grid infrastructure l A source of subtle bugs during development l Bottleneck to interoperability of applications with different discretization strategies l Difficult to implement in general way while maintaining optimal performance n Result has been a use of sub-optimal strategies or lengthy implementation periods n TSTT Goal: to eliminate these barriers by developing a Discretization Library

TSTT-15 The Discretization Library Will... n … contain numerous mathematical operators l Start with +, -, *, /, interpolation, prologation l Move to div, grad, curl, etc. l Both strong and weak (variational) forms of operators when applicable n … contain numerous discretization strategies l Finite Difference, Finite Volume, Finite Element, Discontinuous Galerkin, Spectral Element, Partition of Unity l Emphasize high-order and variable-order methods l various boundary condition operators n … be independent of the underlying mesh infrastructure l Utilizes the common low-level mesh interfaces l All TSTT mesh tools will be available n … be extensible to allow user-defined operators and boundary conditions

TSTT-16 Additional Functionalities n Support for Temporal Discretization l Method of Lines formulation (time steps and temporal methods are spatially independent) l Local Refinement Methods (time steps and methods vary in space) l Space-Time techniques (unstructured meshes are used in both space and time) n Support for Adaptive Methods l Error estimators q Richardson’s extrapolation (meshes of different resolution) q P-refinement estimators q Solution gradient and vorticity metrics l Optimal strategies for mesh enrichment (combinations of p- and h- adaptivity) l Combined with work on mesh quality improvement n Support for Interpolation l Between meshes and operators l Local conservation when mapping between meshes

TSTT-17 Performance of Discretization Library n Kernel operations imply good performance is critical n Single Processor Performance l Compile time optimization of user-defined high level abstractions via ROSE (LLNL) l Consider hierarchical memory performance and cache usage n Terascale Computing l scalability of local operations requires good partitioning strategies l Efficiency determined by the size of the partition boundary relative to the partition volume n Will leverage the experience of l LLNL’s Overture project that supports structured mesh topologies l RPI’s Trellis project for variational discretization

TSTT-18 Benefits of the Discretization Library n Lowers the time, cost, and effort to effectively deploy modern discretization tools l High-level access for new application development on TSTT Level B meshes l Mid- and low-level access for insertion into existing technology n Increases reliability of application codes by eliminating a common source of coding errors n Enhances software reuse n Permits easy experimentation with various combinations of discretization strategies and mesh technologies for a given application

TSTT-19 Issues in Terascale Computing n Observation: Many tools exist that utilize hierarchical design principles to achieve good performance at the terascale l e.g., multi-level partitioners, multigrid solvers, multiresolution visualization tools n Barrier: Their union is not optimized l often difficult to take advantage of the multiresolution representations from one solution stage to the next n TSTT Goal: To design our hierarchy and tools so that downstream tools can take advantage of the multi- resolution information l Actively consider trade-offs across the entire simulation l Allow preservation of information as desired q e.g., subdomain decompositions used in creating a hybrid mesh may be similarly useful in preconditioning of iterative solvers

TSTT-20 Load Balancing n Use existing tools for partitioning l Chaco and Metis for static partitioning l Zoltan library (SNL) for dynamic partitioning n Develop and provide interfaces from TSTT software to Zoltan to ensure seamless operation n Augment Zoltan l Research methods to accommodate hierarchical machine models and heterogeneous parallel computers q different processor speeds, memory capacities, cache structures, networking speeds q RPM (RPI) and PADRE (LLNL) serve as prototypes l Load balancing strategies for adaptive, structured, overlapping grids q MLB (LLNL) serves as a prototype

TSTT-21 TSTT / TOPS Interactions n Interactions of mesh adaptation and solvers l The mesh and resulting equations to be solved evolve as solution process proceeds l Not effective to re-construct system from scratch on each mesh up-date l Can easily describe a good starting solution on most adaptively refined meshes l Critical to determine how to optimize the entire adaptive solution process - sub-optimization of specific components may yield poor overall efficiency l Consistent control of parallel partitions critical - adaptive methods employ dynamic load balancing l Absolutely must avoid “serial bottlenecks”

TSTT-22 TSTT / TOPS Interactions n Interaction of mesh adaptations and multilevel solvers l Mesh-based coarseners l Defining adaptive multiscale levels for solution process l Taking advantage of the natural hierarchies available with p- version adaptivity n Dealing with high-order, mixed and other equation discretization complexities l High order discretizations have different numerical conditioning and sparseness patterns l Mixed methods have conditioning issues l Mixed methods have natural partitioning that can be important to dealing with conditioning issues n Temporal adaptivity l Adaptive control of local time steps critical to many methods

TSTT-23 TSTT / TOPS Interactions Contact Information n Joe Flaherty l Rensselaer Polytechnic Institute l n Mark Shephard l Rensselaer Polytechnic Institute l n Lori Freitag l Argonne National Laboratory l