1. TerascaleSimulation Tools and Technologies The TSTT Interface Definition Effort Lori Freitag Diachin Lawrence Livermore National Lab

2. 2 The TSTT Center Goal: To enable high-fidelity calculations based on multiple coupled physical processes and multiple physical scales –Adaptive methods –Composite or hybrid solution strategies –High-order discretization strategies Barrier: The lack of easy-to-use interoperable meshing, discretization, and adaptive tools requires too much software expertise by application scientists The TSTT center recognizes this gap and is addressing the technical and human barriers preventing use of adaptive, composite, hybrid methods

3. 3 TSTT brings a broad set of meshing- discretization communities together Diverse Partners –Overture –AOMD/Trellis –CUBIT –NWGrid/Phys – –Frontier –Mesquite –Nek5000 Meshing Technologies –Structured/Overset –Mixed element hybrid –Unstructured hex –Tetrahedral –Variational approaches –Mesh quality improvement Adaptive Technologies –Mesh refinement –Front tracking Software Frameworks and PDE Solvers Significant Applications Experience NWGrid mesh MEGA Boundary Layer Mesh (RPI) Overture Mesh (LLNL) CUBIT Mesh (Sandia)

4. 4 The Challenge These tools all meet particular needs, but –They do not interoperate to form hybrid, composite meshes –They cannot be easily interchanged in an application In general the technology requires too much software expertise from application scientists –Difficult to improve existing codes –Difficult to design and implement new codes Our goal is to develop interchangeable and interoperable software components for meshing and discretization

5. 5 TSTT interoperability goals require services build on a common infrastructure All tools must build on a common abstract framework and data model –Provides a single reference for PDE simulation data hierarchy Defining common interfaces is critical –Required for interchangeability of similar tools –Allows vertical integration of tools Applications write to the TSTT specification once and can use all TSTT services Geometry Information (Level A) Full Geometry Meshes (Level B) Mesh Components (Level C)

6. 6 TSTT data model abstracts PDE-simulation data hierarchy Core Data Types –Geometric Data: provides a high level description of the boundaries of the computational domain; e.g., CAD, image, or mesh data –Mesh Data: provides the geometric and topological information associated with the discrete representation of the domain –Field Data: provides access to the time dependent physics variables associated with application solution. These can be scalars, vectors, tensors, and associated with any mesh entity. Data Relation Managers –Provides control of the relationships among the core data types –Resolves cross references between entities in different groups –Provides functionality that depends on multiple core data types Each core data type has a TSTT interface Mesh: TSTTM Geometry: TSTTG Fields: TSTTF Each core data type has a TSTT interface Mesh: TSTTM Geometry: TSTTG Fields: TSTTF

7. 7 An overarching philosophy guides our interface definition efforts Maintain data structure neutrality Create a small set of interfaces that existing packages can support – Small set of ‘core’ functions that must be implemented – Larger set of functions supported by reference implementations Balance performance and flexibility Work with a large tool provider and application community to ensure applicability Target both specific uses of TSTT services and broad dissemination as CCA components Lowers the burden for adoption of the interface Performance is critical for kernel computations involving mesh access; flexibility is critical for covering a broad usage spectrum CCA provides infrastructure and guidance for domain-specific interface definition efforts

8. 8 The TSTT mesh data model consists of entities, entity sets and meshes Entity Definition –Unique type and topology –Canonical ordering defines adjacency relationships Entity Set Definition –Arbitrary grouping of TSTT mesh entities –There is a single “Root Set” –Relationships among entity sets Contained-in Hierarchical Mesh Definition –Simple meshes are a set of connected, non- overlapping entities related through adjacency relationships –Meshes can contain collections of simple meshes

9. 9 Core functions must be implemented to be TSTT compliant The smallest set of functions required to be considered TSTT compliant Provides basic access to vertex coordinates and adjacency information –Mesh loading and saving –Accessing global information such as the root set, geometric dimension, number of entities of a given type or topology –Primitive array access for entity sets –Global entity arrays for entity sets Reference implementations can be used to provide advanced capabilities needed for TSTT services –Requires a copy of mesh data –Allows incremental implementation of advanced functions

10. 10 Advanced traversal functions provide additional flexibility and performance Single Entity Traversal –Iterator functionality (init, next, reset, end) –Query (topology, type, adjacency) Workset Traversal –Block iterators allow access of entities in a user-defined array size –Array based query functions for topology, type, adjacency –Increases performance by reducing the number of calls through the interface –Interface is not as intuitive

11. 11 Advanced functionality is supported through optional interfaces Entity Sets –Creation/Destruction –Boolean Operations –Relations Tags –Allows the incorporate user-defined data in the TSTT data model –Associated with entities, meshes, and entity sets Mesh Modification –Simple mesh geometry and topology modification –Requires simple classification mechanisms against the geometric model –Intended to support higher-level functionality

12. 12 The geometry interface provides access to the computational domain Must support –automatic mesh generation –mesh adaptation –tracking domain changes –relating information between alternative discretizations Builds on boundary representations of geometry Used to support various underlying representations –Commercial modelers (e.g., Parasolid, ACIS) –Modelers that operate from standard files (e.g. IGES, STEP) –Models constructed from an input mesh

13. 13 Both basic and advanced functionalities are supported in the geometry interface Model loading and initiation Topological queries of entities and adjacencies Pointwise geometric shape interrogation Parametric coordinate systems Model topology modification Tags

14. 14 Relating mesh and geometry data is critical for advanced TSTT services Required for e.g., adaptive loops, mesh quality improvement Mesh/Geometry Classification Interface –Manages the relationship between the high level geometric description and the mesh –Called by an application that knows about both Capabilities –For a given mesh entity, get the geometric entity against which it is classified –Establish a classification relationship between a mesh entity and a geometric entity

15. 15 Preliminary interface implementations are well underway Working documents describing the mesh and geometry data models as well as the overall vision TSTT Mesh Interface (version 0.6) complete Preliminary interfaces for geometry and mesh/geometry classification defined Implementations – AOMD, FronTier, GRUMMP, MOAB, NWGrid, Overture for mesh – CGM for geometry C, C++, and Fortran language interoperability through SIDL/Babel (CCA) – Analyzing performance ramifications of SIDL/Babel language interoperability tools (joint with the CCA)

16. 16 Performance studies will provide ‘best practices’ guidelines Working with SIDL/Babel team to characterize the interface performance Focus on entity-based access –Single entity iterators –Workset iterators Compare –Native data structures –Native language interfaces –SIDL/Babel interfaces Workset Iterators Initialize Workset(entity_type, num_entities, &wrk_iter) While (!done){ GetNextWorkset(&entity_handles[ ], &num, wrk_iter); GetInfo(entity_handles); for i=0:num-1 { do_something(entity_handle[i]); } Destroy Workset(wrk_iter); Workset Iterators Initialize Workset(entity_type, num_entities, &wrk_iter) While (!done){ GetNextWorkset(&entity_handles[ ], &num, wrk_iter); GetInfo(entity_handles); for i=0:num-1 { do_something(entity_handle[i]); } Destroy Workset(wrk_iter);

17. 17 Preliminary performance focus on TSTT mesh access patterns Workset Iterators –Increased workset size amortizes overhead –15-40% compared to native data structures –SIDL a few percent more than native language –Single entity access via workset iterators 2X- 4X for native and SIDL interfaces Single entity iterators have similar performance to a workset size of 5-10 Array based access for NWGrid has.5% overhead independent of mesh size Ongoing research to further improve performance when interoperability is not critical (w/ PERC)

18. 18 The TSTT mesh interface working group uniquely engages a large number of people Internal –ANL: Fischer, Buschelman –LLNL: Freitag, Chand, Henshaw, Dahlgren –PNNL: Trease –RPI: Shephard, Seole –SNL: Tautges, Kim –SUNY SB: Li, Fix Internal –ANL: Fischer, Buschelman –LLNL: Freitag, Chand, Henshaw, Dahlgren –PNNL: Trease –RPI: Shephard, Seole –SNL: Tautges, Kim –SUNY SB: Li, Fix External (active) –UofC: Loy –UBC: Ollivier-Gooch External (active) –UofC: Loy –UBC: Ollivier-Gooch External (monitoring) –UIUC: Guoy –Boeing: Michal (UGC) –LLNL: Keasler –SNL: Pebay, Haynes, Edwards, Jung –ElemTech: Meyers –UU: Parker –ORNL: Bernholdt External (monitoring) –UIUC: Guoy –Boeing: Michal (UGC) –LLNL: Keasler –SNL: Pebay, Haynes, Edwards, Jung –ElemTech: Meyers –UU: Parker –ORNL: Bernholdt Managing the interactions across institutions Face to face meetings (all-hands and opportunistic) Telecons (approximately every three weeks) Email discussion group (39 participants from 13 institutions) Working documents and SIDL files CVS repository

19. 19 Defining the TSTT interfaces has been an interesting challenge Interface definition is harder than we anticipated Cannot achieve the 100 percent solution, so... –What level of functionality should be supported? –How to best support existing (different) packages? The devil is in the details –Initial interface completed a year ago –Lots of interesting issues Invariant handles Consistent error handling across packages Memory management issues Core interface functions Performance, performance, performance

20. 20 TSTT services support SciDAC applications developers Domains from multiple geometry source Use of alternative mesh generators Use alternative equation discretization methods and tools Application of adaptive control of the –Geometry - shape optimization –Mesh - discretization error control –Both - evolving geometry simulations Combining multiple analysis codes to solve multiphysics problems

21. 21 TSTT services “Current” TSTT Services –Mesh services libraries (AOMD, MOAB, NWGrid) –Mesh quality improvement (Mesquite) –Discretization libraries (Overture, Trellis) –Parallel mesh generation (MOAB) –Front tracking (FronTier) –Adaptive control loops (combination of tools) Services Under Development –Geometry interface functions (not doing geometry) –Fields library –Mesh control for shape optimization

22. 22 Mesquite Mesh Quality Improvement Stand alone library of mesh quality improvement algorithms –2D/3D unstructured and hybrid meshes –Triangles, Quads, Tets, Hex elements –State-of-the-art optimization-based improvement schemes –Joint work with TOPS created fastest mesh quality optimizer in Mesquite tool suite – improves 1M elements in 15 seconds TSTT interface to access mesh and geometry information –Vertex coordinates, element connectivity, tags –Successfully integrated with AOMD, MOAB, Overture, Cubit Application uses –Surface mesh smoothing on complex geometries for accelerator modeling –Mesh quality analysis for SLAC Tau3P code –Mesh deformation in design optimization problems –Anisotropic smoothing for error minimization

23. 23 TSTT developed a new interoperable AMR/Front tracking scheme Combine front tracking in Frontier with adaptive techniques in Overture –Increase problem resolution using AMR –Maintain exact interface representation capabilities Applications –Richtmeyer-Meshkov instability –Diesel jet spray break up Using TSTT interface to combine front-tracking schemes with NWgrid capabilities

24. 24 TSTT/SLAC collaboration provides adaptive mesh capabilities for Omega3P Omega3P is SLAC’s quadratic FEM eigensolver for computing normal modes in RF cavities Extremely accurate solutions (0.01% error) are required for requirements of accelerator design Using new TSTT technology developed at RPI, adaptive mesh capability has been provided for Omega3P Simulations achieve same accuracy with 1/3 the number of unknowns 3 levels of refinement First RefinementFinal Adaptive MeshInitial Mesh

25. 25 Using geometry operators means alternate solid modelers can be inserted Using TSTT mesh operators means alternate mesh generators and mesh adaptation procedures can be inserted Using TSTT field operators allows easy construction of alternative error estimators Components used in construction of adaptive loop for SLAC Projection-based error estimator used to construct new mesh size field given to mesh modification Mesh adaptation based on local modification linked directly to CAD Unaltered SLAC code Unaltered SLAC code Error estimators from RPI and SLAC

26. 26 Contact Information TSTT Web Site: www.tstt-scidac.org David Brown: dlb@llnl.gov Lori Freitag Diachin: diachin2@llnl.gov Jim Glimm: glimm@ams.sunysb.edu