1.  Jean-François Remacle,  Joe E. Flaherty and Mark S. Shephard  Rensselaer Polytechnic Institute  remacle@scorec.rpi.edu Parallel Algorithm Oriented Mesh Datastructure  Outline Basics of Mesh Representation Parallel Extensions Software issues Examples n http://www.scorec.rpi.edu/AOMD

2. AOMD-PAOMD  AOMD and PAOMD deals with meshes A core for basic mesh representation (topological adjacencies) Advanced design patterns : Iterator, Observer, Visitor... Some extensions Parallel services : message passing, load balancing, partitioning Meshing Toolbox : quality measures, mesh modifications, cavity mesher,… Calculus : coordinate systems, integration <>  AOMD supports Geometry based analysis design, classification Parasolid, ProE, STL (not open source yet) Hybrid meshes Hexes, Tets, Quads … Non conforming meshes (hanging nodes, AMR) Curved meshes (Bézier)  AOMD is an Open Source Project

3.  Advanced analysis techniques Are automated with automatic mesh generation from geometric models Employ variable order elements based on other than Lagrange basis Use various weak forms that required alternative mesh relationships Adaptive the mesh as the simulation proceeds  To meet these needs the mesh data structure must Understand the relationship of mesh entities with the geometric model - ensure mesh validity and associate physical parameters with the mesh Understand the interactions of various mesh entities - different relationships used during automatic mesh generation and analysis Support assignment of independent geometry to the entities in the mesh - must control geometric approximation with higher order methods Be able to associate dof to various mesh entities - provides flexibility to support variable p-order and different collections of dofs Effectively maintain relationships during modification - needed in mesh generation, mesh adaptation and mesh modification for evolving geometry Motivations

4. Automated Adaptive Analysis  FEM PUM AOMD-MeshSim Trellis AOMD AOMD-MeshSim

5. Topological Mesh Data Structure  Classic node point coordinates / element connectivities do not meet this need  Mesh representations based on topological entities and their adjacencies fill the need: Can be proven to be complete and unique - can effectively support all relationships and associations needed by any mesh generation or analysis procedure Provide a shape independent abstraction for associating geometry Effectively supports the linkage to the geometric model since systems maintain a model topology  Various approaches to support mesh topology have been taken  A new approach considered here curved mesh for p-version analysis

6. Modern Geometric Modeling Systems Employ non-manifold boundary representation Provide access to model and its geometry through a geometric modeling kernel driven by topological entities Simulation processes (mesh generation, p-version analysis...) can directly interact with the modeler 

7. Basics of mesh representation  A geometrical domain G Is the highest level representation of the domain is composed of geometrical entities G i d  A Mesh M is a discrete representation of G is composed of mesh entities M i d, i=1,… N d (M) together with their adjacencies M i d {M q }  Mesh entities M i d 4 different topological kinds Vertices (d=0), edges (d=1), faces (d=2) and regions (d=3) The unique association of a mesh entity, M i d i, to a geometric model entity, G j d j, where d i <d j is denoted by  M i d i G j d j

8. Adjacencies sets Downward Adjacenties Upward Adjacenties One-Level Circular Face to Edge Examples of complete adjacencies sets IncompleteComplete

9. Cost of a mesh representation

10. Memory cost of some mesh representation

11. Higher order adjacencies  First order adjacencies Direct access, unit cost Full representation : only way to have all adjacencies as first order  Higher order adjacencies sets Regions know faces : M i 3 {M 2 } Faces know edges : M i 2 {M 1 } Edges know vertices : M i 2 {M 1 } Third order adjacencies sets : M i 3 {M 2 } {M 1 } {M 0 }

12. Functionally complete representation  Minimum information Equally dimension classified entities must be present All vertices, all regions, all edges classified on model edges and all faces classified on model faces This is a sufficient minimum, not necessary but this choice allow to complete the representation without geometrical checks

13. Basics of the Algorithm Oriented Mesh Database  Mesh entity description A mesh entity described by a set of lower dimension entities : M i d {M q }, d > q All vertices are always required Vertices are atomic mesh entities, must be differentiated (e.g., using iD (M i 0 )  Mesh entities comparison Two entities are equal if their set of vertices are equal allows to compare mesh entities (,=) Not absolutely general but key to practical implementation

14. Downward adjacencies ordering : templates  Entity described using their boundaries unique description i.e. non ambiguous shape  Weaker hypothesis Need for ordering, templates Used for computing uses n T ev and T fe Same vertices but different entities Invert templates

15. Mesh Entity iD  Need of search in AOMD Add, search and remove operations are crucial Comparing entities is always possible but... std::set log behavior is not acceptable  Hash tables n Elements in a hash table not sorted n complexity : worst is linear, average is constant std::hash_set Hash function needed, deterministic and stateless, a mesh entity iD iD (M 1 ) = iD (M 2 )  M 1 = M 2 true iD (M 1 ) = iD (M 2 )  M 1 = M 2 false iD is a function of vertices for being independent of the representation iD (M 1 ) = iD (M 2 ) and M 1  M 2 should not happen too often, efficiency of the hash table because equalEntity is to be used in this case

16. Choice if the iD  Efficiency N e is the number of elements N k is the number of keys

17.  Stokes problem, tet mesh Number of dofs, use previous statistics  Counting of degrees of freedom (Szabo basis) Higher order finite elements

18. Basics of the Parallel AOMD  Basics of parallel AOMD Partition boundaries treated like model boundaries Equal order mesh entities must exist on partition boundaries (partition faces, edges and vertices) Mesh vertices must have a unique global label On processor : serial AOMD  Implementation aspects Simplicity, no master, no owner Round of communication standardized, no MPI calls visible, messages automatically packed

19. Parallel AOMD - Mesh Adaptation  Target is transient applications with thousands of mesh adaptation steps Want fast and simple adaptation Need efficient interprocessor communications  Mesh Refinement Apply templates Include support of non-conforming meshes Refined entities with remote copies must be split on all partitions Round of communication needed to ensure unique vertex ID’s  Mesh Coarsening (will be same for local mesh modifications) Collect all mesh entities involved onto one partition Carry out operation using serial operators on processor

20. Dynamic Load Balancing and Mesh Migration  Need dynamic load balancing after mesh adaptation Procedures build on balancing procedures in Zoltan (from Sandia) PAOMD used to provide Zoltan needed entities and connections Load balancing procedure indicates which mesh entities are to be migrated to which processor PAOMD only migrates minimum set, unless user specifically asks to migrate other entities classification after load balancing and before migration configuration after migration

21. Mesh Migration  Steps in process Collect the mesh entities to be migrated to another partition Determine needed higher order mesh entities to be migrated (use AOMD to determine minimal set needed) Collect entities and any user attached data Perform communications to send entities and update links (following methods of the Rensselaer Partition Model (RPM))  Message passing At PAOMD operator level it appears messages are sent one at a time This would lead to unacceptable communication costs Message packing used - AUTOPACK (from Argonne) Automatically controls message packing process Includes information and tools to optimize message size for network architecture used  Communications costs In the examples that follow communications were on the order of 1% of the total costs

22. Implementation issues Orientation of entities computed on the fly  Language C++ and generic programming STL, significant new feature of the language Programming with concepts C++ and OO programming generic and OO are complementary Trade off efficiency vs. flexibility We believe not Templates, functors… generic programming is efficient Classical example, quick sort  stl::sort is 4 times faster (with VC6) than C qsort  Parallel Autopack, automatic message packing Zoltan, dynamic load balancing and partitioning STL, associative containers, algorithms

23. Mesh refinement  class AOMD_RefCallback  { public :  virtual int operator () (const meshEntity *) const = 0;  virtual void callback (std::list &before,  std::list &after  ) const = 0;  }; Conformal or not (hanging nodes or mixed meshes) Typically  class myAOMD_RefCallback : public AOMD_RefCallback;  AOMD:: RefUnref(theMesh, myAOMD_RefCallback);

24. Communications  class AOMD_RoundOfComm  { public :  virtual char * sendBuffer (const meshEntity *,  int dest_proc,  size_t&sizebuf) const = 0;  virtual char * recvBuffer (const meshEntity *,  int src_proc,  size_tsizebuf) const = 0;  }; Messages are packed (autopack) Typically  class myAOMD_RoundOfComm : public AOMD_RoundOfComm;  AOMD::roundOfComm(theMesh, myAOMD_RoundOfComm);

25. Load balancing  class AOMD_LBCallback  { public :  virtual char * sendBuffer (const meshEntity *,  int dest_proc,  size_t&sizebuf) const = 0;  virtual char * recvBuffer (const meshEntity *,  int src_proc,  size_tsizebuf) const = 0;  }; Messages are packed (autopack) Typically  class myAOMD_LBCallback : public AOMD_LBCallback;  AOMD::LB(theMesh, myAOMD_LBCallback);

26. Demonstration of Load Balancing  Refined to moving level function - non-conforming triangular mesh

27. http://www.scorec.rpi.edu/DG  Desing specifications Solve any conservation law, in any dimension, in parallel and adaptively, using any system of curvilinear coordinates, any discretization, any spatial basis and any time stepping scheme class ConservationLaw : fluxes, numerical fluxes f n and g, right hand side r, equation of state, initial and boundary conditions : the physics class Integrator : specialization's for geometrical elements class Metric : Euclidian, axisymmetric class FunctionSpace : Orthogonal or not, class Solver : forward Euler, Runge-Kutta, matrix free GMRES.  Examples Navier-Stokes, Euler, Burgers, Maxwell-Boltzmann...

28. Demonstration of Load Balancing

29. 2-D Examples 2D Rayleigh Taylor Red (heavy), blue (light), On left, 4 levels of refinement, On right, 2 levels of refinement, Faster growth with more refinement as expected

30. 2-D Animation of Instability  Linear DG elements, 30,000 to 800,000 dof  Atwood Number, A = 1/3  10 fourier modes in “random” distribution  time for the bubble to reach top  of the window (y = 0.5) : 5 sec  This calculation: alpha = 0.06  Experiments: alpha = 0.058 - 0.065  Theory (Glimm, et al) alpha 0.045 = 0.06

31. Refined 3-D Meshes for Rayleigh Taylor Instability  24 steps of refinement 72 steps of refinement 104 steps of refinement heavy fluid light fluid non-conforming hexahedron mesh

32. Rayleigh Taylor Instability 64 processors of the PSC alpha cluster 1  10 6 to 2.0  10 7 dof’s 128 processors of Blue Horizon 10 8 dof’s

33. The cannon

34. Conclusions  PAOMD advantages Quite small piece of software, documented Focused, mesh management only Asks for minimum user knowledge about parallel issues Efficient implementation  Future work Terascale computers, more than 1000 processors (in progress, ASCI & SciDAC projects) Anisotropic mesh refinement (in progress, with X Li) TSTT Mesh component Hardware heterogeneity, machine and network models have to be added in partitioners (in progress) Modification of the design, storing less