1. Interoperable Mesh Tools for Petascale Applications Lori Diachin, LLNL Representing the ITAPS Team

2. 2 ITAPS focuses on interoperable meshing and geometry services for SciDAC ITAPS Goal – Improve SciDAC applications’ ability to take advantage of state-of-the-art meshing and geometry tools – Develop the next generation of meshing and geometry tools for petascale computing Technology Focus Areas – Complex geometry – High quality meshes and adaptivity – Coupled phenomenon – Dynamic mesh calculations – Tera/Petascale computing

3. 3 Accomplishing the ITAPS interoperability goal requires a strong team with diverse expertise Lori Diachin LLNL Ed d’Azevedo ORNL Jim Glimm BNL/SUNY SB Ahmed Khamayseh ORNL Bill Henshaw LLNL Pat Knupp SNL Xiaolin Li SUNY SB Roman Samulyak BNL Ken Jansen RPI Mark Shephard RPI Harold Trease PNNL Tim Tautges ANL Carl Ollivier-Gooch UBC Our senior personnel are experts in complex geometry tools, mesh generation, mesh quality improvement, front tracking, partitioning, mesh refinement, PDE solvers, and working with application scientists Our senior personnel are experts in complex geometry tools, mesh generation, mesh quality improvement, front tracking, partitioning, mesh refinement, PDE solvers, and working with application scientists Karen Devine SNL

4. 4 ITAPS will enable use of interoperable and interchangeable mesh and geometry tools Build on successes with SciDAC-1 applications and explore new opportunities with SciDAC-2 application teams Develop and deploy key mesh, geometry and field manipulation component services needed for petascale computing applications Develop advanced functionality integrated services to support SciDAC application needs – Combine component services together – Unify tools with common interfaces to enable interoperability Common Interfaces Component Tools Petascale Integrated Tools Build on Are unified by

5. 5 We have a suite of ITAPS services that are built on the ITAPS common interfaces MeshGeometryRelationsField Common Interfaces Component Tools Are unified by Petascale Integrated Tools Build on Mesh Improve Front tracking Mesh Adapt Interpolation Kernels Swapping Dynamic Services Geom/Mesh Services AMR Front tracking Shape Optimization Solution Adaptive Loop Solution Transfer Petascale Mesh Generation N N N N N N         P P P P P P   P P   P P    

6. 6 ITAPS technologies impact SciDAC applications in three ways 1.Direct use of ITAPS technology in applications – Geometry tools, mesh generation and optimization for accelerators and fusion – Mesh adaptivity for accelerators and fusion – Front tracking for astrophysics and groundwater – Partitioning techniques for accelerators and fusion 2.Technology advancement through demonstration and insertion of key new technology areas – Design optimization loop for accelerators (w/ TOPS) – Petascale mesh generation for accelerators 3.Enabling future applications with ITAPS services and interfaces – Parallel mesh-to-mesh transfer for multi-scale, multi- physics applications – Dynamic mesh services for adaptive computations Current ITAPS/TOPS/SLAC shape optimization activities will improve ILC design process

7. 7 Work with the COMPASS project spans many different ITAPS service areas Mesh Control – Correct curvilinear meshes that properly satisfy the geometric approximation requirements Adaptive Mesh Refinement Parallel Mesh Generation Embedded Boundary Discretization Partitioning Techniques Geometry Search and Sort ILC cryomodule consisting of 8 TDR cavities Beamline Absorber

8. 8 Mesh correction for curves required new technology developments Bezier higher order mesh shape – Analytical validity determination – Determine key mesh entity causing invalidity Curved mesh modifications for invalid elements – Reshape, split, collapse, swap and refinement Curved Edge Split to Fix Model Tangency Curved Region Split Reshape

9. 9 Results allowed simulations to run faster and further than before Initial mesh – 108k mesh regions – 250 invalid regions – Solution blows-up unless negative contribution removed Corrected mesh – No invalid regions – Solution process and 37.8% faster (CG iterations per time step reduced) Valid curved mesh after operations 2.97M regions by correcting 1,357 invalid curved regions

10. 10 Goals for curved meshes in thin sections – Thermal/Mechanical multiphysics simulations – Curved anisotropic meshes for thin sections for computational efficiency

11. 11 Technology Developments Automatically identify thin sections for complex geometry Construct curved anisotropic layer elements with proper order

12. 12 Initial results are promising; working with SLAC on use in simulation setting Straight-sided and curved anisotropic mesh for the cell model Close-up of the three curved layer elements on top of the model faces 2,664 tetrahedral regions To Do: Technically complex problem; may need fine- tuning in application setting

13. 13 New area: use moving mesh refinement regions to increase computational efficiency SciDAC codes - Pic3P/Pic2P – Isotropic refinement for defined particle domains – Coarse possible meshes for the rest domains – Pre-defined particle domain is time-dependent – Achieve computational efficiency and accuracy Technical development – Built on the mesh modification tool (does all the mesh level work) – Define moving mesh size field with blend PIC in long structures

14. 14 Early results show significant savings New Procedure using Adaptive Controlled Mesh - 1,030,121 Regions Compared to a Uniform Mesh - 4,880,593 Regions To Do: Iterate to determine best use Correct small number of poor quality elements

15. 15 Adapted mesh (23,082,517 tets) Initial mesh (1,595 tets) Parallel adaptive loop for SLAC accelerator design – Geometry defined in CAD modeler – Omega3P code from SLAC – High level modeling accuracy needed – Adaptive mesh control to provide accuracy needed – Adaptive loop runs in serial and parallel To Do: Initial results used file transfer; need to coordinate parallel I/O Ensure results are satisfactory

16. 16 Parallel adaptive loop for SLAC uses many different software components Using geometry operators means alternate solid modelers can be inserted Using ITAPS mesh operators means alternate mesh generators and mesh adaptation procedures can be inserted Using ITAPS field operators allows easy construction of alternative error estimators Projection-based error estimator used to construct new mesh size field given to mesh modification Mesh adaptation based on local modification linked directly to CAD Unaltered SLAC code Unaltered SLAC code Error estimators from RPI and SLAC

17. 17 Optimizing a cavity design is still mostly a manual process Future accelerators employ complex cavity shapes that require optimization to improve performance Geometry & meshing support: bl al a1 a2 b1 ar br b ra1 ra2 zcl zcr zcbzcc zcll Fixed mesh topology: Convergence No re-meshing Re-use factorization Shape Optimization for Accelerator Cavity Design Omega3P Sensitivity optimization ITAPS Geometry & Mesh Update Services Omega3P

18. 18 Geometry MeshUpdate ITAPS Geometry & Mesh Update Services Projection x r (d’)→ G(d’) δd, d’ x r (d) x Ω (d) x r (d’) Omega3P d : Design parameter vector d’ : d + δd, new design parameters G(d)↔x r (d) : Geometry-mesh classification G(d’) : Geometry for parameters d’ x r (d’), x Ω (d’) : Mesh for new iteration G(d’) Design Velocity ∂x r / ∂G, ∂n/∂x r ∂G/ ∂d n(x r (d’)) Omega3P Sensitivity x r (d) x r (d’) x Ω (d’)

19. 19 Services provided by ITAPS DDRIV tool Parameterized geometric model construction – You write function which constructs model using iGeom – DDRIV acts as driver and handles IO Coordination of mesh smoothing on geometric model Re-classification of “old” mesh on “new” model Target matrix-based smoothing of re-classified mesh Computation of design velocities & embedding on mesh using iMesh Tags Smooth Curves Smooth Volume Smooth Surfaces New geom, old mesh Project to CAD, inverted elements To Do: Incorporate with TOPS/SLAC optimization tools Parallelize all aspects

20. 20 Parallel Mesh Generation Needed for Petascale Simulations Modeling long range electro- magnetic effects in the ILC requires meshes 4X – 8X current meshes Status: – Have a prototype CAD-based parallel mesh generation capability – Demonstrated good scaling for SLAC NLC accelerator To Do: update to production capability – Adaptive size control – Link to mesh partitioning/load balancing

21. 21 Dynamic load balancing and partitioning via the Zoltan toolkit Reduce total execution time – Distributing work evenly among processors – Reducing applications’ interprocessor communication – Keeping data movement costs low Important in many SciDAC technologies – Adaptive mesh refinement – Particle-in-cell methods – Parallel remeshing – Linear solvers and preconditioners Adaptive Mesh Refinement Particle Simulations

22. 22 Contact Information ITAPS Web Site: www.itaps-scidac.orgwww.itaps-scidac.org Email: itaps-mgmt@lists.llnl.gov Lori Diachin: diachin2@llnl.gov We actively seek and welcome your input!

23. 23 LLNL Disclaimer and Auspices This document was prepared as an account of work sponsored by an agency of the United States Government. Neither the United States Government nor the University of California nor any of their employees, makes any warranty, express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial products, process, or service by trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government or the University of California. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government or the University of California, and shall not be used for advertising or product endorsement purposes. This work was performed under the auspices of the U.S. Department of Energy by University of California Lawrence Livermore National Laboratory under contract No. W-7405-Eng-48.