March 12, 2008 A Parallel Algorithm for Optimization-Based Smoothing of Unstructured 3-D Meshes by Vincent C. Betro

Outline Introduction Objectives Serial Optimization-Based Smoothing Parallel Implementation Results Conclusions Future Work

Introduction Parallel Hierarchical Unstructured Cartesian Mesh Generation Using P_HUGG, developed with Steve L. Karman, Jr. as part of an NRA with NASA Langley Uses hexahedral, Cartesian elements Body conforming through general cutting, yielding the four basic element types or polyhedral elements Can cause small, high aspect ratio elements to be formed Allows user to determine refinement based on geometry spacing

Introduction Why do we need optimization-based smoothing? < BEFORE AFTER >

Introduction Terminology Ghost nodes: Nodes that exist and will be worked on by a neighboring processor but that are part of elements on the given processor. They allow information about movement to be shared with the neighboring processors fluidly and yet only allow one processor to move them in the smoothing process in order to preserve uniqueness. Threshold: User-defined value that determines the level of the cost function (between 0 and 1 for non-inverted elements) that will be allowed to exist as a maximum in the final mesh. Usual values for a very smooth mesh are between 0.1 and 0.3. Also, low thresholds cause most, if not all, nodes in the mesh to be worked on, making the METIS partitioning more accurate and relevant.

Objectives To determine that one can smooth a mesh in parallel using ghost nodes on processor borders and obtain the same results in a comparable number of iterations / time To determine if weighting based on nodes being on the geometric boundaries (needing to be re-projected), nodes being completely internal (being perturbed in more directions), or no weighting at all would yield the most benefit in determining the METIS partition To determine if the algorithm would only allow larger meshes to be smoothed or if it would also garner speed-up on smaller meshes, if only to a point

Serial Optimization-Based Mesh Smoothing In order to remove high aspect ratio elements (sliver cells) and get improved results from the flow solver, optimization-based smoothing is performed on the mesh. Each node is perturbed based on a cost function calculated using Jacobians and condition numbers of the surrounding elements.

Serial Optimization-Based Mesh Smoothing First, the surrounding triangles’ cost function is computed: Then, the node’s cost function is computed using the max and average: The blending function is used to avoid the average cost being overshadowed:

Serial Optimization-Based Mesh Smoothing If the perturbation improves the cost function for the node, the node is moved permanently to the new position –Along direction of opposite face normal –Along direction of included edge –Along cardinal x, y, or z coordinate axis Note: A node is always perturbed by a pre-determined epsilon, based on a fraction of the length of the smallest edge

Serial Optimization-Based Mesh Smoothing The mesh is moved until: –eventually all perturbations cannot improve the cost function or –the threshold value supplied by the user is obtained or –the maximum number of perturbations supplied by the user is attained. Experience will tell the user when to stop working on the boundaries and/or internal points and/or change threshold values or maximum iterations.

Parallel Implementation Using the METIS libraries, one can partition the mesh to be distributed to multiple processors on which the optimization-based smoothing can then be carried out Available at METIS finds optimal configuration to assist in load balancing and decrease surface area for communication Using OPENMPI for the message-passing protocol

Parallel Implementation Similar implementation except: –Ghost nodes processed first to allow bias to propagate as normal –Communication of changes required between each iteration using processor-index triple Processor index Owning processor local index Global index –Global communication to discern global threshold, action taken based on global value; processors already attaining value continue to improve smoothness and adjust for neighbors who have not attained threshold of cost function

Results: DTMB 5415 Surface Ship Geometry extracted from existing tetrahedral mesh. 382,481 nodes 904,452 elements –391,828 tetrahedra –187,274 hexahedra –325,350 pyramids 1 to 64 processors on Dell Linux Cluster –325 nodes, 1300 processors (4 per node) –4 Gbytes per node –Dual-core Intel EM64T 3.0 GHz Xeon

Results: DTMB 5415 Surface Ship Weighting proved not to assist in time used or smoothness of final mesh Due to similar numbers of different operations needing done whichever type the node was Also, the connectivity of each node and type of opposite face come to bear on workload. Cases run with high thresholds tended to benefit less from the parallelization and the addition of extra processors.

Results: DTMB 5415 Surface Ship

Results: DPW III Surface geometry created in Gridgen –High-aspect ratio quads on wing converted to triangles 1,739,455 points 4,382,046 elements –2,326,329 tetrahedra –899,786 hexahedra –1,155,931 pyramids 1 to 64 processors on Dell Linux Cluster –325 nodes, 1300 processors (4 per node) –4 Gbytes per node –Dual-core Intel EM64T 3.0 GHz Xeon Surface definition Symmetry plane mesh

Results: DPW III Weighting proved not to assist in time used or smoothness of final mesh Due to similar numbers of different operations needing done whichever type the node was Also, the connectivity of each node and type of opposite face come to bear on workload. Cases run with high thresholds tended to benefit less from the parallelization and the addition of extra processors.

Results: DPW III

DPW volume mesh before and after optimization-based smoothing is applied to airfoil. < BEFORE AFTER >

Results: DPW III DPW volume mesh after extra, internal-only optimization-based smoothing is applied to the airfoil with threshold of 0.01 for 50 iterations.

Conclusions An effective optimization-based smoothing algorithm can be easily parallelized using OPENMPI Using low threshold values on both large and small meshes showed the most gains due to the fact that nearly all nodes in the mesh are being worked on the whole time to improve the cost function Large threshold values showed some gains with large meshes but generally showed no improvement past four processors for small meshes Moreover, regardless of the threshold values, very large meshes that could not fit on a serial machine were smoothed, such as one based on USGS data for Yosemite National Park Weighting boundary and non-boundary nodes differently has almost no affect on speed-up or smoothness, and can actually adversely affect both of the above The bias that might be attributable to ghost nodes was successfully damped by smoothing them first

Future Work Add in ParMETIS functionality to allow for pre-partitioned mesh files, which allows the decomposition step to be skipped and greater load balancing and decreased surface area to be obtained through the use of ParMETIS Allow the smoothing to work on general polyhedral elements, not only the four basic element types, by creating a cost function that can handle the corners of any type of polyhderal element Use this simpler optimization as a template for parallelizing linear-elastic smoothing and viscous layer insertion codes Use optimization-based and/or linear-elastic smoothing in conjunction with solution and feature based adaptation in parallel using the octree-based P_HUGG to re-mesh between solution steps for moving, body-conforming, viscous meshes on the fly without intermediate processing