1. Adaptive Mesh Applications
Sathish Vadhiyar
Sources:
- Schloegel, Karypis, Kumar. Multilevel Diffusion Schemes for Repartitioning of Adaptive Meshes. JPDC 1997 (Taken verbatim)

2. Adaptive Applications
Highly adaptive and irregular applications
Amount of work per task can vary drastically throughout the execution – similar to earlier applications, but..
Has notions of “interesting” regions
Computations in the “interesting” regions of the domain larger than for other regions
It is difficult to predict which regions will become interesting

3. AMR Applications
An example of such applications is Parallel Adaptive Mesh Refinement (AMR) for multi-scale applications
Adaptive Mesh – Mesh or grid size is not fixed as in Laplace/Jacobi, but interesting regions are refined to form finer level grids/mesh
E.g.: to study crack growth through a macroscopic structure under stress

4. AMR Applications – Crack propagation
Such a system is subject to the laws of plasticity and elasticity and can be solved using finite element method
Crack growth forces the geometry of the domain to change
This in turn necessitates localized remeshing

5. AMR Applications- Adaptivity
Adaptivity arises when advances crosses from one subdomain to another
It is unknown in advance when or where the crack growth will take place and which subdomains will be affected
The computational complexity of a subdomain can increase dramatically due to greater levels of mesh refinement
Difficult to predict future workloads

6. Repartitioning
In adaptive meshes computation, areas of the mesh are selectively refined or derefined in order to accurately model the dynamic computation
Hence, repartitioning and redistributing the adapted mesh across processors is necessary

7. Repartitioning
The challenge is to keep the repartitioning cost to minimum limits
Similar problems to MD, GoL and Nbody
The primary difference in AMR is that loads can drastically change; cannot predict; will have to wait for refinement, then repartition

8. Structure of Parallel AMR

9. Repartitioning
2 methods for creating a new partitioning from an already distributed mesh that has become load imbalanced due to mesh refinement and coarsening
Scratch-remap schemes create an entirely new partition
Diffusive schemes attempt to tweak the existing partition to achieve better load balance, often minimizing migration costs

10. Work Stealing A method of load balancing
Each processor partnered with a neighbor
If its load goes below an application-level threshold, it sends messages to partner asking for work; i.e. tries to steal work
If partner does not have enough units, another partner is contacted
Can also be implemented using a master-worker approach; i.e., worker asking for more work from master

11. Graph Representation of Mesh
For irregular mesh applications, the computations associated with a mesh can be represented as a graph
Vertices represent the grid cells; vertex weights represent the amount of computations associated with the grid cells
Edges represent the communication between the grid cells; edge weights represent the amount of interactions

12. Graph Representation of Mesh
The objective is to partition across P processors
Each partition has equal amount of vertex weight
Total weight of the edges cut by the partition is minimized

13. Scratch-map Method
Partitioning from scratch will result in high vertex migration since as the partitioning does not take the initial location of the vertices into account
Hence a partitioning method should incrementally construct a new partition as simply a modification of the input partition

14. Notations Let B(q) be the set of vertices with partition q
Weight of any partition q can be defined as:
Average partition weight:
A graph is imbalanced if it is partitioned, and:

15. Terms
A partition is over-balanced if its weight is greater than the average partition weight times (1+ )
If less, under-balanced
The graph is balanced when no partition is over-balanced
Repartitioning – existing partition used as an input to form a new partition

16. Terms
A vertex is clean if its current partition is its initial partition; else dirty
Border vertex – adjacent vertex in another partition; those partitions are neighbor partitions
TotalV – sum of the sizes of the vertices which change partitions; i.e., sum of the sizes of the dirty vertices

17. 3 Objectives Maintain balance between partitions Minimize edge cuts
Minimize TotalV

18. Different Schemes Repartitioning from scratch
Cut-and-paste repartitioning: excess vertices in an overbalanced partition are simply swapped into one or more underbalanced partitions in order to bring these partitions up to balance
The method can optimize TOTALV, but can have a negative effect on the edge-cut

19. Different Schemes Another method is analogous to diffusion
Concept is for vertices to move from overbalanced to neighboring underbalanced partitions

20. Example (Assuming edge and vertex weights as equal to 1)

21. Example (contd..)

22. Analysis of the 3 schemes
Thus, cut-and-paste repartitioning minimizes TotalV, while completely ignoring edge-cut
Partitioning the graph from the scratch minimizes edge-cut, while resulting in high TotalV
Diffusion attempts to keep both TotalV and edge-cut low

23. Diffusion Scheme: An Algorithm

24. Multi-level Graph Repartitioning
Based on diffusive scheme
Multi-level graph repartitioning based on the k-way multilevel partitioning
Applying graph partitioning directly on a graph representation of a mesh can be time consuming since the graph can be very large

25. K-way multilevel partitioning algorithm
Has 3 phases: coarsening, partitioning, refinement (uncoarsening)
Coarsening - a sequence of smaller graphs constructed out of an input graph by collapsing vertices together

26. K-way multilevel partitioning algorithm
When enough vertices are collapsed together so that the coarsest graph is sufficiently small, a k-way partition is found
Finally, the partition of the coarsest graph is projected back to the original graph by refining it at each uncoarsening level using a k-way partitioning refinement algorithm

27. K-way partitioning refinement
A simple randomized algorithm that moves vertices among the partitions to minimize edge-cut and improve balance
For a vertex v, let neighborhood N(v) be the union of the partitions to which the vertices adjacent to v belong
In a k-way refinement algorithm, vertices are visited randomly

28. K-way partitioning refinement
A vertex v is moved to one of the neighboring partitions N(v) if any of the following vertex migration criteria is satisfied
The edge-cut is reduced while maintaining the balance
The balance improves while maintaining the edge-cut
This process is repeated until no further reduction in edge-cut is obtained

29. Multi-level diffusion repartitioning
In the coarsening phase, only pairs of nodes that belong to the same partition are considered for merging
Hence, initial partition of the coarsest level graph is identical to the input partition that is being repartitioned
This makes the coarsening phase completely parallelizable, as coarsening is local to each processor

30. Multi-level diffusion repartitioning
Uncoarsening phase contains 2 subphases
Multi-level diffusion
Multi-level refinement
Multi-level diffusion – balance is attempted on the coarsest graph

31. Multi-level diffusion repartitioning
A vertex is visited in a random order
If it is in overbalanced partition and is neighbors with nonoverbalanced partitions, moved to that nonoverbalanced partition that produces greatest improvement in edge-cut
Vertex is migrated even if this results in increased edge-cut
The process repeats until no balancing progress is made

32. Multi-level diffusion repartitioning
But it may not be possible to balance the graph at the coarsest level since there may not be many vertices of small weight
Hence graph needs to be uncoarsened one level to increase the number of smaller weight vertices
The process is repeated on the next coarsest graph and so on until some threshold

33. Multi-level diffusion repartitioning
After graph is balanced, multi-level diffusion ends, and multilevel refinement begins on the current graph
Here, the emphasis is on improving the edge-cut

34. Multi-level diffusion repartitioning
Each border vertex is visited randomly to potentially migrate based on the following refinement phase vertex migration criteria
The selected partition is the vertex’s initial partition from the input graph; the move does not increase the edge-cut; the balance is maintained
The edge-cut is reduced while maintaining the balance
The balance is improved while maintaining the edge-cut

35. Summary
Multi-level diffusion repartitioning algorithms are made up of three phases: graph coarsening, multilevel diffusion, multilevel refinement
Coarsening phase results in a series of contracted graphs
Multilevel diffusion phase balances the graph using the very coarsest graphs
Mutlilevel refinement phase seeks to improve the edge-cut disturbed by the balancing process

36. Illustration

37. Space Filling Curves for Partitioning and Load Balancing

38. Space Filling Curves
The underlying idea is to map a multidimensional space to one dimension where the partitioning is trivial
There are many different ways
But a mapping for partitioning algorithms should preserve the proximity information present in the multidimensional space to minimize communication costs

39. Space Filling Curve
Space filling curves are quick to run, can be implemented in parallel, and produce good load balancing with locality
A space-filling curve is formed over grid/mesh cells by using the centroid of the cells to represent them
The SFC produces a linear ordering of the cells such that cells that are close together in a linear ordering are also close together in the higher dimensional space

40. Space Filling Curve
The curve is then broken into segments based on the weights of the cells (weights computed using size and number of particles)
The segments are distributed to processors; thus cells that are close together in space are assigned to the same processor
This reduces overall amount of communication that occur, i.e., increases locality

41. SFC representation
One method is to define recursively – curve for a 2^k x 2^k grid composed of four 2^(k-1) x 2^(k-1) curves
Another method is using bit interleaving – the position of a grid point along the curve can be specified by interleaving bits of the coordinates of the point
The interleaving function is a characteristic of the curve

42. Z-curve or Morton ordering
The curve for a 2^k x 2^k grid composed of four 2^(k-1) x 2^(k-1) curves, one in each quadrant of the 2^k x 2^k grid
Order in which the curves are connected is the same as the order for 2x2 grid

43. Graycode Curve Uses same interleaving function as Z-curve
But visits points in the graycode order
Graycode – two successive values differ in only one bit
The one-bit gray code is (0,1)

44. Graycode Curve
The gray code list for n bits can be generated recursively using n-1 bits
By reflecting the list (reversing the list) [1,0]
Concatenating original with the reflected [0,1,1,0]
Prefixing entries in the original list with 0, and prefixing entries in the reflected list with 1 [00,01,11,10]

45. Graycode Curve
3-bit gray code: 000,001,011,010,110,111,101,100

46. Hilbert Curve
Hilbert curve is a smooth curve that avoids the sudden jumps in Z-curve and graycode curve
Curve composed of four curves of previous resolution in four quadrants
Curve in the lower left quadrant rotated clockwise by 90 degree, and curve in lower right quadrant rotated anticlockwise by 90 degree

47. SFCs for AMR
All these curve based partitioning techniques can also be applied for adaptive mesh by forming hierarchical SFCs