1. PIMA-motivation PIMA: Partition Improvement using Mesh Adjacencies  Parallel simulation requires that the mesh be distributed with equal work-load and minimum inter-part communications  Graph/hyper-graph partitions are powerful for unstructured meshes, however they use one type of mesh entities as the graph nodes, hence the balance of other mesh entities may not be optimal  Different applications have different requirement for load balance, different types of mesh entities may have to be balanced at the same time  LIIPBMod* has been developed to knock down the peaks of vertices *M. Zhou, O. Sahni, K.D. Devine, M.S. Shephard, K.E. Jansen, “Controlling unstructured mesh partitions for massively parallel simulations”, SIAM Journal on Scientific Computing, 32(6):3201-3227, 2010

2. PIMA-advantages  Problems in partitions obtained by graph/hyper-graph based methods are limited to a small number of heavily loaded parts, referred to as spikes, the peak limits the scalabilities of applications  Uses mesh adjacencies -- Richer information than graph/hyper- graph based method, chances to provide better partitions  All adjacencies are obtainable in O(1) operations (not a function of mesh size) -- algorithm is efficient  Takes advantages of neighborhood communications -- Work well on massively parallel computations, since the limited communications used even at extreme scale PIMA is designed to migrate a small number of mesh entities on inter-part boundaries from heavily loaded parts to their lightly loaded neighbors to improve load balance

3. PIMA-algorithm  Input from users: – Types of mesh entities need to be balanced (Rgn, Face, Edge, Vtx) – The relative importance (priority) between them (= or >) e.g., “Vtx=Edge>Rgn” or “Rgn>Face=Edge>Vtx”, etc. The ones not specified in the input has no interest for balance  Steps of PIMA: – From high to low priority if separated by “>” (different groups) – From low to high dimensions based on entities topologies if separated by “=” (same group) e.g., “Rgn>Face=Edge>Vtx” is the user’s input Step 1: reduce the spikes for mesh regions Step 2.1: reduce the spikes for mesh edges Step 2.2: reduce the spikes for mesh faces Step 3: reduce the spikes for mesh vertices

4. PIMA-algorithm  PIMA: migrate a small number of mesh entities on inter-part boundaries (candidate mesh entities) to their lightly loaded neighboring parts (candidate parts) to improve the partitions  Candidate parts: – Absolutely and relatively lightly loaded – Lightly loaded for the mesh entities in the current group and the ones with higher priority – “Relatively” lightly loaded and iterative nature of the algorithm allow diffusive improvement of the partition balance  Candidate mesh entities – Ones on inter-part boundaries on heavily loaded parts are selected to be migrated s.t. the migration reduces the load imbalance and meanwhile maintains/improves the inter-part boundary

5. PIMA-algorithm: Candidate mesh entities 1.Vertex balance improvement: the vertices on inter-part boundaries bounding a small number of regions 2. Edge balance improvement: the edges on inter-part boundaries bounding a small number of faces 3. Face/Region balance improvement: Regions which have two or three faces on inter- part boundaries

6. PIMA-Tests 133M region mesh on 16k parts Table 1: Users input Table 2:Balance of partitions Table 3: Time usage and iterations (tests on Jaguar Cray XT5 system)