1. Sandia National Laboratories Graph Partitioning Workshop Oct. 15, 1999 1 Load Balancing Myths, Fictions & Legends Bruce Hendrickson Sandia National Laboratories Parallel Computing Sciences Dept.

2. Sandia National Laboratories Graph Partitioning Workshop Oct. 15, 1999 2 Introduction l Two happy occurrences. »(1) Good graph partitioning tools & software. »(2) Good parallel efficiencies for many applications. l Is the latter due to the former? l Yes, but also no. l We have been lucky! »Wrong objectives. »Models insufficiently general. »Software tools often poorly designed.

3. Sandia National Laboratories Graph Partitioning Workshop Oct. 15, 1999 3 Myth 1: The Edge Cut Deceit l Generally believed that “Edge Cuts = Communication Cost”. l This assumption is behind the use of graph partitioning. l In reality: »Edge cuts are not equal to communication volume. »Communication volume is not equal to communication cost.

4. Sandia National Laboratories Graph Partitioning Workshop Oct. 15, 1999 4 Edge Cuts Versus Volume Edge cuts = 10. Communication volume: 8 (from left partition to right partition). 7 (from right partition to left partition).

5. Sandia National Laboratories Graph Partitioning Workshop Oct. 15, 1999 5 Communication Volume l Assume graph edges reflect data dependencies. l Correct accounting of communication volume is: »Number of vertices on boundary of partition. l Elegant alternative is hypergraph model of Çatalyürek, Aykanat, Pinar and Pinar. »Volume is number of cut hyperedges.

6. Sandia National Laboratories Graph Partitioning Workshop Oct. 15, 1999 6 Communication Cost l Cost of single message involves volume and latency. l Cost of multiple messages involves congestion. l Cost within application depends only on slowest processor. l Our models don’t optimize the right metrics!

7. Sandia National Laboratories Graph Partitioning Workshop Oct. 15, 1999 7 Why Does Graph Partitioning Work? l Vast majority of applications are computational meshes. »Geometric properties ensure that good partitions exist. –Communication/Computation = n 1/2 in 2D, n 2/3 in 3D. –Runtime is dominated by computation. »Vertices have bounded numbers of neighbors. –Error in edge cut metric is bounded. »Homogeneity ensures all processors have similar subdomains. –No processor has dramatically more communication. l Other applications aren’t so forgiving. –E.g. Interior point methods, latent semantic indexing, etc. l We have been lucky!

8. Sandia National Laboratories Graph Partitioning Workshop Oct. 15, 1999 8 Myth 2: Simple Graphs are Sufficient l Graphs are widely used to encode data dependencies. »Vertex weights reflect computational cost. »Edge weights encode volume of data transfer. l Graph partitioning determines data decomposition. l However, many problems are not easily expressed this way! »Complex relationships or constraints on partitioning. –E.g. computation on nodes and on elements. –DRAMA has a uniquely rich model for this problem. »Dependencies are directed (e.g. unsymmetric matvec). »Computation consists of multiple phases.

9. Sandia National Laboratories Graph Partitioning Workshop Oct. 15, 1999 9 Alternative Graph Models l Hypergraph model (Aykanat, et al.) »Vertices represent computations. »Hyperedge connects all objects which produce/use datum. »Handles directed dependencies. l Bipartite graph model (H. & Kolda) »Directed graph replaced by equivalent bipartite graph. »Handles directed dependencies. »Can model two-phase calculations. l Multi-Objective, Multi-Constraint (Schloegel, Karypis & Kumar) »Each vertex/edge can have multiple weights. »Can model multiple-phase calculations.

10. Sandia National Laboratories Graph Partitioning Workshop Oct. 15, 1999 10 Myth 3: Partition Quality is Paramount l Partitioners compete on edge cuts » and (sometimes) runtime. l This isn’t the full story » particularly for dynamic load balancing!

11. Sandia National Laboratories Graph Partitioning Workshop Oct. 15, 1999 11 7 Habits of Highly Effective Dynamic Load Balancers l Balance the load »Balance each phase or some combination? »How is load measured? l Minimize communication cost »Volume? Number of messages? Something else? l Run fast in parallel l Be incremental »Make new partition similar to current one l Not use too much memory l Support determination of new communication pattern l Be easy to use

12. Sandia National Laboratories Graph Partitioning Workshop Oct. 15, 1999 12 Performance Tradeoffs l From Touheed, Selwood, Jimack and Berzins. »~1 million elements with adaptive refinement. »32 processors of SGI Origin. l Timing data for different partitioning algorithms. »Repartitioning time per invocation: 3.0 - 15.2 seconds. »Migration time per invocation: 17.8 - 37.8 seconds. »Explicit solve time per timestep: 2.54 - 3.11 seconds. l Observations: »Migration time more important than partitioner runtime. »Importance of quality depends on: – Frequency of rebalancing. –Cost of solver.

13. Sandia National Laboratories Graph Partitioning Workshop Oct. 15, 1999 13 Myth 4: Existing Tools Solve the Problem l Lots of good software exists. l Static partitioning is fairly mature. »However, it addresses an imperfect model. l Dynamic partitioning is more complicated. »Applications differ in need for cost, quality or incrementality. »No algorithm is uniformly best. »Good library should support several (e.g. DRAMA). »Subroutine interface requires good software engineering. –E.g. Zoltan?.

14. Sandia National Laboratories Graph Partitioning Workshop Oct. 15, 1999 14 Myth 5: The Key is Finding the Right Partition l Assignment of work to processors is the key to scalability. l However, this needn’t be a single partition. l Example: parallel crash simulations. »In each timestep: –(1) Do finite element analysis & predict new deformations. –(2) Search for grid intersections (contact detection). –(3) If found, correct deformations & forces. »Each stage has different objects & different data dependencies. »Very difficult to balance them all with one decomposition. –But most work on this problem has taken this approach.

15. Sandia National Laboratories Graph Partitioning Workshop Oct. 15, 1999 15 Multiple Decompositions l Finite element analysis: »Topological »Static l Contact detection: »Geometric »Dynamic l Key Idea: »Use graph partitioning for finite element phase »Use geometric partitioner for contact detection –We use recursive coordinate bisection (RCB)

16. Sandia National Laboratories Graph Partitioning Workshop Oct. 15, 1999 16 Parallel Crash Simulations l Outline of parallel timestep in Pronto code: »1. Finite element analysis in static, graph-based decomposition. »2. Map contact objects to RCB decomposition. »3. Update RCB decomposition. »4. Communicate objects crossing processor boundaries. »5. Each processor searches for contacts. »6. Communicate contacts back to finite element decomposition. »7. Correct deformations, etc. l Observations: »Reuse serial contact software. »RCB is incremental & facilitates determination of communication. »Each phase should scale well. »Cost of mapping between decompositions?

17. Sandia National Laboratories Graph Partitioning Workshop Oct. 15, 1999 17 Can Crush Example Finite element decomposition RCB Decomposition as can gets crushed RCB on undeformed geometry

18. Sandia National Laboratories Graph Partitioning Workshop Oct. 15, 1999 18 Scalability l Can crush problem, 3800 elements per compute node, run to 100 ms. A 13.8-million element simulation on 3600 compute nodes ran at 0.32 seconds per time step (120.4 Gflops/s). contacts FEM overhead

19. Sandia National Laboratories Graph Partitioning Workshop Oct. 15, 1999 19 Micro-mechanics of Foam

20. Sandia National Laboratories Graph Partitioning Workshop Oct. 15, 1999 20 Myth 6: All the Problems are Solved l Biggest & most damaging myth of all! l Already discussed need for: »More accurate & expressive models. »Algorithms for partitioning new models. »More user-friendly tools. l Lots of other open problems.

21. Sandia National Laboratories Graph Partitioning Workshop Oct. 15, 1999 21 Open Problems l Partitioning for multiple goals. »Examples: –Multiple phases in a calculation. –Minimize communication volume and number of messages. » Multi-objective/constraint work is partial answer. –Only models costs on graph vertices or edges. –Can’t minimize number of messages. »New ideas are needed. l Partition to balance work of sparse solve on each subdomain. »Applications to FETI preconditioner, parallel sparse solvers, etc. »Complicated dependence on topology of submesh. »Can’t predict cost by standard edge/vertex weights.

22. Sandia National Laboratories Graph Partitioning Workshop Oct. 15, 1999 22 More Open Problems l Partitioning for heterogeneous parallel architectures. »E.g. Clusters of SMPs, Beowulf machines, etc. »How to model heterogeneous parallel machines? »How to adapt partitioners to address non-standard architectures? »(See Teresco, Beall, Flaherty & Shephard.) l Partitioning to minimize congestion. »Communication is limited by most heavily used wire. »How can we predict and avoid contention for wires? »(See Pellegrini or H., Leland & Van Driessche.)

23. Sandia National Laboratories Graph Partitioning Workshop Oct. 15, 1999 23 More Information l Contact info: »bah@cs.sandia.gov »http://www.cs.sandia.gov/~bahendr l Collaborators: »Rob LelandKaren DevineTammy Kolda »Steve PlimptonBill Mitchell l Work supported by DOE’s MICS program l Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed-Martin Company, for the US DOE under contract DE-AC-94AL85000