1. 15-853Page1 Separatosr in the Real World Practice Slides combined from www.cs.berkeley.edu/~demmel/cs267_Spr07 www.cs.cmu.edu/~guyb/realworld/slidesF03/src/separators1.ppt

2. 15-853Page2 Edge Separators 7 8 3 4 0 1 2 5 6 An edge separator : a set of edges E’ ½ E which partitions V into V 1 and V 2 Criteria: |V 1 |, |V 2 | balanced |E’| is small V1V1 V2V2 E’

3. 15-853Page3 Vertex Separators 7 8 3 4 0 1 2 5 6 An vertex separator : a set of vertices V’ ½ V which partitions V into V 1 and V 2 Criteria: |V 1 |, |V 2 | balanced |V’| is small V1V1 V2V2

4. 15-853Page4 Other names Sometimes referred to as –graph partitioning (probably more common than “graph separators”) –graph bisectors –graph bifurcators –balanced graph cuts

5. Application 15-853Page5 Telephone network design –Original application, algorithm due to Kernighan Load Balancing while Minimizing Communication Sparse Matrix times Vector Multiplication –Solving PDEs –N = {1,…,n}, (j,k) in E if A(j,k) nonzero, –W N (j) = #nonzeros in row j, W E (j,k) = 1 VLSI Layout –N = {units on chip}, E = {wires}, W E (j,k) = wire length Sparse Gaussian Elimination –Used to reorder rows and columns to increase parallelism, and to decrease “fill-in” Data mining and clustering Physical Mapping of DNA

6. 15-853Page6 Recursive Separation

7. 15-853Page7 What graphs have small separators Planar graphs: O(n 1/2 ) vertex separators 2d meshes, constant genus, excluded minors Almost planar graphs: the internet, power networks, road networks Circuits need to be laid out without too many crossings Social network graphs: phone-call graphs, link structure of the web, citation graphs, “friends graphs” 3d-grids and meshes: O(n 1/2 )

8. 15-853Page8 What graphs don’t have small separatos Expanders: Can someone give a quick proof that they don’t have small separators Hypercubes: O(n) edge separators O(n/(log n) 1/2 ) vertex separators Butterfly networks: O(n/log n) separators ? It is exactly the fact that they don’t have small separators that make them useful.

9. 15-853Page9 Applications of Separators Circuit Layout (dates back to the 60s) VLSI layout Solving linear systems (nested dissection) n 3/2 time for planar graphs Partitioning for parallel algorithms Approximations to certain NP hard problems TSP, maximum-independent-set Clustering and machine learning Machine vision

10. 15-853Page10 More Applications of Separators Out of core algorithms Register allocation Shortest Paths Graph compression Graph embeddings

11. 15-853Page11 Available Software METIS: U. Minnessota PARTY: University of Paderborn CHACO: Sandia national labs JOSTLE: U. Geenwich SCOTCH: U. Bordeaux GNU: Popinet Benchmarks: Graph Partitioning Archive

12. 15-853Page12 Different Balance Criteria Bisectors: 50/50 Constant fraction cuts: e.g. 1/3, 2/3 Trading off cut size for balance: min cut criteria: min quotient separator: All versions are NP-hard

13. 15-853Page13 Other Variants of Separators k-Partitioning: Might be done with recursive partitioning, but direct solution can give better answers. Weighted: Weights on edges (cut size), vertices (balance) Hypergraphs: Each edge can have more than 2 end points common in VLSI circuits Multiconstraint: Trying to balance different values at the same time.

14. 15-853Page14 Edge vs. Vertex separators If a class of graphs satisfies an f(n) edge-separator theorem then it satisfies an f(n) vertex-separator. The other way is not true (unless degree is bounded) |C| = n/2

15. 15-853Page15 Separator Trees

16. 15-853Page16 Asymptotics If S is a class of graphs closed under the subgraph relation, then Definition: S satisfies a f(n) vertex- separator theorem if there are constants  0 so that for every G  S there exists a cut set C  V, with 1.|C| <  f(|G|) cut size 2.|A| <  |G|, |B| <  |G|balance Similar definition for edge separators.

17. 15-853Page17 Separator Trees Theorem: For S satisfying an ( ,  ) n 1-  edge separator theorem, we can generate a perfectly balanced separator tree with separator size |C| < k  f(|G|). Proof: by picture |C| =  n 1-  (1 +  +  2 + …) =  n 1-  (1/1-  )

18. 15-853Page18 Algorithms All are either heuristics or approximations –Kernighan-Lin (heuristic) –Planar graph separators (finds O(n 1/2 ) separators) –Geometric separators (finds O(n (d-1)/d ) separators) –Spectral (finds O(n (d-1)/d ) separators) –Flow techniques (give log(n) approximations) –Multilevel recursive bisection (heuristic, currently most practical)

19. 04/13/2007CS267 Guest Lecture19 Partitioning via Breadth First Search BFS identifies 3 kinds of edges –Tree Edges - part of T –Horizontal Edges - connect nodes at same level –Interlevel Edges - connect nodes at adjacent levels No edges connect nodes in levels differing by more than 1 (why?) BFS partioning heuristic –N = N 1 U N 2, where N 1 = {nodes at level <= L}, N 2 = {nodes at level > L} –Choose L so |N 1 | close to |N 2 | BFS partition of a 2D Mesh using center as root: N1 = levels 0, 1, 2, 3 N2 = levels 4, 5, 6 root

20. 15-853Page20 Kernighan-Lin Heuristic Local heuristic for edge-separators based on “hill climbing”. Will most likely end in a local-minima. Two versions: Original: takes n 2 times per step Fiduccia-Mattheyses: takes n times per step

21. 15-853Page21 High-level description for both Start with an initial cut that partitions the vertices into two equal size sets V 1 and V 2 Want to swap two equal sized sets X  A and Y  B to reduce the cut size. Note that finding the optimal subsets X and Y solves the optimal separator problem, so it is NP hard. We want some heuristic that might help. AB YX C

22. 15-853Page22 Some Terminology C(A,B) : the weighted cut between A and B I(v) : the number of edges incident on v that stay within the partition E(v) : the number of edges incident on v that go to the other partition D(v) : E(v) - I(v) D(u,v) : D(u) + D(v) - 2 w(u,v) the gain for swapping u and v A B C v u

23. 15-853Page23 Kernighan-Lin improvement step KL(G,A 0,B 0 )  (u  A 0, v  B 0 ) put (u,v) in a PQ based on D(u,v) for k = 1 to |V|/2 (u,v) = max(PQ) (A k,B k ) = (A k-1,B k-1 ) swap (u,v) delete u and v entries from PQ update D on neighbors (and PQ) select A k,B k with best C k Note that can take backward steps (D(u,v) can be negative). A B C v u

24. 15-853Page24 Fiduccia-Mattheyses improvement step FM(G,A 0,B 0 )  u  A 0 put u in PQ A based on D(u)  v  B 0 put v in PQ B based on D(v) for k = 1 to |V|/2 u = max(PQ A ) put u on B side and update D v = max(PQ b ) put v on A side and update D select A k,B k with best C k A B C v u

25. 15-853Page25 A Bad Example for KL or FM Consider following graph with initial cut given in red. KL (or FM) will start on one side of grid and flip pairs over moving across the grid until the whole thing is flipped. The cut will never be better than the original bad cut. 222 111

26. 15-853Page26 Performance in Practice In general the algorithms do very well at smoothing a cut that is approximately correct. Used by most separator packages either 1.to smooth final results 2.to smooth partial results during the algorithm 3.K/L tested on very small graphs (|N|<=360) and got convergence after 2-4 sweeps 4.For random graphs (of theoretical interest) the probability of convergence in one step appears to drop like 2 -|N|/30

27. 27 Introduction to Multilevel Partitioning If we want to partition G(N,E), but it is too big to do efficiently, what can we do? –1) Replace G(N,E) by a coarse approximation G c (N c,E c ), and partition G c instead –2) Use partition of G c to get a rough partitioning of G, and then iteratively improve it What if G c still too big? –Apply same idea recursively

28. 28 Multilevel Partitioning - High Level Algorithm (N+,N- ) = Multilevel_Partition( N, E ) … recursive partitioning routine returns N+ and N- where N = N+ U N- if |N| is small (1) Partition G = (N,E) directly to get N = N+ U N- Return (N+, N- ) else (2) Coarsen G to get an approximation G c = (N c, E c ) (3) (N c +, N c - ) = Multilevel_Partition( N c, E c ) (4) Expand (N c +, N c - ) to a partition (N+, N- ) of N (5) Improve the partition ( N+, N- ) Return ( N+, N- ) endif (2,3) (1) (4) (5) How do we Coarsen? Expand? Improve? “V - cycle:”

29. 29 Multilevel Kernighan-Lin Coarsen graph and expand partition using maximal matchings Improve partition using Kernighan-Lin

30. 30 Maximal Matching: Example

31. 31 Example of Coarsening

32. 32 Coarsening using a maximal matching 1) Construct a maximal matching E m of G(N,E) for all edges e=(j,k) in E m 2) collapse matched nodes into a single one Put node n(e) in N c W(n(e)) = W(j) + W(k) … gray statements update node/edge weights for all nodes n in N not incident on an edge in E m 3) add unmatched nodes Put n in N c … do not change W(n) … Now each node r in N is “inside” a unique node n(r) in N c … 4) Connect two nodes in Nc if nodes inside them are connected in E for all edges e=(j,k) in E m for each other edge e’=(j,r) or (k,r) in E Put edge ee = (n(e),n(r)) in E c W(ee) = W(e’) If there are multiple edges connecting two nodes in N c, collapse them, adding edge weights

33. 33 Expanding a partition of G c to a partition of G

34. 34 Multilevel Spectral Bisection Coarsen graph and expand partition using maximal independent sets Improve partition using Rayleigh Quotient Iteration

35. 35 Example of Coarsening - encloses domain D k = node of N c

36. 36 Coarsening using Maximal Independent Sets … Build “domains” D(k) around each node k in N i to get nodes in N c … Add an edge to E c whenever it would connect two such domains E c = empty set for all nodes k in N i D(k) = ( {k}, empty set ) … first set contains nodes in D(k), second set contains edges in D(k) unmark all edges in E repeat choose an unmarked edge e = (k,j) from E if exactly one of k and j (say k) is in some D(m) mark e add j and e to D(m) else if k and j are in two different D(m)’s (say D(mi) and D(mj)) mark e add edge (mi, mj) to E c else if both k and j are in the same D(m) mark e add e to D(m) else leave e unmarked endif until no unmarked edges

37. 37 Expanding a partition of G c to a partition of G Need to convert an eigenvector v c of L(G c ) to an approximate eigenvector v of L(G) Use interpolation: For each node j in N if j is also a node in N c, then v(j) = v c (j) … use same eigenvector component else v(j) = average of v c (k) for all neighbors k of j in N c end if

38. 38 Example: 1D mesh of 9 nodes

39. 39 Improve eigenvector: Rayleigh Quotient Iteration j = 0 pick starting vector v(0) … from expanding v c repeat j=j+1 r(j) = v T (j-1) * L(G) * v(j-1) … r(j) = Rayleigh Quotient of v(j-1) … = good approximate eigenvalue v(j) = (L(G) - r(j)*I) -1 * v(j-1) … expensive to do exactly, so solve approximately … using an iteration called SYMMLQ, … which uses matrix-vector multiply (no surprise) v(j) = v(j) / || v(j) || … normalize v(j) until v(j) converges … Convergence is very fast: cubic

40. 40 Example of convergence for 1D mesh

41. 41 Available Implementations Multilevel Kernighan/Lin –METIS (www.cs.umn.edu/~metis) –ParMETIS - parallel version Multilevel Spectral Bisection –S. Barnard and H. Simon, “A fast multilevel implementation of recursive spectral bisection …”, Proc. 6th SIAM Conf. On Parallel Processing, 1993 –Chaco (www.cs.sandia.gov/CRF/papers_chaco.html) Hybrids possible –Ex: Using Kernighan/Lin to improve a partition from spectral bisection

42. 42 Comparison of methods Compare only methods that use edges, not nodal coordinates –CS267 webpage and KK95a (see below) have other comparisons Metrics –Speed of partitioning –Number of edge cuts –Other application dependent metrics Summary –No one method best –Multi-level Kernighan/Lin fastest by far, comparable to Spectral in the number of edge cuts –Spectral give much better cuts for some applications Ex: image segmentation See “Normalized Cuts and Image Segmentation” by J. Malik, J. Shi

43. 43 Number of edges cut for a 64-way partition Graph 144 4ELT ADD32 AUTO BBMAT FINAN512 LHR10 MAP1 MEMPLUS SHYY161 TORSO # of Nodes 144649 15606 4960 448695 38744 74752 10672 267241 17758 76480 201142 # of Edges 1074393 45878 9462 3314611 993481 261120 209093 334931 54196 152002 1479989 Description 3D FE Mesh 2D FE Mesh 32 bit adder 3D FE Mesh 2D Stiffness M. Lin. Prog. Chem. Eng. Highway Net. Memory circuit Navier-Stokes 3D FE Mesh # Edges cut 88806 2965 675 194436 55753 11388 58784 1388 17894 4365 117997 Expected # cuts 6427 2111 1190 11320 3326 4620 1746 8736 2252 4674 7579 3D mesh 31805 7208 3357 67647 13215 20481 5595 47887 7856 20796 39623 Expected # cuts for 64-way partition of 2D mesh of n nodes n 1/2 + 2*(n/2) 1/2 + 4*(n/4) 1/2 + … + 32*(n/32) 1/2 ~ 17 * n 1/2 Expected # cuts for 64-way partition of 3D mesh of n nodes = n 2/3 + 2*(n/2) 2/3 + 4*(n/4) 2/3 + … + 32*(n/32) 2/3 ~ 11.5 * n 2/3 For Multilevel Kernighan/Lin, as implemented in METIS (see KK95a)

44. Thank you! 44