Complexity of direct methods n 1/2 n 1/3 2D3D Space (fill): O(n log n)O(n 4/3 ) Time (flops): O(n 3/2 )O(n 2 ) Time and space to solve any problem on any well- shaped finite element mesh

Complexity of linear solvers 2D3D Dense Cholesky:O(n 3 ) Sparse Cholesky:O(n 1.5 )O(n 2 ) CG, exact arithmetic: O(n 2 ) CG, no precond: O(n 1.5 )O(n 1.33 ) CG, modified IC0: O(n 1.25 )O(n 1.17 ) CG, support trees: O(n 1.20 ) -> O(n 1+ )O(n 1.75 ) -> O(n 1+ ) Multigrid:O(n) n 1/2 n 1/3 Time to solve model problem (Poisson’s equation) on regular mesh

Hierarchy of matrix classes (all real) General nonsymmetric Diagonalizable Normal Symmetric indefinite Symmetric positive (semi)definite = Factor width n Factor width k... Factor width 4 Factor width 3 Symmetric diagonally dominant = Factor width 2 Generalized Laplacian = Symmetric diagonally dominant M-matrix Graph Laplacian

Definitions The Laplacian matrix of an n-vertex undirected graph G is the n-by-n symmetric matrix A with a ij = -1 if i ≠ j and (i, j) is an edge of G a ij = 0 if i ≠ j and (i, j) is not an edge of G a ii = the number of edges incident on vertex i Theorem: The Laplacian matrix of G is symmetric, singular, and positive semidefinite. The multiplicity of 0 as an eigenvalue is equal to the number of connected components of G. A generalized Laplacian matrix (more accurately, a symmetric weakly diagonally dominant M-matrix) is an n-by-n symmetric matrix A with a ij ≤ 0 if i ≠ j a ii ≥ Σ |a ij | where the sum is over j ≠ i

Edge-vertex factorization of generalized Laplacians A generalized Laplacian matrix A can be factored as A = UU T, where U has: a row for each vertex a column for each edge, with two nonzeros of equal magnitude and opposite sign a column for each excess-weight vertex, with one nonzero AUUTUT =× vertices edges ( 2 nzs/col) excess- weight vertices ( 1 nz/col) vertices

Support Graph Preconditioning +: New analytic tools, some new preconditioners +: Can use existing direct-methods software -: Current theory and techniques limited CFIM: Complete factorization of incomplete matrix Define a preconditioner B for matrix A Explicitly compute the factorization B = LU Choose nonzero structure of B to make factoring cheap (using combinatorial tools from direct methods) Prove bounds on condition number using both algebraic and combinatorial tools

Spanning Tree Preconditioner Spanning Tree Preconditioner [Vaidya] A is generalized Laplacian (symmetric diagonally dominant with negative off-diagonal nzs) B is the gen Laplacian of a maximum-weight spanning tree for A (with diagonal modified to preserve row sums) Form B: costs O(n log n) or less time (graph algorithms for MST) Factorize B = R T R: costs O(n) space and O(n) time (sparse Cholesky) Apply B -1 : costs O(n) time per iteration G(A) G(B)

Combinatorial analysis: cost of preconditioning A is generalized Laplacian (symmetric diagonally dominant with negative off-diagonal nzs) B is the gen Laplacian of a maximum-weight spanning tree for A (with diagonal modified to preserve row sums) Form B: costs O(n log n) time or less (graph algorithms for MST) Factorize B = R T R: costs O(n) space and O(n) time (sparse Cholesky) Apply B -1 : costs O(n) time per iteration (two triangular solves) G(A) G(B)

Numerical analysis: quality of preconditioner support each edge of A by a path in B dilation(A edge) = length of supporting path in B congestion(B edge) = # of supported A edges p = max congestion, q = max dilation condition number κ (B -1 A) bounded by p·q (at most O(n 2 )) G(A) G(B)

Spanning Tree Preconditioner Spanning Tree Preconditioner [Vaidya] can improve congestion and dilation by adding a few strategically chosen edges to B cost of factor+solve is O(n 1.75 ), or O(n 1.2 ) if A is planar in experiments by Chen & Toledo, often better than drop-tolerance MIC for 2D problems, but not for 3D. G(A) G(B)

Numerical analysis: Support numbers Intuition from networks of electrical resistors: graph = circuit; edge = resistor; weight = 1/resistance = conductance How much must you amplify B to provide as much conductance as A? How big does t need to be for tB – A to be positive semidefinite? What is the largest eigenvalue of B -1 A ? The support of B for A is σ(A, B) = min { τ : x T (tB – A)x  0 for all x and all t  τ } If A and B are SPD then σ(A, B) = max{ λ : Ax = λ Bx} = λ max (A, B) Theorem: If A and B are SPD then κ (B -1 A) = σ(A, B) · σ(B, A)

Old analysis, splitting into paths and edges Split A = A 1 + A 2 + ··· + A k and B = B 1 + B 2 + ··· + B k such that A i and B i are positive semidefinite Typically they correspond to pieces of the graphs of A and B (edge, path, small subgraph) Theorem: σ(A, B)  max i {σ(A i, B i )} Lemma: σ(edge, path)  (worst weight ratio) · (path length) In the MST case: A i is an edge and B i is a path, to give σ(A, B)  p·q B i is an edge and A i is the same edge, to give σ(B, A)  1

Edge-vertex factorization of generalized Laplacians A generalized Laplacian matrix A can be factored as A = UU T, where U has: a row for each vertex a column for each edge, with two nonzeros of equal magnitude and opposite sign a column for each excess-weight vertex, with one nonzero AUUTUT =× vertices edges ( 2 nzs/col) excess- weight vertices ( 1 nz/col) vertices

New analysis: Algebraic Embedding Lemma New analysis: Algebraic Embedding Lemma vv [Boman/Hendrickson] Lemma: If V·W=U, then σ(U·U T, V·V T )  ||W|| 2 2 (with equality for some choice of W) Proof: take t  ||W|| 2 2 = λ max ( W·W T ) = max y  0 { y T W·W T y / y T y } then y T (tI - W·W T ) y  0 for all y letting y = V T x gives x T (tV·V T - U·U T ) x  0 for all x recall σ(A, B) = min{ τ : x T (tB – A)x  0 for all x, all t  τ } thus σ(U·U T, V·V T )  ||W|| 2 2

A B - a 2 - b 2 -a 2 -c 2 -b 2 [ ] a 2 +b 2 - a 2 - b 2 - a 2 a 2 +c 2 - c 2 - b 2 - c 2 b 2 +c 2 [ ] a 2 +b 2 - a 2 - b 2 - a 2 a 2 - b 2 b 2 [ ] a b - a c - b - c [ ] a b - a c - b U V =VV T =UU T

A B - a 2 - b 2 -a 2 -c 2 -b 2 [ ] a 2 +b 2 - a 2 - b 2 - a 2 a 2 +c 2 - c 2 - b 2 - c 2 b 2 +c 2 [ ] a 2 +b 2 - a 2 - b 2 - a 2 a 2 - b 2 b 2 [ ] a b - a c - b - c [ ] a b - a c - b U V =VV T =UU T [ ] 1 - c / a 1 c / b /b W = x A edges B edges

A B - a 2 - b 2 -a 2 -c 2 -b 2 [ ] a 2 +b 2 - a 2 - b 2 - a 2 a 2 +c 2 - c 2 - b 2 - c 2 b 2 +c 2 [ ] a 2 +b 2 - a 2 - b 2 - a 2 a 2 - b 2 b 2 [ ] a b - a c - b - c [ ] a b - a c - b U V =VV T =UU T σ(A, B)  ||W|| 2 2  ||W||  x ||W|| 1 = (max row sum) x (max col sum)  (max congestion) x (max dilation) [ ] 1 - c / a 1 c / b /b W = x A edges B edges

Using another matrix norm inequality [Boman]: ||W|| 2 2  ||W|| F 2 = sum(w ij 2 ) = sum of (weighted) dilations, and [Alon, Karp, Peleg, West] construct spanning trees with average weighted dilation exp(O((log n loglog n) 1/2 )) = o(n  ). This gives condition number O(n 1+  ) and solution time O(n 1.5+  ), compared to Vaidya’s O(n 1.75 ) with augmented MST. Is there a graph construction that minimizes ||W|| 2 2 directly? [Spielman, Teng]: complicated recursive partitioning construction with solution time O(n 1+  ) for all generalized Laplacians! (Uses yet another matrix norm inequality.) Extensions, remarks, open problems I

Extensions, remarks, open problems II Make spanning tree methods more effective in 3D? Vaidya gives O(n 1.75 ) in general, O(n 1.2 ) in 2D. Issue: 2D uses bounded excluded minors, not just separators. Support theory methods for more general matrices? [Boman, Chen, Hendrickson, Toledo]: different matroid for all symmetric diagonally dominant matrices (= factor width 2). Matrices of bounded factor width? Factor width 3? All SPD matrices? Is there a version that’s useful in practice? Maybe for non-geometric graph Laplacians? [Koutis, Miller, Peng 2010] simplifies Spielman/Teng a lot. [Kelner et al. 2013] : random Kaczmarz projections in the dual space – even simpler, good O() theorems, but not yet fast enough in practice.

Support-graph analysis of modified incomplete Cholesky B has positive (dotted) edges that cancel fill B has same row sums as A Strategy: Use the negative edges of B to support both the negative edges of A and the positive edges of B A A = 2D model Poisson problem B B = MIC preconditioner for A

Supporting positive edges of B Every dotted (positive) edge in B is supported by two paths in B Each solid edge of B supports one or two dotted edges Tune fractions to support each dotted edge exactly 1/(2  n – 2) of each solid edge is left over to support an edge of A

Analysis of MIC: Summary Each edge of A is supported by the leftover 1/(2  n – 2) fraction of the same edge of B. Therefore σ(A, B)  2  n – 2 Easy to show σ(B, A)  1 For this 2D model problem, condition number is O(n 1/2 ) Similar argument in 3D gives condition number O(n 1/3 ) or O(n 2/3 ) (depending on boundary conditions)