1. Discrete Algorithms & Math Department Preconditioning ‘03 Algebraic Tools for Analyzing Preconditioners Bruce Hendrickson Erik Boman Sandia National Labs

2. Discrete Algorithms & Math Department Preconditioning ‘03 Long History l Many have worked on algebraic analysis methods l Abridged history of this line of work: »Beauwens, Notay, Axelsson & others (80s) »Vaidya (’91) »Miller, Gremban, Guattery (mid 90s) »Gilbert, Boman, Toledo, Chen, H., etc. (current)

3. Discrete Algorithms & Math Department Preconditioning ‘03 Outline l Definitions and concepts l Basic tools l Rectangular factorizations l Special cases »Vaidya’s spanning trees »Recent extensions l Approximate inverse preconditioning

4. Discrete Algorithms & Math Department Preconditioning ‘03 Starting Point: Preconditioned CG l Solving system Ax=f with preconditioner B »For now, focus on symmetric positive definite matrices »General systems later in the talk l Iterations of preconditioned CG bounded by spectral condition number of matrix pencil »

5. Discrete Algorithms & Math Department Preconditioning ‘03 Support Number Support number  (A,B) »  ( A,B) = min {  | x T (  B-A)x  0  x} l Closely related to largest eigenvalue » max (A,B) ≤  (A,B) » min (A,B) = 1/ max (B,A) ≥ 1/  (B,A) »(equality if full rank) »So,  2 (A,B) ≤  (A,B)  (B,A)

6. Discrete Algorithms & Math Department Preconditioning ‘03 Why Support Numbers?  ( A,B) is largest generalized eigenvalue projected onto range space of B »Equals max (A,B) when B full rank »Remains well defined when B rank deficient »Robust extension of largest eigenvalue Easier to work with  ( A,B) than with (A,B)

7. Discrete Algorithms & Math Department Preconditioning ‘03 Properties of Support Numbers Splitting Lemma: If A =  i=1,k A i and B=  i=1,k B i »Then  (A,B) ≤ max i  (A i,B i ) –Used in finite elements and domain decomposition –Our interest is algebraic –How to split? l Triangle Inequality: If B, C positive semidefinite »Then  (A,C) ≤  (A,B)  (B,C) When A, B are psd, then  ( A,B) ≤ 1/{1 -  (A-B, A)} »Useful for analysis of incomplete Cholesky

8. Discrete Algorithms & Math Department Preconditioning ‘03 More Properties of Support Numbers l Symmetric Product Support: If A=UU T and B=VV T »Then  (A,B) = min W ||W|| 2 2 such that VW=U »Special case: –  ( uu T,VV T ) = min w w T w, where Vw=u l (Many more properties in our papers)

9. Discrete Algorithms & Math Department Preconditioning ‘03 M-Matrices, Graphs & Rectangular Factorizations l Consider a simple Laplacian

10. Discrete Algorithms & Math Department Preconditioning ‘03 Rectangular Factorizations l A = UU T, where U is arbitrary l Interesting special cases: »Columns of U have 1 nonzero –Non-negative diagonal matrices »+ columns with 2 nonzeros, same magnitude, opposite sign –Symmetric, diagonally dominant M-matrices »+ columns with 2 nonzeros, same magnitude –Symmetric, diagonally dominant matrices »+ columns with 2 nonzeros –Symmetric H-matrices with non-negative diagonal

11. Discrete Algorithms & Math Department Preconditioning ‘03 Rectangular Factorizations and Finite Elements l Factor each element matrix before assembly l A= UU T, where U rectangular, and few nonzeros per column l Block column of U for each element l “Natural Factor” – Argyris & Bronlund

12. Discrete Algorithms & Math Department Preconditioning ‘03 Very Special Case: Vaidya’s Spanning Trees l Let u and columns of V look like »  ( + , 0, …, 0, -  ) T, or  (…, 0, , 0, …) T l Matrices representable as UU T : »Symmetric, diagonally-dominant, M-matrices l Note: vectors with 2 nonzeros can be thought of as edges in a (weighted) graph

13. Discrete Algorithms & Math Department Preconditioning ‘03 Support Paths u = Vw,  (uu T,VV T ) = w T w l Support number = length-of-path

14. Discrete Algorithms & Math Department Preconditioning ‘03 Vaidya’s Spanning Tree Preconditioners l Vaidya used this to analyze preconditioners built from max-weight spanning trees l Using support-path analysis, easy to show that » Worst case condition number = O(nm) »n = matrix size, m = number of nonzeros l “Exact factorization of an incomplete matrix.” (J. Gilbert)

15. Discrete Algorithms & Math Department Preconditioning ‘03 Matrix Interpretation l Given rectangular U (m<n) »Find subset of columns V that makes a good basis »That is, VW=U, where W has small 2-norm l Vaidya’s max-weight spanning tree ensures »Entries of V -1 U are all of magnitude no more than 1 »O(nm) condition number follows

16. Discrete Algorithms & Math Department Preconditioning ‘03 Vaidya’s Augmentation l Can add edges in special way »Reduce condition number, but increase factorization cost »O(n 1.75 ) runtime for solving general problems »O(n 1.2 ) runtime to solve for planar graphs l Bounds independent of sparsity pattern & numerical values!

17. Discrete Algorithms & Math Department Preconditioning ‘03 How does Vaidya work in practice? l Chen & Toledo, ETNA 2003 l Sensitive to structure, not numerical values l Competitive with relaxed Modified Incomplete Cholesky on 2D problems l Sometimes worse, sometimes much better on 3D problems l Interesting convergence behavior

18. Discrete Algorithms & Math Department Preconditioning ‘03 Vaidya on Easy Problem

19. Discrete Algorithms & Math Department Preconditioning ‘03 Vaidya on Harder Problem

20. Discrete Algorithms & Math Department Preconditioning ‘03 Beyond Vaidya: Other Spanning Trees l Max-weight spanning tree might have bad topology (long support paths) l Trade worse numerics for better structure l MASST: (min average stretch spanning tree) »Alon/Karp/Peleg/West(’95) for networks »Gives condition number bound of ~O(m lg n) for general graphs

21. Discrete Algorithms & Math Department Preconditioning ‘03 Hybrid Idea l Augmenting MASST trees »Spielman & Teng (’03) »Add extra edges to improve condition number »Solve general diagonally dominant M-matrices in O(n 1.31 ) time »No implementation yet

22. Discrete Algorithms & Math Department Preconditioning ‘03 Beyond Vaidya: Broader Matrix Classes l Allow columns of U of the form »  ( + , 0, …, 0, +  ) T l Now, UU T = »all symmetric, diagonally dominant matrices (SDD) l Two types of graph edges: Signed Graphs »Max spanning tree becomes max-weight basis of matroid »With Chen/Toledo, devised efficient algorithms & Vaidya- like analysis l O(nm) condition number bound for all SDD matrices »Can augment and get better bounds for planar graphs

23. Discrete Algorithms & Math Department Preconditioning ‘03 Factorized Approximate Inverses l A = UU T l What if we have V, an approximate “inverse” of U? »Could use V T V as a preconditioner l Possible advantages: »For some matrices, U is cheap to compute –Symmetric diagonally dominant –Finite elements »Columns of U capture natural structure –E.g. few columns = one finite element –Allows preconditioner to focus on bad elements

24. Discrete Algorithms & Math Department Preconditioning ‘03 “Inverting” Rectangular Factors l Let A=U D U T, and let V be a solution of » »Or alternatively, A V T = U D l Then, A -1 = V T D -1 V

25. Discrete Algorithms & Math Department Preconditioning ‘03 Nonsymmetric Systems l Let A=E F T, and X and Y solve » »And l Then A -1 = X T Y

26. Discrete Algorithms & Math Department Preconditioning ‘03 Status l Can solve KKT-like systems approximately »Use few steps of iterative method, or »Specify sparsity and minimize Frobineus norm l Empirical testing underway

27. Discrete Algorithms & Math Department Preconditioning ‘03 Other Ongoing Work l Finite elements »Use splitting lemma to decompose into elements »Use symmetric-product lemma to approximate each element »Assemble approximations and approximate the result l Incomplete factorizations »Simple proof of model problem results »Suggests alternative dropping strategies l Domain decomposition »Easy proof of known results for block Jacobi on model problem »Can generalize to some unstructured grids l Generalizing tools to handle nonsymmetric matrices

28. Discrete Algorithms & Math Department Preconditioning ‘03 Conclusions l Support numbers are nice analytical tool »Easy to prove algebraic properties l Rectangular factorizations are useful for analyzing and constructing preconditioners l Lots of open questions & opportunities for new insights

29. Discrete Algorithms & Math Department Preconditioning ‘03 Acknowledgements l Collaborators »Doron Chen »John Gilbert »Steve Guattery »Ojas Parekh »Clark Dohrmann »Nhat Nguyen l Work supported by DOE’s Applied Mathematical Science Program l Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed-Martin Company, for the U.S. DOE under contract DE-AC-94AL85000. »Sivan Toledo »Marshall Bern »Darin Diachin »Edmond Chow »Alex Pothen

30. Discrete Algorithms & Math Department Preconditioning ‘03 For More Information l www.cs.sandia.gov/~bahendr/support.html www.cs.sandia.gov/~bahendr/support.html l bah@cs.sandia.gov