The Landscape of Sparse Ax=b Solvers Direct A = LU Iterative y’ = Ay Non- symmetric Symmetric positive definite More RobustLess Storage More Robust More General D

Four views of the conjugate gradient method To solve Ax = b, where A is symmetric and positive definite. 1.Operational view 2.Orthogonal sequences view 3.Optimization view 4.Convergence view

Four views of the conjugate gradient method To solve Ax = b, where A is symmetric and positive definite. 1.Operational view 2.Orthogonal sequences view 3.Optimization view 4.Convergence view

Conjugate gradient iteration for Ax = b x 0 = 0 approx solution r 0 = b residual = b - Ax d 0 = r 0 search direction for k = 1, 2, 3,... x k = x k-1 + … new approx solution r k = … new residual d k = … new search direction

Conjugate gradient iteration for Ax = b x 0 = 0 approx solution r 0 = b residual = b - Ax d 0 = r 0 search direction for k = 1, 2, 3,... α k = … step length x k = x k-1 + α k d k-1 new approx solution r k = … new residual d k = … new search direction

Conjugate gradient iteration for Ax = b x 0 = 0 approx solution r 0 = b residual = b - Ax d 0 = r 0 search direction for k = 1, 2, 3,... α k = (r T k-1 r k-1 ) / (d T k-1 Ad k-1 ) step length x k = x k-1 + α k d k-1 new approx solution r k = … new residual d k = … new search direction

Conjugate gradient iteration for Ax = b x 0 = 0 approx solution r 0 = b residual = b - Ax d 0 = r 0 search direction for k = 1, 2, 3,... α k = (r T k-1 r k-1 ) / (d T k-1 Ad k-1 ) step length x k = x k-1 + α k d k-1 new approx solution r k = … new residual β k = (r T k r k ) / (r T k-1 r k-1 ) d k = r k + β k d k-1 new search direction

Conjugate gradient iteration for Ax = b x 0 = 0 approx solution r 0 = b residual = b - Ax d 0 = r 0 search direction for k = 1, 2, 3,... α k = (r T k-1 r k-1 ) / (d T k-1 Ad k-1 ) step length x k = x k-1 + α k d k-1 new approx solution r k = r k-1 – α k Ad k-1 new residual β k = (r T k r k ) / (r T k-1 r k-1 ) d k = r k + β k d k-1 new search direction

Conjugate gradient iteration One matrix-vector multiplication per iteration Two vector dot products per iteration Four n-vectors of working storage x 0 = 0, r 0 = b, d 0 = r 0 for k = 1, 2, 3,... α k = (r T k-1 r k-1 ) / (d T k-1 Ad k-1 ) step length x k = x k-1 + α k d k-1 approx solution r k = r k-1 – α k Ad k-1 residual β k = (r T k r k ) / (r T k-1 r k-1 ) improvement d k = r k + β k d k-1 search direction

Four views of the conjugate gradient method To solve Ax = b, where A is symmetric and positive definite. 1.Operational view 2.Orthogonal sequences view 3.Optimization view 4.Convergence view

Krylov subspaces Eigenvalues: Av = λv { λ 1, λ 2,..., λ n } Cayley-Hamilton theorem implies (A – λ 1 I)·(A – λ 2 I) · · · (A – λ n I) = 0 Therefore Σ c i A i = 0 for some c i so A -1 = Σ (–c i /c 0 ) A i–1 Krylov subspace: Therefore if Ax = b, then x = A -1 b and x  span (b, Ab, A 2 b,..., A n-1 b) = K n (A, b) 0  i  n 1  i  n

Conjugate gradient: Orthogonal sequences Krylov subspace: K i (A, b) = span (b, Ab, A 2 b,..., A i-1 b) Conjugate gradient algorithm: for i = 1, 2, 3,... find x i  K i (A, b) such that r i = (b – Ax i )  K i (A, b) Notice r i  K i+1 (A, b), so r i  r j for all j < i Similarly, the “directions” are A-orthogonal: (x i – x i-1 ) T ·A· (x j – x j-1 ) = 0 The magic: Short recurrences... A is symmetric => can get next residual and direction from the previous one, without saving them all.

Four views of the conjugate gradient method To solve Ax = b, where A is symmetric and positive definite. 1.Operational view 2.Orthogonal sequences view 3.Optimization view 4.Convergence view

Four views of the conjugate gradient method To solve Ax = b, where A is symmetric and positive definite. 1.Operational view 2.Orthogonal sequences view 3.Optimization view 4.Convergence view

Conjugate gradient: Convergence In exact arithmetic, CG converges in n steps (completely unrealistic!!) Accuracy after k steps of CG is related to: consider polynomials of degree k that are equal to 1 at 0. how small can such a polynomial be at all the eigenvalues of A? Thus, eigenvalues close together are good. Condition number: κ (A) = ||A|| 2 ||A -1 || 2 = λ max (A) / λ min (A) Residual is reduced by a constant factor by O(κ 1/2 (A)) iterations of CG.

Complexity of direct methods n 1/2 n 1/3 2D3D Space (fill): O(n log n)O(n 4/3 ) Time (flops): O(n 3/2 )O(n 2 ) Time and space to solve any problem on any well- shaped finite element mesh

Complexity of linear solvers 2D3D Dense Cholesky:O(n 3 ) Sparse Cholesky:O(n 1.5 )O(n 2 ) CG, exact arithmetic: O(n 2 ) CG, no precond: O(n 1.5 )O(n 1.33 ) CG, modified IC: O(n 1.25 )O(n 1.17 ) CG, support trees: O(n 1.20 ) -> O(n 1+ )O(n 1.75 ) -> O(n 1+ ) Multigrid:O(n) n 1/2 n 1/3 Time to solve model problem (Poisson’s equation) on regular mesh

Hierarchy of matrix classes (all real) General nonsymmetric Diagonalizable Normal Symmetric indefinite Symmetric positive (semi)definite = Factor width n Factor width k... Factor width 4 Factor width 3 Diagonally dominant SPSD = Factor width 2 Generalized Laplacian = Symm diag dominant M-matrix Graph Laplacian

Other Krylov subspace methods Nonsymmetric linear systems: GMRES: for i = 1, 2, 3,... find x i  K i (A, b) such that r i = (Ax i – b)  K i (A, b) But, no short recurrence => save old vectors => lots more space (Usually “restarted” every k iterations to use less space.) BiCGStab, QMR, etc.: Two spaces K i (A, b) and K i (A T, b) w/ mutually orthogonal bases Short recurrences => O(n) space, but less robust Convergence and preconditioning more delicate than CG Active area of current research Eigenvalues: Lanczos (symmetric), Arnoldi (nonsymmetric)