1. MA5233 Lecture 6 Krylov Subspaces and Conjugate Gradients Wayne M. Lawton Department of Mathematics National University of Singapore 2 Science Drive 2 Singapore 117543 Email matwml@nus.edu.sg http://www.math.nus/~matwml Tel (65) 6874-2749 1

2. EUCLIDEAN SPACES that satisfies 2 Definition A Euclidean structure on a vector space is a function Bilinearand Symmetric for all Positive Definite and Definition The norm is

3. EXAMPLES 3 Example 1. (Standard) is positive definite and symmetric.where is positive except at possible a finite Example 2. Example 3. where number of points – hence p is nonnegative. Example 4. the Euclidean space in example 3. Then V is a Real andare obtained by Hilbert Space = Complete Real Euclidean Space.

4. ORTHONORMAL BASES 4 Definition is an orthonormal basis if Example for the standard Euclidean space iff the matrix is an orthonormal basis for V satisfies is the transpose matrix defined bywhere andis the identity matrix defined by Such a matrix is called orthogonal and satisfies and

5. GRAM-SCHMIDT PROCESS 5 Given a basis there exists a unique upper triangular matrix with positive numbers on its diagonal such that for a Euclidean space V are orthogonal (and therefore are a basis for V). Proof We apply the Gram-Schmidt Process For j = 2 to d

6. QR FACTORIZATION 6 Given a basis yields an upper triangular matrix with positive numbers on its diagonal such that for are therefore, sinceis upper triangular, Gram-Schmidt a factorization that has important applications to least-squares problems (section 5.3) and to compute eigenvalues and eigenvectors (section 5.5)

7. PARTIAL HESSENBERG FACTORIZATION 7 Definition A (not necessarily square) matrix We consider a matrix and orthonormal vectors is upper Hessenberg if and integer such that or, equivalently

8. KRYLOV SPACES AND ARNOLDI ITERATION 8 has dimension n, then an orthonormal basis If the Krylov space can be computed by GS using the Arnoldi Iteration based on the equation For j = 2 to n (Recall that

9. COMPLETE HESSENBERG FACTORIZATION 9 Possibly using more than one Krylov subspace we can construct an orthonormal basisfor such that where We observe that the number of Krylov subspaces equals 1+ number of zeros on the diagonal beneath the main diagonal.

10. TRI-DIAGONAL MATRIX 10 Theoremiff Proof. therefore Corollary Ifthenis tridiagonal.

11. LANCZOS ITERATION 11 Theorem If and an orthonormal basis for then For j = 1 to n-1 can be computed by GS using the Lanczos Iteration

12. CONJUGATE GRADIENT ITERATION 12 that Hestenes and Stiefel made famous solves Ax = b For j = 1 to n-1 under the assumption that A is symmetric and pos. def.

13. CONJUGATE GRADIENT ITERATION 13 Theorem 1. The following sets all = and Proof By induction if j < n-1then andsince if j < n-1then since and

14. CONJUGATE GRADIENT ITERATION 14 Theorem 2. If A is symmetric and positive definite then if the CG algorithm to solve Ax = 0 has not minimizes Proof If and convergence is monotonic thenconverged, that is then therefore for Theorem 3. If subordinate to the 2-norm then Proof See the handouts