Modern iterative methods For basic iterative methods, converge linearly Modern iterative methods, converge faster –Krylov subspace method Steepest descent method Conjugate gradient (CG) method --- most popular Preconditioning CG (PCG) method GMRES for nonsymmetric matrix –Other methods (read yourself) Chebyshev iterative method Lanczos methods Conjugate gradient normal residual (CGNR)

Modern iterative methods Ideas: –Minimizing the residual –Projecting to Krylov subspace Thm: If A is an n-by-n real symmetric positive definite matrix, then have the same solution Proof: see details in class

Steepest decent method Suppose we have an approximation Choose the direction as negative gradient of –If –Else, choose to minimize

Steepest decent method Computation Choose as

Algorithm– Steepest descent method

Theory Suppose A is symmetric positive definite. Define A-inner product Define A-norm Steepest decent method

Theory Thm: For steepest decent method, we have Proof: Exercise

Theory Rewrite the steepest decent method Let errors Lemma: For the method, we have

Theory Thm: For steepest decent method, we have Proof: See details in class (or as an exercise)

Steepest decent method Performance –Converge globally, for any initial data –If, then it converges very fast –If, then it converges very slow!!! Geometric interpretation –Contour plots are flat!! –Local best direction (steepest direction) is not necessarily a global best direction –Computational experience shows that the method suffers a decreasing convergence rate after a few iteration steps because the search directions become linearly dependent!!!

Conjugate gradient (CG) method Since A is symmetric positive definite, A-norm In CG method, the direction vectors are chosen to be A-orthogonal (and called as conjugate vectors), i.e.

CG method In addition, we take the new direction vector as a linear combination of the old direction vector and the descent direction as By the assumption we get

Algorithm– CG Method

An example Initial guess The approximate solutions

CG method In CG method, are A-orthogonal! Define the linear space as Lemma: In CG method, for m=0,1,…., we have –Proof: See details in class or as an exercise

CG method In CG method, is A-orthogonal to or Lemma: In CG method, we have –Proof: See details in class or as an exercise Thm: Error estimate for CG method

CG method Computational cost –At each iteration, 2 matrix-vector multiplications. This can be further reduced to 1 matrix-vector multiplications –At most n steps, we can get the exact solution!!! Convergence rate depends on the condition # –K 2 (A)=O(1), converges very fast!! –K 2 (A)>>1, converges slow but can be accelerated by preconditioning!!

Preconditioning Ideas: Replace by satisfying –C is symmetric positive definite – is well-conditioned, i.e. – can be easily solved Conditions for choosing the preconditioning matrix – as small as possible – is easy to compute –Trade-off

Algorithm– PCG Method

Preconditioning Ways to choose the matrix C (read yourself) –Diagonal part of A –Tri-diagonal part of A –m-step Jacobi preconditioner –Symmetric Gauss-Seidel preconditioner –SSOR preconditioner –In-complete Cholesky decomposition –In-complete block preconditioning –Preconditioning based on domain decomposition –…….

Extension of CG method to nonsymmetric Biconjugate gradient (BiCG) method: –Solve simultaneously –Works well for A is positive definite, not symmetric –If A is symmetric, BiCG reduces to CG Conjugate gradient squared (CGS) method –A has a special formula in computing Ax, its transport hasn’t –Multiplication by A is efficient but multiplication by its transport is not

Krylov subspace methods Problem I. Linear system Problem II. Variational formulation Problem III. Minimization problem –Thm1: Problem I is equivalent to Problem II –Thm2: If A is symmetric positive definite, they are equivalent

Krylov subspace methods To reduce problem size, we replace by a subspace Subspace minimization: –Find –Such that Subspace projection

Krylov subspace methods To determine the coefficients, we have – Normal Equations –It is a linear system with degree m!! m=1: line minimization or linear search or 1D projection By converting this formula into an iteration, we reduce the original problem into a sequence of line minimization (successive line minimization ).

For symmetric matrix Positive definite –Steepest decent method –CG method –Preconditioning CG method Non-positive definite –MINRES (minimum residual method)

For nonsymmetric matrix Normal equations method (or CGNR method) GMRES (generalized minimium residual method) –Saad & Schultz, 1986 –Ideas: In the m-th step, minimize the residual over the set Use Arnoldi (full orthogonal) vectors instead of Lanczos vectors If A is symmetric, it reduces to the conjugate residual method

Algorithm– GMRES

More topics on Matrix computations Eigenvalue & eigenvector computations If A is symmetric: Power method If A is general matrix –Householder matrix (transform) –QR method

More topics on matrix computations Singular value decomposition (SVD) Thm: Let A be an m-by-n real matrix, there exists orthogonal matrices U & V such that Proof: Exercise