1. Tutorial 10 Iterative Methods and Matrix Norms

2. 2 In an iterative process, the k+1 step is defined via: Iterative processes Eigenvector decomposition dramatically simplifies the solution: The iterative process converges if: (1)

3. 3 Spectral radius of a matrix The iterative process converges if the spectral radius of T is below 1, where spectral radius of a matrix is defined as: The largest eigenvalue (the first after ordering) governs the dynamic system, i.e.: Unfortunately, straightforward calculation of spectral radius is itself a complicated task, and for large matrices is solved by ITERATIVE (!!) process.

4. 4 A norm on the vector space is a mapping,, satisfying the following axioms: Vector Norms The popular example of the norm is the p-norm: and its three special cases, p=1: sum of abs. values, p=2: Euclidian norm, and p=∞: the maximal absolute value element of the vector.

5. 5 For any two norms and, defined in a finite dimensional vector space, there exist positive constants C≥c>0, such that: Norms Equivalence Proof (outline): Due to property (b) of norms, it is enough to show (2) on the unit sphere,. Due to property (c) the norm on this sphere is a continuous function, thus it has the minimum m and maximum M, which are both positive due to property (a). Taking C=1/m and c = 1/M satisfies (2). (2)

6. 6 The induced matrix norm is defined via the vector norm, by considering the matrix as a linear operator in the vector space (isn’t that the essence of what is matrix for ?!). The induced norm of the matrix T is the maximum factor by which it can multiply the norm ( = the ‘length’ ) of the input vector. Induced Matrix Norm The last equality is due to the fact that matrices are linear operators and vector length can be reduced.

7. 7 The properties of vector norms (p.4) are naturally extended to matrices. Now let us prove that natural matrix norm satisfies these properties. The Norm Axioms are Satisfied (a) Positivity: Since vector norm is non-negative, the maximum is taken over non-negative numbers and therefore non-negative by itself: Now, assume that matrix T has at least one non-zero element, t jk, and let us show that in this case : Consider the vector,. Then the vector will have at least one non-zero element

8. 8 (b) Linearity: Follows from the fact that The Norm Axioms are Satisfied

9. 9 Natural Matrix Norm (c) Triangle inequality: Consider The right hand side is the norm of the sum of two vectors and therefore must satisfy the triangle inequality for the vector norm: searching for the maximums independently is a looser constraint – it can yield the same vector v or two other vectors u and w in case they give even greater maximum:

10. 10 Matrix norm for Convergence Estimation The norms of matrices are a useful alternative to the spectral radius in evaluation of matrix convergence due to the following Statement: if some induced norm of A,, then A is convergent. Proof: Apply A to any vector v, and get: Continuing the iterative process we obtain: Since the process converges.

11. 11 Spectral radius is not a norm. The matrix norm is a sufficient but not necessary condition for testing for convergence: on the next slide we will see an example of the converging matrix with Euclidian norm >1. Question: Is the spectral radius of a matrix a valid norm? NO! Proof: Consider the matrix Its both eigenvalues are 0, and therefore the spectral radius=0. However, A≠0 and therefore its norm must be positive. In fact, this matrix gives a convergence within one iteration ( A 2 =0).

12. 12 Example – See the Gap Statement: The Euclidian (p=2) norm of a matrix equals its maximum singular value. Proof: Done in class. Consider the matrix: We have that and yet, the eigenvalues are 0.5 and 0.25, implying a spectral radius of 0.5. This matrix leads to convergence.

13. 13 Gerschgorin Circle Theorem Let A be an n×n matrix. For each 1 ≤ i ≤ n, the Gerschgorin disk is defined as The Gerschgorin domain is defined as a union of all the Gerschgorin disks: Gerschgorin Circle Theorem: All eigenvalues of Matrix A lie in its Gerschgorin domain D A.

14. 14 Proof of Gerschgorin Circle Theorem Let v be an eigenvector of A with an eigenvalue λ. Let be a corresponding “normalized” eigenvector. Let u i be a maximal entry of u: u i = 1. Writing an i-th component of the eigenvalue equation Au= λu, we obtain: Now, we obtain