1. E. T. S. I. Caminos, Canales y Puertos1 Engineering Computation Lecture 6

2. E. T. S. I. Caminos, Canales y Puertos2 Forward Elimination: DO k = 1 to n–1 DO i = k+1 to n r = A(i,k)/A(k,k) DO j = k+1 to n A(i,j)=A(i,j) – r*A(k,j) ENDDO B(i) = B(i) – r*B(k) ENDDO Gauss elimination (operation counting)

3. E. T. S. I. Caminos, Canales y Puertos3 Operation Counting for Gaussian Elimination Back Substitution: X(n) = B(n)/A(n,n) DO i = n–1 to 1 by –1 SUM = 0 DO j = i+1 to n SUM = SUM + A(i,j)*X(j) ENDDO X(i) = [B(i) – SUM]/A(i,i) ENDDO Gauss elimination (operation counting)

4. E. T. S. I. Caminos, Canales y Puertos4 Operation Counting for Gaussian Elimination Forward Elimination Inner loop: Second loop: = (n 2 + 2n) – 2(n + 1)k + k 2 Gauss elimination (operation counting)

5. E. T. S. I. Caminos, Canales y Puertos5 Operation Counting for Gaussian Elimination Forward Elimination (cont'd) Outer loop= Gauss elimination (operation counting)

6. E. T. S. I. Caminos, Canales y Puertos6 Operation Counting for Gaussian Elimination Back Substitution Inner Loop: Outer Loop: Gauss elimination (operation counting)

7. E. T. S. I. Caminos, Canales y Puertos7 Total flops = Forward Elimination + Back Substitution = n 3 /3 + O (n 2 )+n 2 /2 + O (n)  n 3 /3 + O (n 2 ) To convert (A,b) to (U,b') requires n 3 /3, plus terms of order n 2 and smaller, flops. To back solve requires: 1 + 2 + 3 + 4 +... + n = n (n+1) / 2 flops; Grand Total: the entire effort requires n 3 /3 + O(n 2 ) flops altogether. Gauss elimination (operation counting)

8. E. T. S. I. Caminos, Canales y Puertos8 Diagonalization by both forward and backward elimination in each column. Perform elimination both backwards and forwards until: Operation count for Gauss-Jordan is: (slower than Gauss elimination) Gauss-Jordan Elimination

9. E. T. S. I. Caminos, Canales y Puertos9 Example (two-digit arithmetic): x 1 = 0.015 (vs. 0.016,  t = 6.3%) x 2 = 0.041 (vs. 0.041,  t = 0%) x 3 = 0.093 (vs. 0.091,  t = 2.2%) Gauss-Jordan Elimination

10. E. T. S. I. Caminos, Canales y Puertos10 The solution of: [A]{x} = {b} is:{x} = [A] -1 {b} where [A] -1 is the inverse matrix of [A] Consider:[A] [A] -1 = [ I ] 1) Create the augmented matrix: [ A | I ] 2) Apply Gauss-Jordan elimination: ==> [ I | A -1 ] Gauss-Jordan Matrix Inversion

11. E. T. S. I. Caminos, Canales y Puertos11 Gauss-Jordan Matrix Inversion (with 2 digit arithmetic): MATRIX INVERSE [A -1 ] Gauss-Jordan Matrix Inversion

12. E. T. S. I. Caminos, Canales y Puertos12 CHECK: [ A ] [ A ] -1 = [ I ] [ A ] -1 { b } = { x } Gaussian Elimination Gauss-Jordan Matrix Inversion

13. E. T. S. I. Caminos, Canales y Puertos13 LU decomposition LU decomposition - The LU decomposition is a method that uses the elimination techniques to transform the matrix A in a product of triangular matrices. This is specially useful to solve systems with different vectors b, because the same decomposition of matrix A can be used to evaluate in an efficient form, by forward and backward sustitution, all cases.

14. E. T. S. I. Caminos, Canales y Puertos14 LU decomposition Decomposition Initial system Transformed system 1Substitution Transformed system 2 Forward sustitutionBackward sustitution

15. E. T. S. I. Caminos, Canales y Puertos15 LU decomposition LU decomposition is very much related to Gauss method, because the upper triangular matrix is also looked for in the LU decomposition. Thus, only the lower triangular matrix is needed. Surprisingly, during the Gauss elimination procedure, this matrix L is obtained, but one is not aware of this fact. The factors we use to get zeroes below the main diagonal are the elements of this matrix L. Substract

16. E. T. S. I. Caminos, Canales y Puertos16 LU decomposition

17. E. T. S. I. Caminos, Canales y Puertos17 Basic Approach Consider [A]{x} = {b} a) Gauss-type "decomposition" of [A] into [L][U] n 3 /3 flops [A]{x} = {b} becomes [L] [U]{x} = {b}; let [U]{x}  {d} b) First solve [L] {d} = {b} for {d} by forward subst. n 2 /2 flops c) Then solve [U]{x} = {d} for {x} by back substitution n 2 /2 flops LU decomposition (Complexity)

18. E. T. S. I. Caminos, Canales y Puertos18 [A] = [L] + [U 0 ] [A] = [L 0 ] + [U] [A] = [L 0 ] + [U 0 ] + [D] [A] = [L 1 ] [U] [A] = [L] [U 1 ] LU Decompostion: notation

19. E. T. S. I. Caminos, Canales y Puertos19 LU Decomposition Variations Doolittle[L 1 ][U]General [A] Crout[L][U 1 ]General [A] Cholesky[L][L] T Pos. Def. Symmetric [A] Cholesky works only for Positive Definite Symmetric matrices Doolittle versus Crout: Doolittle just stores Gaussian elimination factors where Crout uses a different series of calculations (see C&C 10.1.4). Both decompose [A] into [L] and [U] in n 3 /3 FLOPS Different location of diagonal of 1's Crout uses each element of [A] only once so the same array can be used for [A] and [L\U] saving computer memory! LU decomposition

20. E. T. S. I. Caminos, Canales y Puertos20 Matrix Inversion Definition of a matrix inverse: [A] [A] -1 = [ I ] ==> [A] {x} = {b} [A] -1 {b} = {x} First Rule: Don’t do it. (numerically unstable calculation) LU decomposition

21. E. T. S. I. Caminos, Canales y Puertos21 Matrix Inversion If you really must -- 1) Gaussian elimination: [A | I ] –> [U | B'] ==> A -1 2) Gauss-Jordan: [A | I ] ==> [I | A -1 ] Inversion will take n 3 + O(n 2 ) flops if one is careful about where zeros are (taking advantage of the sparseness of the matrix) Naive applications (without optimization) take 4n 3 /3 + O(n 2 ) flops. For example, LU decomposition requires n 3 /3 + O(n 2 ) flops. Back solving twice with n unit vectors e i : 2 n (n 2 /2) = n 3 flops. Altogether: n 3 /3 + n 3 = 4n 3 /3 + O(n 2 ) flops LU decomposition

22. E. T. S. I. Caminos, Canales y Puertos22 Summary FLOP Counts for Linear Algebraic Equations, [A]{x} = {b} Gaussian Elimination (1 r.h.s)n 3 /3 + O (n 2 ) Gauss-Jordan (1 r.h.s) n 3 /2 + O (n 2 ) LU decomposition n 3 /3 + O (n 2 ) Each extra LU right-hand-siden 2 Cholesky decomposition (symmetric A) n 3 /6 + O (n 2 ) Inversion (naive Gauss-Jordan)4n 3 /3 +O (n 2 ) Inversion (optimal Gauss-Jordan)n 3 + O (n 2 ) Solution by Cramer's Rulen! FLOP Counts for Linear Algebraic Equations