1. Linear Systems Numerical Methods

2. Linear equations N unknowns, M equations N unknowns, M equations where coefficient matrix

3. 3 Determinants and Cramer’s Rule [A] : coefficient matrix D : Determinant of A matrix

4. Solving methods Direct methodsDirect methods – Gauss elimination – Gauss-Jordan elimination – LU decomposition – Singular value decomposition – … Iterative methodsIterative methods – Jacobi iteration – Gauss-Seidel iteration – …

5. Linear Systems Solve Ax=b, where A is an n  n matrix and b is an n  1 column vectorSolve Ax=b, where A is an n  n matrix and b is an n  1 column vector Can also talk about non-square systems where A is m  n, b is m  1, and x is n  1Can also talk about non-square systems where A is m  n, b is m  1, and x is n  1 – Overdetermined if m > n : “more equations than unknowns” – Underdetermined if n > m : “more unknowns than equations” Can look for best solution using least squares

6. 6 Gauss Elimination Solve Ax = bSolve Ax = b Consists of two phases:Consists of two phases: – Forward elimination – Back substitution Forward EliminationForward Elimination reduces Ax = b to an upper triangular system Tx = b’ Back substitution can then solve Tx = b’ for xBack substitution can then solve Tx = b’ for x Forward Elimination Back Substitution

7. Gauss Elimination Fundamental operations:Fundamental operations: 1. Replace one equation with linear combination of other equations 2. Interchange two equations 3. Re-label two variables Combine to reduce to trivial systemCombine to reduce to trivial system Simplest variant only uses #1 operations, but get better stability by adding #2 (partial pivoting) or #2 and #3 (full pivoting)Simplest variant only uses #1 operations, but get better stability by adding #2 (partial pivoting) or #2 and #3 (full pivoting)

8. 8 Gaussian Elimination Forward Elimination x 1 - x 2 + x 3 = 6 3 x 1 + 4 x 2 + 2 x 3 = 9 2 x 1 + x 2 + x 3 = 7 x 1 - x 2 + x 3 = 6 0 +7 x 2 - x 3 = - 9 0 + 3 x 2 - x 3 = - 5 x 1 - x 2 + x 3 = 6 0 7 x 2 - x 3 = -9 0 0 -(4/7) x 3 =-(8/7) -(3/1) Solve using BACK SUBSTITUTION: x 3 = 2 x 2 =-1 x 1 =3 -(2/1) -(3/7) 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

9. 9 Back Substitution 1x01x0 +1x 1 –1x2–1x2 +4x 3 8= – 2x 1 –3x2–3x2 +1x 3 5= 2x22x2 – 3x 3 0= 2x32x3 4= x 3 = 2

10. 10 1x01x0 +1x 1 –1x2–1x2 0= – 2x 1 –3x2–3x2 3= 2x22x2 6= Back Substitution x 2 = 3

11. 11 1x01x0 +1x 1 3= – 2x 1 12= Back Substitution x 1 = –6

12. 1x01x0 9= Back Substitution x 0 = 9

13. 13 for i  n down to 1 do /* calculate x i */ x [ i ]  b [ i ] / a [ i, i ] /* substitute in the equations above */ for j  1 to i-1 do b [ j ]  b [ j ]  x [ i ] × a [ j, i ] endfor Back Substitution (* Pseudocode *) Time Complexity?  O(n 2 )

14. 14 Forward Elimination 4x04x0 +6x 1 +2x 2 – 2x 3 =8 2x02x0 +5x 2 – 2x 3 =4 –4x0–4x0 – 3x 1 – 5x 2 +4x 3 =1 8x08x0 +18x 1 – 2x 2 +3x 3 =40 -(2/4) MULTIPLIERSMULTIPLIERS -(-4/4) -(8/4)

15. 15 4x04x0 +6x 1 +2x 2 – 2x 3 =8 +4x 2 – 1x 3 =0 +3x 1 – 3x 2 +2x 3 =9 +6x 1 – 6x 2 +7x 3 =24 – 3x 1 -(3/-3) MULTIPLIERSMULTIPLIERS Forward Elimination -(6/-3)

16. 16 4x04x0 +6x 1 +2x 2 – 2x 3 =8 +4x 2 – 1x 3 =0 1x21x2 +1x 3 =9 2x22x2 +5x 3 =24 – 3x 1 ?? MULTIPLIERMULTIPLIER Forward Elimination

17. 17 4x04x0 +6x 1 +2x 2 – 2x 3 =8 +4x 2 – 1x 3 =0 1x21x2 +1x 3 =9 3x33x3 =6 – 3x 1 Forward Elimination

18. 18 Gauss-Jordan Elimination 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 a 11 x 22 0 x 33 0 0 x 44 0 0 0 x 55 0 0 0 0 x 66 x 77 x 88 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 b 11 b 22 b 33 b 44 b 55 b 66 b 77 b 66 000 x 99

19. 19 Gauss-Jordan Elimination: Example R2  R2 - (-1)R1 R3  R3 - ( 3)R1 Scaling R2: R2  R2/( -1 ) R1  R1 - (1)R2 R3  R3-(4)R2 Scaling R3: R3  R3/( 18 ) R1  R1 - (7)R3 R2  R2-(-5)R3 RESULT: x 1 =8.45, x 2 =-2.89, x 3 =1.23 Time Complexity?  O(n 3 )

20. Gauss-Jordan Elimination Solve:Solve: Only care about numbers – form “tableau” or “augmented matrix”:Only care about numbers – form “tableau” or “augmented matrix”:

21. Gauss-Jordan Elimination Given:Given: Goal: reduce this to trivial system and read off answer from right columnGoal: reduce this to trivial system and read off answer from right column

22. Gauss-Jordan Elimination Basic operation 1: replace any row by linear combination with any other rowBasic operation 1: replace any row by linear combination with any other row Here, replace row1 with 1 / 2 * row1 + 0 * row2Here, replace row1 with 1 / 2 * row1 + 0 * row2

23. Gauss-Jordan Elimination Replace row2 with row2 – 4 * row1Replace row2 with row2 – 4 * row1 Negate row2Negate row2

24. Gauss-Jordan Elimination Replace row1 with row1 – 3 / 2 * row2Replace row1 with row1 – 3 / 2 * row2 Read off solution: x 1 = 2, x 2 = 1Read off solution: x 1 = 2, x 2 = 1

25. Augmented Matrices Matrices are rectangular arrays of numbers that can aid us by eliminating the need to write the variables at each step of the reduction.Matrices are rectangular arrays of numbers that can aid us by eliminating the need to write the variables at each step of the reduction. For example, the systemFor example, the system may be represented by the augmented matrix Coefficie nt Matrix

26. Matrices and Gauss-Jordan Every step in the Gauss-Jordan elimination method can be expressed with matrices, rather than systems of equations, thus simplifying the whole process:Every step in the Gauss-Jordan elimination method can be expressed with matrices, rather than systems of equations, thus simplifying the whole process: Steps expressed as systems of equations:Steps expressed as systems of equations: Steps expressed as augmented matrices:Steps expressed as augmented matrices: Toggle slides back and forth to compare before and changes

27. Matrices and Gauss-Jordan Every step in the Gauss-Jordan elimination method can be expressed with matrices, rather than systems of equations, thus simplifying the whole process:Every step in the Gauss-Jordan elimination method can be expressed with matrices, rather than systems of equations, thus simplifying the whole process: Steps expressed as systems of equations:Steps expressed as systems of equations: Steps expressed as augmented matrices:Steps expressed as augmented matrices: Toggle slides back and forth to compare before and changes

28. Matrices and Gauss-Jordan Every step in the Gauss-Jordan elimination method can be expressed with matrices, rather than systems of equations, thus simplifying the whole process:Every step in the Gauss-Jordan elimination method can be expressed with matrices, rather than systems of equations, thus simplifying the whole process: Steps expressed as systems of equations:Steps expressed as systems of equations: Steps expressed as augmented matrices:Steps expressed as augmented matrices: Toggle slides back and forth to compare before and changes

29. Matrices and Gauss-Jordan Every step in the Gauss-Jordan elimination method can be expressed with matrices, rather than systems of equations, thus simplifying the whole process:Every step in the Gauss-Jordan elimination method can be expressed with matrices, rather than systems of equations, thus simplifying the whole process: Steps expressed as systems of equations:Steps expressed as systems of equations: Steps expressed as augmented matrices:Steps expressed as augmented matrices: Toggle slides back and forth to compare before and changes

30. Matrices and Gauss-Jordan Every step in the Gauss-Jordan elimination method can be expressed with matrices, rather than systems of equations, thus simplifying the whole process:Every step in the Gauss-Jordan elimination method can be expressed with matrices, rather than systems of equations, thus simplifying the whole process: Steps expressed as systems of equations:Steps expressed as systems of equations: Steps expressed as augmented matrices:Steps expressed as augmented matrices: Toggle slides back and forth to compare before and changes

31. Matrices and Gauss-Jordan Every step in the Gauss-Jordan elimination method can be expressed with matrices, rather than systems of equations, thus simplifying the whole process:Every step in the Gauss-Jordan elimination method can be expressed with matrices, rather than systems of equations, thus simplifying the whole process: Steps expressed as systems of equations:Steps expressed as systems of equations: Steps expressed as augmented matrices:Steps expressed as augmented matrices: Toggle slides back and forth to compare before and changes

32. Matrices and Gauss-Jordan Every step in the Gauss-Jordan elimination method can be expressed with matrices, rather than systems of equations, thus simplifying the whole process:Every step in the Gauss-Jordan elimination method can be expressed with matrices, rather than systems of equations, thus simplifying the whole process: Steps expressed as systems of equations:Steps expressed as systems of equations: Steps expressed as augmented matrices:Steps expressed as augmented matrices: Toggle slides back and forth to compare before and changes

33. Matrices and Gauss-Jordan Every step in the Gauss-Jordan elimination method can be expressed with matrices, rather than systems of equations, thus simplifying the whole process:Every step in the Gauss-Jordan elimination method can be expressed with matrices, rather than systems of equations, thus simplifying the whole process: Steps expressed as systems of equations:Steps expressed as systems of equations: Steps expressed as augmented matrices:Steps expressed as augmented matrices: Toggle slides back and forth to compare before and changes

34. Matrices and Gauss-Jordan Every step in the Gauss-Jordan elimination method can be expressed with matrices, rather than systems of equations, thus simplifying the whole process:Every step in the Gauss-Jordan elimination method can be expressed with matrices, rather than systems of equations, thus simplifying the whole process: Steps expressed as systems of equations:Steps expressed as systems of equations: Steps expressed as augmented matrices:Steps expressed as augmented matrices: Row Reduced Form of the Matrix Toggle slides back and forth to compare before and changes