Linear Systems Numerical Methods

Linear equations N unknowns, M equations coefficient matrix where

Determinants and Cramer’s Rule [A] : coefficient matrix D : Determinant of A matrix

Solving methods Direct methods Iterative methods Gauss elimination Gauss-Jordan elimination LU decomposition Singular value decomposition … Iterative methods Jacobi iteration Gauss-Seidel iteration

Linear Systems Solve Ax=b, where A is an nn matrix and b is an n1 column vector Can also talk about non-square systems where A is mn, b is m1, and x is n1 Overdetermined if m>n: “more equations than unknowns” Underdetermined if n>m: “more unknowns than equations” Can look for best solution using least squares

Gauss Elimination Solve Ax = b Consists of two phases: Forward Forward elimination Back substitution Forward Elimination reduces Ax = b to an upper triangular system Tx = b’ Back substitution can then solve Tx = b’ for x Forward Elimination Back Substitution

Gauss Elimination Fundamental operations: Replace one equation with linear combination of other equations Interchange two equations Re-label two variables Combine to reduce to trivial system Simplest variant only uses #1 operations, but get better stability by adding #2 (partial pivoting) or #2 and #3 (full pivoting)

Gaussian Elimination Forward Elimination x1 - x2 + x3 = 6 -(3/1) -(3/7) -(2/1) x1 - x2 + x3 = 6 0 7x2 - x3 = -9 0 0 -(4/7)x3=-(8/7) Solve using BACK SUBSTITUTION: x3 = 2 x2=-1 x1 =3

Back Substitution 1x0 +1x1 –1x2 +4x3 = 8 – 2x1 –3x2 +1x3 = 5 2x2 – 3x3 2x3 = 4 x3 = 2

Back Substitution 1x0 +1x1 –1x2 = – 2x1 –3x2 = 3 2x2 = 6 x2 = 3

Back Substitution 1x0 +1x1 = 3 – 2x1 = 12 x1 = –6

Back Substitution 1x0 = 9 x0 = 9

Back Substitution (* Pseudocode *) for i  n down to 1 do /* calculate xi */ x [ i ]  b [ i ] / a [ i, i ] /* substitute in the equations above */ for j  1 to i-1 do b [ j ]  b [ j ]  x [ i ] × a [ j, i ] endfor Time Complexity?  O(n2)

Forward Elimination 4x0 +6x1 +2x2 – 2x3 = 8 2x0 +5x2 – 2x3 = 4 -(2/4) U L T I P E R S 4x0 +6x1 +2x2 – 2x3 = 8 2x0 +5x2 – 2x3 = 4 -(2/4) –4x0 – 3x1 – 5x2 +4x3 = 1 -(-4/4) 8x0 +18x1 – 2x2 +3x3 = 40 -(8/4)

Forward Elimination 4x0 +6x1 +2x2 – 2x3 = 8 – 3x1 +4x2 – 1x3 = +3x1 U L T I P E R S 4x0 +6x1 +2x2 – 2x3 = 8 – 3x1 +4x2 – 1x3 = +3x1 – 3x2 +2x3 = 9 -(3/-3) +6x1 – 6x2 +7x3 = 24 -(6/-3)

Forward Elimination 4x0 +6x1 +2x2 – 2x3 = 8 – 3x1 +4x2 – 1x3 = 1x2 U L T I P E R – 3x1 +4x2 – 1x3 = 1x2 +1x3 = 9 2x2 +5x3 = 24 ??

Forward Elimination 4x0 +6x1 +2x2 – 2x3 = 8 – 3x1 +4x2 – 1x3 = 1x2 1x2 +1x3 = 9 3x3 = 6

Gauss-Jordan Elimination b11 x22 b22 x33 b33 x44 b44 x55 b55 x66 b66 x77 b77 x88 b66 x99 b66

Gauss-Jordan Elimination: Example Scaling R2: R2  R2/(-1) R2  R2 - (-1)R1 R3  R3 - ( 3)R1 R1  R1 - (1)R2 R3  R3-(4)R2 Scaling R3: R3  R3/(18) R1  R1 - (7)R3 R2  R2-(-5)R3 RESULT: x1=8.45, x2=-2.89, x3=1.23 Time Complexity?  O(n3)

Gauss-Jordan Elimination Solve: Only care about numbers – form “tableau” or “augmented matrix”:

Gauss-Jordan Elimination Given: Goal: reduce this to trivial system and read off answer from right column

Gauss-Jordan Elimination Basic operation 1: replace any row by linear combination with any other row Here, replace row1 with 1/2 * row1 + 0 * row2

Gauss-Jordan Elimination Replace row2 with row2 – 4 * row1 Negate row2

Gauss-Jordan Elimination Replace row1 with row1 – 3/2 * row2 Read off solution: x1 = 2, x2 = 1

Augmented Matrices Matrices are rectangular arrays of numbers that can aid us by eliminating the need to write the variables at each step of the reduction. For example, the system may be represented by the augmented matrix Coefficient Matrix

Matrices and Gauss-Jordan Every step in the Gauss-Jordan elimination method can be expressed with matrices, rather than systems of equations, thus simplifying the whole process: Steps expressed as systems of equations: Steps expressed as augmented matrices: Toggle slides back and forth to compare before and changes

Matrices and Gauss-Jordan Every step in the Gauss-Jordan elimination method can be expressed with matrices, rather than systems of equations, thus simplifying the whole process: Steps expressed as systems of equations: Steps expressed as augmented matrices: Toggle slides back and forth to compare before and changes

Matrices and Gauss-Jordan Every step in the Gauss-Jordan elimination method can be expressed with matrices, rather than systems of equations, thus simplifying the whole process: Steps expressed as systems of equations: Steps expressed as augmented matrices: Toggle slides back and forth to compare before and changes

Matrices and Gauss-Jordan Every step in the Gauss-Jordan elimination method can be expressed with matrices, rather than systems of equations, thus simplifying the whole process: Steps expressed as systems of equations: Steps expressed as augmented matrices: Toggle slides back and forth to compare before and changes

Matrices and Gauss-Jordan Every step in the Gauss-Jordan elimination method can be expressed with matrices, rather than systems of equations, thus simplifying the whole process: Steps expressed as systems of equations: Steps expressed as augmented matrices: Toggle slides back and forth to compare before and changes

Matrices and Gauss-Jordan Every step in the Gauss-Jordan elimination method can be expressed with matrices, rather than systems of equations, thus simplifying the whole process: Steps expressed as systems of equations: Steps expressed as augmented matrices: Toggle slides back and forth to compare before and changes

Matrices and Gauss-Jordan Every step in the Gauss-Jordan elimination method can be expressed with matrices, rather than systems of equations, thus simplifying the whole process: Steps expressed as systems of equations: Steps expressed as augmented matrices: Toggle slides back and forth to compare before and changes

Matrices and Gauss-Jordan Every step in the Gauss-Jordan elimination method can be expressed with matrices, rather than systems of equations, thus simplifying the whole process: Steps expressed as systems of equations: Steps expressed as augmented matrices: Toggle slides back and forth to compare before and changes

Matrices and Gauss-Jordan Every step in the Gauss-Jordan elimination method can be expressed with matrices, rather than systems of equations, thus simplifying the whole process: Steps expressed as systems of equations: Steps expressed as augmented matrices: Toggle slides back and forth to compare before and changes Row Reduced Form of the Matrix