G AUSSIAN E LIMINATION Presented by: Max Le Max Le Justin Lavorgna Data Structures II: Algorithm Design & Analysis Algorithm Design & Analysis

Gaussian Elimination Itinerary

Introduction: Introduction: Who was Carl Friedrich Gauss ? Who was Carl Friedrich Gauss ? What is Gaussian Elimination ? What is Gaussian Elimination ? Some Basic Terminology ? Some Basic Terminology ?

Main Presentation: Main Presentation: Backward Substitution method Backward Substitution method LU method LU method Compute Determinant of a matrix Compute Determinant of a matrix

Conclusion: Conclusion: Which Algorithm is more efficient ? Which Algorithm is more efficient ? Which Algorithm is more practical ? Which Algorithm is more practical ?

CARL FRIEDRICH GAUSS Who Was He?

Born: April 30 th, 1777 Brunswick Died: February 23 rd,1822 Göttingen One of the all time great German Mathematicians. His field of study consisted of most every aspect in mathematics today. One of the all time great German Mathematicians. His field of study consisted of most every aspect in mathematics today.

Gauss’s work contributed to a variety of different aspects such as: Gauss’s work contributed to a variety of different aspects such as: Gaussian Elimination (Linear Algebra) Gaussian Elimination (Linear Algebra) Gaussian Primes & Gauss Sums (Number Theory) Gaussian Primes & Gauss Sums (Number Theory) Gaussian Distribution (Statistics) Gaussian Distribution (Statistics) Gauss (Electromagnetism) Gauss (Electromagnetism) Gaussian Curvature & Gauss-Bonnet Formula (Differential Geometry) Gaussian Curvature & Gauss-Bonnet Formula (Differential Geometry) Gaussian Quadrature (Numerical Analysis) Gaussian Quadrature (Numerical Analysis) Gauss’s Identity (Hypergeometric Functions) Gauss’s Identity (Hypergeometric Functions)

GAUSSIAN ELIMINATION What is it?

For complex systems of equations where: The number of equations are equal to n The number of equations are equal to n The number of unknowns are equal to n The number of unknowns are equal to n We must solve by a process of elimination.

Elimination is implied by reducing the amount of unknowns and equations in the system.

1. Subtract multiples of the first equation from all other equations 1. Subtract multiples of the first equation from all other equations The elimination process: 2. The goal is to eliminate the first variables in each equation. variables in each equation. a11 a12 a13 a11 a12 a13 a21 a22 a23 a21 a22 a23 a31 a32 a33 a31 a32 a33 u11 u12 u13 u11 u12 u13 0 u22 u23 0 u22 u u u33 A= A= U= U=

TERMINOLOGY

The above conditions are what we call a linear equation. If h = 0 then the linear equation is said to be homogeneous. All linear equations make up a linear system while all homogenous linear equations make up a homogenous linear system. The above conditions are what we call a linear equation. If h = 0 then the linear equation is said to be homogeneous. All linear equations make up a linear system while all homogenous linear equations make up a homogenous linear system. Equation:ax + by + cz + dw = h Equation:ax + by + cz + dw = h where:a,b,c,d, and h are known numbers where:a,b,c,d, and h are known numbers while:x,y,z, and w are unknown numbers while:x,y,z, and w are unknown numbers

HOW IT WORKS

Given System: x + y + z = 0 x - 2y + 2z = 4 Equation 1 x + 2y - z = 2 Equation 3 Equation 2

Make system into a matrix: x + y + z | 0 x + y + z | 0 x - 2y + 2z | 4 x - 2y + 2z | 4 Equation 1 x + 2y - z | 2 x + 2y - z | 2 Equation 3 Equation 2

Objective: Kill ‘ X ’ Variable in Equation 2 x + y + z | 0 x + y + z | 0 x - 2y + 2z | 4 x - 2y + 2z | 4 Equation 1 x + 2y - z | 2 x + 2y - z | 2 Equation 3 Equation 2

1x +1y + 1z | 0 1x - 2y + 2z | 4 Multiple: 1 1x + 1y - 1z | 0 - __________ - 3y + z | 4 - 3y + z | 4 New Equation 2 Equation 1 Equation 2 Matrix 1x + 2y - 1z | 2 Equation 3 Step 1. Observe first column from left of the matrix. Step 1. Observe first column from left of the matrix. Step 2. If leading term is any number other than 1, multiply the row by its reciprocal to obtain a 1. Step 2. If leading term is any number other than 1, multiply the row by its reciprocal to obtain a 1. 1x Step 3. Subtract first row from all other rows until a zero appears below the leading column. zero appears below the leading column. ‘X’ unknown is DEAD… Equation 1 x + y + z | 0 x + y + z | 0 x - 2y + 2z | 4 x - 2y + 2z | 4 x + 2y - z | 2 x + 2y - z | 2

Objective: Kill ‘ X ’ Variable in Equation 3 x + y + z | 0 x + y + z | 0 x - 2y + 2z | 4 x - 2y + 2z | 4 Equation 1 x + 2y - z | 2 x + 2y - z | 2 Equation 3 Equation 2

1x +1y + 1z | 0 1x + 2y - 1z | 2 Multiple: 1 1x + 1y - 1z | 0 - __________ y - 2z | 2 y - 2z | 2 New Equation 3 Equation 1 Equation 3 Matrix - 3y + 1z | 4 Equation 2 Step 4. Observe first column from left of the matrix. Step 4. Observe first column from left of the matrix. Step 5. If leading term is any number other than 1, multiply the row by its reciprocal to obtain a 1. Step 5. If leading term is any number other than 1, multiply the row by its reciprocal to obtain a 1. 1x Step 6. Subtract first row from all other rows until a zero appears below the leading column. zero appears below the leading column. ‘X’ unknown is DEAD… Equation 1 x + y + z | 0 x + y + z | 0 x - 2y + 2z | 4 x - 2y + 2z | 4 x + 2y - z | 2 x + 2y - z | 2

Objective: Kill ‘ Y ’ Variable in Equation 3 x + y + z | 0 x + y + z | 0 x - 2y + 2z | 4 x - 2y + 2z | 4 Equation 1 x + 2y - z | 2 x + 2y - z | 2 Equation 3 Equation 2

1y - 2z | 2 - 3y + 1z | 4 Multiple: 3 3y - 6z | 6 - __________ - 5z | z | 10 New Equation 3 Equation 3 Equation 2 Matrix 1x +1y + 1z | 0 Equation 1 Step 7. Observe first column from left of the matrix. Step 7. Observe first column from left of the matrix. Step 8. If leading term is any number other than 1, multiply the row by its reciprocal to obtain a 1. Step 8. If leading term is any number other than 1, multiply the row by its reciprocal to obtain a 1. 1y Step 9. Add second row to the third row until a zero appears below the leading column. zero appears below the leading column. ‘Y’ unknown is DEAD… x + y + z | 0 x + y + z | 0 x - 2y + 2z | 4 x - 2y + 2z | 4 x + 2y - z | 2 x + 2y - z | 2

Objective: Solve for Unknowns using Backward Substitution

BACKWARD SUBSTITUTION

Linear System Solution: Final matrix: x + y + z | 0 x + y + z | 0 - 3y + z | 4 - 3y + z | 4 Equation 1 - 5z | z | 10 Equation 3 Equation 2 U = U =

Solving for z in equation 3: ___ ___ -2 z = Solving for y in equation 2: ___ ______ = -2 y ( + 2) = + (-2) Solving for x in equation 1: ___ _________ = 4 = 4 ( ) = + (-2) + (-2) x

Objective: Solve using LU Decomposition

LU DECOMPOSITION

LU Decomposition Information on LU Method Information on LU Method Example Example

LU Decomposition Given a system A*x = b Given a system A*x = b Solving the system Lp*y=b to solve for y. Solving the system Lp*y=b to solve for y. We want to find Lp and U matrices such that Lp*U=A We want to find Lp and U matrices such that Lp*U=A Then solving the system U*x=y to get x Then solving the system U*x=y to get x

OBJECTIVE: Given a system A*x = b Given a system A*x = b

LU Decomposition Consider the system: 2x + 2y + 1z 2x + 2y + 1z 2x + 3y - 2z 2x + 3y - 2z 4x + 1y - 2z 4x + 1y - 2z Equation in matrix form for A: A x z y X = b = = = =

Identity Matrix Identity Matrix is a matrix with all 1 in its diagonal and zeros elsewhere Identity Matrix is a matrix with all 1 in its diagonal and zeros elsewhere I = I =

Find the U matrix and keep track of Elementary matrices First Operation: Elementary operation equivalent: Multiply row 1 by -1 Add row 1 to row Multiply row 1 by -1 Add row 1 to row = E1 = E1

Find the U matrix and keep track of Elementary matrices Second Operation: Elementary operation equivalent: Multiply row 1 by -2 Add row 1 to row Multiply row 1 by -2 Add row 1 to row = E2 = E2

Find the U matrix and keep track of Elementary matrices Third Operation: Elementary operation equivalent: Multiply row 2 by 3 Add row 2 to row Multiply row 2 by 3 Add row 2 to row = E3 = E3 = U

E1 = E2 = E2 = S = S = E1*E2*E3 Forming the Lp Matrix

Product of the elementary matrices S = Inverse of a S will give Lp /S = = Lp

OBJECTIVE: We want to find Lp and U matrices such that Lp*U=A We want to find Lp and U matrices such that Lp*U=A

LU Decomposition We can verify that Lp*U = A = A = A Lp Lp U =

LU Decomposition Recall that Upper Triangular Matrix: U = U = Lp = Lp = Recall that Lower Triangular Matrix:

OBJECTIVE: Solving the system Lp*y=b to solve for y. Solving the system Lp*y=b to solve for y.

Solve Lp*y = b The system: Solve for y1, y2 and y3 y1 = 1 y1 = 1 y1+y2 = 1 y1+y2 = 1 2*y1-3*y2 + y3 = 1 2*y1-3*y2 + y3 = 1 b y1 y1 y3 y3 y2 y2 Y= = Lp Lp => y2 = 0 => y2 = 0 => y3 = -1 => y3 =

OBJECTIVE: Then solving the system U*x=y to get x Then solving the system U*x=y to get x

Solve U*x = y The system : Solve for x1, x2 and x3 -13*X3 = *X3 = -1 X2 – 3*X3 = 1 X2 – 3*X3 = 1 2*X1+2*X2 + X3 = 1 2*X1+2*X2 + X3 = 1 Y X1 X1 X3 X3 X2 X2 X= = U => X3 = 1/13 => X3 = 1/13 => X2 = 3/13 => X2 = 3/13 => X1 = 3/13 => X1 = 3/13

Objective: Solve for linear system using Determinant of Matrix

DETERMINANT of MATRIX

Make equations into a matrix: Given Equations: x + y + z = 0 x - 2y + 2z = 4 Equation 1 x + 2y - z = 2 Equation 3 Equation 2

How Determinant works: a 11 a 12 a 21 a 22 a 11 a 12 a 21 a 22 = det a11-a22 – a12a21 det A = Σ s j a 1j det A j det A = Σ s j a 1j det A j a 11 det [ a 22 ] – a 12 det [ a 21 ] = a 11 -a 22 – a 12 a 21 a 11 det [ a 22 ] – a 12 det [ a 21 ] = a 11 -a 22 – a 12 a 21 a11 det [ a22 ] – a12 det [ a21 ] = n j = 1 Formula:

Matrix: = detdet - det

det det- det = 1 ( 2 – 4 ) – 1 ( -1 – 2 ) + 1 (2 – (-2) ) = = 5 det det det

CONCLUSION

Time Complexity Backward substitution O(n^3) Backward substitution O(n^3) LU method O(n^3) LU method O(n^3)

Space Complexity Backward substitution O(n^2) Backward substitution O(n^2) LU method O(n^2) LU method O(n^2)

Application Gaussian Elimination plays a major role in solving Linear systems, including: Gaussian Elimination plays a major role in solving Linear systems, including: - Digital Signal processing - Linear system analysis - Image processing. - Etc…