Chapter 10: Solving Linear Systems of Equations Introduction Requirements for a solution Gaussian elimination This covers Ch. 8 of the Numerical Methods with MATLAB: Implementation and Application book.

Introduction Motivational example: Pump Curve Model Objective: Find the coefficients of the quadratic equation that approximates the data from a pump. Curve model equation: ℎ= 𝑐 1 𝑞 2 + 𝑐 2 𝑞+ 𝑐 3 How to identify 𝑐 1 , 𝑐 2 , 𝑐 3 ? Three data points: 1× 10 −4 ,115 8× 10 −4 ,110 (1.4× 10 −3 ,92.5) ℎ: 𝑞:

Introduction Substitute each pair of data points into the model equation: Rewrite in matrix form as:

Introduction Using more compact symbolic notation: 𝐴𝑥=𝑏 where 𝐴= 1× 10 −8 1× 10 −4 1 64× 10 −8 8× 10 −4 1 196× 10 −8 14× 10 −4 1 , 𝑥= 𝑐 1 𝑐 2 𝑐 3 , 𝑏= 115 110 92.5 Our goal: How to numerically solve for the unknown vector 𝑥?

Introduction Solving systems of equations is one of the most widely automated computing tasks. In this chapter, we will focus on: setting up a system of equations; using the built-in capability of MATLAB to solve the system; developing and understanding the numerical complications that affect the reliability of the computed result.

Introduction The generic system of equations is 𝐴𝑥=𝑏: 𝐴 is an 𝑚×𝑛 matrix of known real numbers; 𝑥 is a length-𝑛 vector of unknowns to be determined; 𝑏 is a length-𝑚 vector of known real numbers. Before solving, we always need to go from the physical problem to the general linear-algebra form (𝐴, 𝑥, and 𝑏). Only consider linear equations here…

Introduction How to put system equations into the standard form? Recommended procedure: Write the equations in their natural form. Identify unknowns, and order them. Isolate the unknowns. Write equations in matrix form.

Requirements for a Solution The nature of the solution to 𝐴𝑥=𝑏 depends on the shape of matrix 𝐴: ∎ ∎ ∎ ∎ ∎ ∎ ∎ ∎ ∎ 𝑚=𝑛 ∎ ∎ ∎ ∎ ∎ ∎ ∎ ∎ ∎ ∎ ∎ ∎ 𝑚>𝑛 ∎ ∎ ∎ ∎ ∎ ∎ ∎ ∎ ∎ ∎ ∎ ∎ 𝑚<𝑛 Solving 𝐴𝑥=𝑏 for 𝑛 unknowns related by 𝑛 equations. Overdetermined systems (more equations than unknowns) Underdetermined systems (less equations than unknowns).

Requirements for a Solution Consistency: the Elements of the unknown vector 𝑥 are the coefficients in the linear combination of the columns of 𝐴 that yield the vector 𝑏. 𝐴𝑥=𝑏 ↔ 𝑥 1 𝑎 11 ⋮ 𝑎 𝑚1 + 𝑥 2 𝑎 12 ⋮ 𝑎 𝑚2 +…+ 𝑥 𝑛 𝑎 1𝑛 ⋮ 𝑎 𝑚𝑛 = 𝑏 1 ⋮ 𝑏 𝑚 If an exact solution to Ax = b exists, b must lie in the column space of A. If it does, then the system is said to be consistent. Otherwise, the system is said to be inconsistent.

Requirements for a Solution How to check if a system is consistent or not? Can be expressed by forming the augmented matrix 𝐴 from the columns of 𝐴 and the vector 𝑏: 𝐴 = 𝐴 𝑏 = 𝑎 11 ⋮ 𝑎 𝑚1 𝑎 12 ⋮ 𝑎 𝑚2 … 𝑎 1𝑛 ⋮ 𝑎 𝑚𝑛 𝑏 1 ⋮ 𝑏 𝑚 𝐴 and 𝑏 are consistent if and only if 𝐴 and 𝐴 have the same rank, i.e. rank( 𝐴 )=rank (𝐴). If rank( 𝐴 )>rank (𝐴), the vector 𝑏 does not lie in the column space of 𝐴. An exact solution is therefore not possible.

Requirements for a Solution Role of 𝑟𝑎𝑛𝑘(𝐴): If 𝑟𝑎𝑛𝑘(𝐴) = 𝑛 and the system is consistent, the solution exists and it is unique. If 𝑟𝑎𝑛𝑘(𝐴) < 𝑛 and the system is consistent, an infinite number of solutions exist. If the system is inconsistent, no solution exists. E.g., 1 1 2 1 3 1 𝑥 1 𝑥 2 = 2 1 0 rank (A)=rank([A b]): the system is consistent, and solution exists. rank(A)=2=# of unknowns: the solution is unique.

Requirements for a Solution Special case, 𝒎=𝒏: For 𝑛×𝑛 systems, the rank of 𝐴 tells us a lot. If 𝑟𝑎𝑛𝑘(𝐴)=𝑛, the columns of 𝐴 span the 𝑹 𝑛 space, and hence any length-𝑛 vector lies in the column space of 𝐴. In other words, 𝑟𝑎𝑛𝑘([𝐴 𝑏])=𝑛 for any vector 𝑏. For a system of 𝑛 equations and 𝑛 unknowns in the form 𝐴𝑥=𝑏, the solution 𝑥 exists and is unique for any 𝑏 if and only if 𝑟𝑎𝑛𝑘(𝐴)=𝑛.

Requirements for a Solution Again, for 𝑛×𝑛 systems, the unique solution is 𝑥= 𝐴 −1 𝑏 if and only if 𝐴 −1 exists. Several ways to check if 𝐴 −1 exists or not (equivalent conditions): 𝑟𝑎𝑛𝑘(𝐴)=𝑛 (𝐴 is a full rank matrix) det⁡(𝐴)≠0 The columns of 𝐴 are linearly independent. The rows of 𝐴 are linearly independent.

Requirements for a Solution If 𝐴 −1 exists, 𝐴𝑥=𝑏 → 𝑥= 𝐴 −1 𝑏 Lots of computer tools allow you to compute 𝐴 −1 directly, so then you can find 𝑥 . However, there are many reasons you may not want to solve equations this way: more roundoff error, more operations (computations), … We use Gaussian elimination to solve the system. Not only saves computer time, also produces a result with less roundoff error (fewer operations). The matrix inverse 𝐴 −1 is more of a conceptual device than a computational tool here. People don’t normally use it to solve linear systems.

Gaussian Elimination To start, let us consider a special case: Triangular systems: Lower triangular systems: 𝑎 𝑖𝑗 =0 for all 𝑖<𝑗 𝐴= 𝑎 11 𝑎 21 ⋮ 𝑎 𝑛1 0 𝑎 22 ⋮ 𝑎 𝑛2 … 0 0 ⋮ 𝑎 𝑛𝑛 Upper triangular systems: 𝑎 𝑖𝑗 =0 for all 𝑖>𝑗 𝐴= 𝑎 11 0 ⋮ 0 𝑎 12 𝑎 22 ⋮ 0 … 𝑎 1𝑛 𝑎 2𝑛 ⋮ 𝑎 𝑛𝑛

Gaussian Elimination Solving triangular systems: (upper-triangular) Equivalent to solve: Solve in backward order (last equation is solved first):

Gaussian Elimination Solving for 𝑥 𝑛 , 𝑥 𝑛−1 ,…, 𝑥 1 for an upper triangular system is called backward substitution.

Gaussian Elimination Solving triangular systems: (lower-triangular) 𝐴= 4 0 0 3 −2 0 −2 1 2 , b= 8 −1 9 Equivalent to solve: 4 𝑥 1 =8 3 𝑥 1 −2 𝑥 2 =−1 −2 𝑥 1 + 𝑥 2 +2 𝑥 3 =9 Solve in forward order (first equation is solved first): 𝑥 1 =8/4=2 𝑥 2 =(3 𝑥 1 +1)/2=7/2 𝑥 3 =(9+2 𝑥 1 − 𝑥 2 )/2=19/4

Gaussian Elimination Solving for 𝑥 1 , 𝑥 2 ,…, 𝑥 𝑛 for a lower triangular system is called forward substitution.

Gaussian Elimination How to solve a general equation by hand? Subtract 2 times the first equation from the second equation This equation is now in upper-triangular form, and can be solved by backward substitution. The elimination phase transforms the matrix and right hand side to an equivalent system. The two systems have the same solution.

Gaussian Elimination Goal is to transform an arbitrary, square system into the equivalent upper triangular system so that it may be easily solved with backward substitution. We need to perform the Gaussian elimination to [𝐴 𝑏], the augmented matrix (not just 𝐴)!

Gaussian Elimination For example, to solve: Form the augmented system: Add 2 times row 1 to row 2, and add (1 times) row 1 to row 3: 𝐴 (1) Subtract (1 times) row 2 from row 3: 𝐴 (2)

Gaussian Elimination The transformed system is now in upper triangular form: Solve by back substitution to get:

Gaussian Elimination The procedure for Gaussian elimination: Pivot: 𝑎 𝑖𝑖 , ratio: 𝑎 𝑘𝑖 / 𝑎 𝑖𝑖 𝑎 𝑘𝑗 = 𝑎 𝑘𝑗 −( 𝑎 𝑘𝑖 / 𝑎 𝑖𝑖 )∗ 𝑎 𝑖𝑗 𝐴 = 𝐴 𝑏 = 𝑎 11 𝑎 12 𝑎 13 𝑎 14 𝑏 1 𝑎 21 𝑎 22 𝑎 23 𝑎 24 𝑏 2 𝑎 31 𝑎 41 𝑎 32 𝑎 42 𝑎 33 𝑎 34 𝑏 3 𝑎 43 𝑎 44 𝑏 4 Column 𝑖=1 Column 𝑖=2 Column 𝑖=3 Row 𝑖=1 Row 𝑖=2 Row 𝑖=3 ×( 𝑎 41 / 𝑎 11 ) ×( 𝑎 31 / 𝑎 11 ) ×( 𝑎 21 / 𝑎 11 ) ×( 𝑎 43 ′′ / 𝑎 33 ′′ ) ×( 𝑎 32 ′ / 𝑎 22 ′ ) ×( 𝑎 42 ′ / 𝑎 22 ′ ) Row 𝑘=2 Row 𝑘=3 Row 𝑘=4

Gaussian Elimination Note : This process requires that all the diagonal elements are nonzero! (Otherwise the ratio 𝑎 𝑘𝑗 / 𝑎 𝑖𝑖 is undefined since denominator is zero.) The GaussianElimination function uses Gaussian elimination to solve a system of equations. >> GaussianElimination(A,b) ans = 2 -1

Gaussian Elimination Unfortunately, the Gaussian elimination algorithm does not work always… One counterexample: matrix 𝐴 is full rank, a unique solution exists. Now form the augmented system:

Gaussian Elimination Start the elimination for the pivot location 𝑎 11 : Subtract 1/2 times the first row from the second row; Add 3/2 times the first row to the third row; Add 1/2 times the first row to the fourth row. The next stage of Gaussian elimination will not work, because there is a zero in the pivot location 𝑎 22 .

Gaussian Elimination What to do if we run into this situation? A smart way is to swap the second and fourth rows of the augmented matrix. Why does this work? Because we are just re-ordering the unknowns here, and not changing the system. This row exchanging method to avoid zeros at pivot locations is called Gaussian Elimination with Pivoting.

Gaussian Elimination Continue with elimination: subtract (1 times) row 2 from row 3. Another zero has appear in the pivot position. Swap row 3 and row 4. The augmented system is now ready for backward substitution.

Gaussian Elimination The GaussianEliminationPivot function uses Gaussian elimination with pivoting to solve a system of equations.

Gaussian Elimination Ex: >> GaussianEliminationPivot(A,b) Swap rows 1 and 3; new pivot = -3 Swap rows 2 and 3; new pivot = 2 ans = 1.0000 2.0000 3.0000 4.0000