Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 1 ~ Linear Algebraic Equations ~ Gauss Elimination Chapter 9 Credit: Prof. Lale Yurttas, Chemical Eng., Texas A&M University

Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 2 Solving Systems of Equations A linear equation in n variables: a 1 x 1 + a 2 x 2 + … + a n x n = b For small (n ≤ 3), linear algebra provides several tools to solve such systems of linear equations: –Graphical method –Cramer’s rule –Method of elimination Nowadays, easy access to computers makes the solution of very large sets of linear algebraic equations possible

Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 3 Determinants and Cramer’s Rule [A] : coefficient matrix D : Determinant of A matrix

Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 4 Computing the Determinant

Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 5 Gauss Elimination Solve Ax = b Consists of two phases: –Forward elimination –Back substitution Forward Elimination reduces Ax = b to an upper triangular system Tx = b’ Back substitution can then solve Tx = b’ for x Forward Elimination Back Substitution

Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 6 Gaussian Elimination Forward Elimination x 1 - x 2 + x 3 = 6 3x 1 + 4x 2 + 2x 3 = 9 2x 1 + x 2 + x 3 = 7 x 1 - x 2 + x 3 = x 2 - x 3 = x 2 - x 3 = -5 x 1 - x 2 + x 3 = 6 0 7x 2 - x 3 = (4/7)x 3 =-(8/7) -(3/1) Solve using BACK SUBSTITUTION: x 3 = 2 x 2 =-1 x 1 =3 -(2/1) -(3/7)

Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 7 Back Substitution 1x01x0 +1x 1 –1x2–1x2 +4x 3 8= – 2x 1 –3x2–3x2 +1x 3 5= 2x22x2 – 3x 3 0= 2x32x3 4= x 3 = 2

Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 8 1x01x0 +1x 1 –1x2–1x2 0= – 2x 1 –3x2–3x2 3= 2x22x2 6= Back Substitution x 2 = 3

Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 9 1x01x0 +1x 1 3= – 2x 1 12= Back Substitution x 1 = –6

Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 1x01x0 9= Back Substitution x 0 = 9

Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 11 for i  n down to 1 do /* calculate x i */ x [ i ]  b [ i ] / a [ i, i ] /* substitute in the equations above */ for j  1 to i-1 do b [ j ]  b [ j ]  x [ i ] × a [ j, i ] endfor Back Substitution (* Pseudocode *) Time Complexity?  O(n 2 )

Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 12 Gaussian Elimination Forward Elimination

Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 13 Forward Elimination 4x04x0 +6x 1 +2x 2 – 2x 3 =8 2x02x0 +5x 2 – 2x 3 =4 –4x0–4x0 – 3x 1 – 5x 2 +4x 3 =1 8x08x0 +18x 1 – 2x 2 +3x 3 =40 -(2/4) MULTIPLIERSMULTIPLIERS -(-4/4) -(8/4)

Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 14 4x04x0 +6x 1 +2x 2 – 2x 3 =8 +4x 2 – 1x 3 =0 +3x 1 – 3x 2 +2x 3 =9 +6x 1 – 6x 2 +7x 3 =24 – 3x 1 -(3/-3) MULTIPLIERSMULTIPLIERS Forward Elimination -(6/-3)

Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 15 4x04x0 +6x 1 +2x 2 – 2x 3 =8 +4x 2 – 1x 3 =0 1x21x2 +1x 3 =9 2x22x2 +5x 3 =24 – 3x 1 ?? MULTIPLIERMULTIPLIER Forward Elimination

Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 16 4x04x0 +6x 1 +2x 2 – 2x 3 =8 +4x 2 – 1x 3 =0 1x21x2 +1x 3 =9 3x33x3 =6 – 3x 1 Forward Elimination

Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 17 Gaussian Elimination Operation count in Forward Elimination n TOTAL 1 st column: 2n(n-1)  2n 2 2(n-1) 2 2(n-2) 2 ……. b 11 b 22 b 33 b 44 b 55 b 66 b 77 b 66

Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 18 Pitfalls of Elimination Methods Division by zero It is possible that during both elimination and back-substitution phases a division by zero can occur. For example: 2x 2 + 3x 3 = x 1 + 6x 2 + 7x 3 = -3 A =467 2x 1 + x 2 + 6x 3 = 5216 Solution: pivoting (to be discussed later)

Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 19 Pitfalls (cont.) Round-off errors Because computers carry only a limited number of significant figures, round-off errors will occur and they will propagate from one iteration to the next. This problem is especially important when large numbers of equations (100 or more) are to be solved. Always use double-precision numbers/arithmetic. It is slow but needed for correctness! It is also a good idea to substitute your results back into the original equations and check whether a substantial error has occurred.

Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. ill-conditioned systems - small changes in coefficients result in large changes in the solution. Alternatively, a wide range of answers can approximately satisfy the equations. (Well-conditioned systems – small changes in coefficients result in small changes in the solution) Problem: Since round off errors can induce small changes in the coefficients, these changes can lead to large solution errors in ill-conditioned systems. Example: x 1 + 2x 2 = x 1 + 2x 2 = 10.4 x 1 + 2x 2 = x 1 + 2x 2 = 10.4 Pitfalls (cont.)

Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Surprisingly, substitution of the erroneous values, x 1 =8 and x 2 =1, into the original equation will not reveal their incorrect nature clearly: x 1 + 2x 2 = 108+2(1) = 10 (the same!) 1.1x 1 + 2x 2 = (8)+2(1)=10.8 (close!) IMPORTANT OBSERVATION: An ill-conditioned system is one with a determinant close to zero If determinant D=0 then there are infinitely many solutions  singular system Scaling (multiplying the coefficients with the same value) does not change the equations but changes the value of the determinant in a significant way. However, it does not change the ill-conditioned state of the equations! DANGER! It may hide the fact that the system is ill-conditioned!! How can we find out whether a system is ill-conditioned or not? Not easy! Luckily, most engineering systems yield well-conditioned results! One way to find out: change the coefficients slightly and recompute & compare ill-conditioned systems (cont.) –

Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 22 COMPUTING THE DETERMINANT OF A MATRIX USING GAUSSIAN ELIMINATION The determinant of a matrix can be found using Gaussian elimination. Here are the rules that apply: Interchanging two rows changes the sign of the determinant. Multiplying a row by a scalar multiplies the determinant by that scalar. Replacing any row by the sum of that row and the multiple of any other row does NOT change the determinant. The determinant of a triangular matrix (upper or lower triangular) is the product of the diagonal elements. i.e. D=t 11 t 22 t 33

Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Techniques for Improving Solutions

Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 24 Gauss-Jordan Elimination a 11 x 22 0 x x x x 66 x 77 x b 11 b 22 b 33 b 44 b 55 b 66 b 77 b 88 b x 99

Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 25 Gauss-Jordan Elimination: Example R2  R2 - (-1)R1 R3  R3 - ( 3)R1 Scaling R2: R2  R2/(-1) R1  R1 - (1)R2 R3  R3-(4)R2 Scaling R3: R3  R3/(18) R1  R1 - (7)R3 R2  R2-(-5)R3 RESULT: x 1 =8.45, x 2 =-2.89, x 3 =1.23 Time Complexity?  O(n 3 )

Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 26 Systems of Nonlinear Equations Locate the roots of a set of simultaneous nonlinear equations: Systems of Nonlinear Equations

Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. First-order Taylor series expansion of a function with more than one variable: The root of the equation occurs at the value of x 1 and x 2 where f 1(i+1) =0 and f 2(i+1) =0 Rearrange to solve for x 1(i+1) and x 2(i+1) Since x 1(i), x 2(i), f 1(i), and f 2(i) are all known at the i th iteration, this represents a set of two linear equations with two unknowns, x 1(i+1) and x 2(i+1) You may use several techniques to solve these equations

Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 28 General Representation for the solution of Nonlinear Systems of Equations Solve this set of linear equations at each iteration:  Jacobian Matrix

Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 29 Solution of Nonlinear Systems of Equations Solve this set of linear equations at each iteration: Rearrange:

Copyright © 2006 The McGraw-Hill Companies, Inc. Permission required for reproduction or display. n non-linear equations n unknowns to be solved: Jacobian Matrix: Need to solve this set of Linear Equations in each and every iterative solution of the original Non-Linear system of equations. For n unknowns, the size of the Jacobian matrix is n 2 and the time it takes to solve the linear system of equations (one iteration of the non-linear system) is proportional to O(n 3 ) There is no guarantee for the convergence of the non-linear system. There may also be slow convergence. In summary, finding the solution set for the Non-Linear system of equations is an extremely compute-intensive task. 30 Notes on the solution of Nonlinear Systems of Equations