E. T. S. I. Caminos, Canales y Puertos1 Engineering Computation Lecture 5

E. T. S. I. Caminos, Canales y Puertos2 1. Motivate Study of Systems of Equations and particularly Systems of Linear Equations 2. Review steps of Gaussian Elimination 3. Examine how roundoff error can enter and be magnified in Gaussian Elimination 4. Introduce Pivoting and Scaling as defenses against roundoff. 5. Consider what an engineer can do to generate well formulated problems. Learning Objectives for Lecture

E. T. S. I. Caminos, Canales y Puertos3 Systems of Equations In the previous lecture we have tried to determine the value x, satisfying f(x)=0. In this part we try to obtain the values x 1,x 2, x n, satisfying the system of equations: These systems can be linear or nonlinear, but in this part we deal with linear systems:

E. T. S. I. Caminos, Canales y Puertos4 Systems of Equations where a and b are constant coefficients, and n is the number of equations. Many of the engineering fundamental equations are based on conservation laws. In mathematical terms, these principles lead to balance or continuity equations relating the system behavior with respect to the amount of the magnitude being modelled and the extrenal stimuli acting on the system.

E. T. S. I. Caminos, Canales y Puertos5 Systems of Equations Matrices are rectangular sets of elements represented by a single symbol. If the set if horizontal it is called row, and if it is vertical, it is called column. Row 2 Column 3 Row vector Column vector

E. T. S. I. Caminos, Canales y Puertos6 Systems of Equations There are some special types of matrices: Symmetric matrix Identity matrix Diagonal matrix Upper triangular matrix

E. T. S. I. Caminos, Canales y Puertos7 Systems of Equations Banded matrix All elements are null with the exception of those in a band centered around the main diagonal. This matrix has a band width of 3 and has the name of tridiagonal. Half band width Lower triangular matrix

E. T. S. I. Caminos, Canales y Puertos8 Linear Algebraic Equations a 11 x 1 + a 12 x 2 + a 13 x 3 + … + a 1n x n = b 1 a 21 x 1 + a 22 x 2 + a 23 x 3 + … + a 2n x n = b 2 ….. a n1 x 1 + a n2 x 2 + a n3 x 3 + … + a n x n = b n where all a ij 's and b i 's are constants. In matrix form: n x nn x 1n x 1 or simply [A]{x} = {b} Systems of Equations

E. T. S. I. Caminos, Canales y Puertos9 Systems of Equations Matrix representation of a system Matrix product: Resulting dimensions

E. T. S. I. Caminos, Canales y Puertos10 Systems of Equations Graphic Solution: Systems of equations are hyperplanes (straight lines, planes, etc.). The solution of a system is the intersection of these hyperplanes. Compatible and determined system. Vectors are linearly independent. Unique solution. Determinant of A is non-null.

E. T. S. I. Caminos, Canales y Puertos11 Systems of Equations Incompatible system, Linearly dependent vectors. Null determinant of A. There is no solution. Compatible but undetermined system. Linearly dependent vectors. Null determinant of A. There exists an infinite number of solutions.

E. T. S. I. Caminos, Canales y Puertos12 Systems of Equations Compatible and determined system. Linearly independent vectors. Nonnull determinant of A, but close to zero. There exists a solution but it is difficult to find precisely. It is an ill conditioned system leading to numerical errors.

E. T. S. I. Caminos, Canales y Puertos13 Gauss elimination Naive Gauss elimination method: The Gauss’ method has two phases: Forward elimination and backsustitution. In the first, the system is reduced to an upper triangular system: First, the unknown x 1 is eliminated. To this end, the first row is multiplied by - a 21 /a 11 and added to the second row. The same is done with all other succesive rows (n-1 times) until only the first equation contains the first unknown x 1. Pivot equation substract pivot

E. T. S. I. Caminos, Canales y Puertos14 Gauss elimination This operation is repeated with all variables xi, until an upper triangular matrix is obtained. Next, the system is solved by backsustitution. The number of operations (FLOPS) used in the Gauss method is: Pass 1Pass 2

E. T. S. I. Caminos, Canales y Puertos15 b. By elementary row manipulations, reduce [A|b] to [U|b'] where U is an upper triangular matrix: DO i = 1 to n-1 DO k = i+1 to n Row(k) = Row(k) - (a ki /a ii ) * Row(i) ENDDO 1. Forward Elimination (Row Manipulation): a. Form augmented matrix [A|b]: Gauss elimination

E. T. S. I. Caminos, Canales y Puertos16 Gauss elimination 2. Back Substitution Solve the upper triangular system [U]{x} = {b´} x n = b' n / u nn DO i = n-1 to 1 by (-1) END

E. T. S. I. Caminos, Canales y Puertos17 Consider the system of equations To 2 significant figures, the exact solution is: We will use 2 decimal digit arithmetic with rounding. Gauss elimination (example)

E. T. S. I. Caminos, Canales y Puertos18 Start with the augmented matrix: Multiply the first row by –1/50 and add to second row. Multiply the first row by –2/50 and add to third row: Multiply the second row by –6/40 and add to third row: Gauss elimination (example)

E. T. S. I. Caminos, Canales y Puertos19 Now backsolve: (vs ,  t = 2.2%) (vs ,  t = 2.5%) (vs ,  t = 0%) Gauss elimination (example)

E. T. S. I. Caminos, Canales y Puertos20 Consider an alternative solution interchanging rows: After forward elimination, we obtain: Now backsolve: x 3 = (vs , e t = 4.4%) x 2 = (vs , e t = 50%) x 1 = (vs , e t = 100%) Apparently, the order of the equations matters! Gauss elimination (example)

E. T. S. I. Caminos, Canales y Puertos21 WHAT HAPPENED? When we used 50 x x x 3 = 1 to solve for x 1, there was little change in other equations. When we used 2 x x x 3 = 3 to solve for x 1 it made BIG changes in the other equations. Some coefficients for other equations were lost! The second equation has little to do with x 1. It has mainly to do with x 3. As a result we obtained LARGE numbers in the table, significant roundoff error occurred and information was lost. Things didn't go well! If scaling factors | a ji / a ii | are  1 then the effect of roundoff errors is diminished. Gauss elimination (example)

E. T. S. I. Caminos, Canales y Puertos22 Effect of diagonal dominance: As a first approximation roots are: x i  b i / a ii Consider the previous examples: Gauss elimination (example)

E. T. S. I. Caminos, Canales y Puertos23 Goals: 1. Best accuracy (i.e. minimize error) 2. Parsimony (i.e. minimize effort) Possible Problems: A. Zero on diagonal term  ÷ by zero. B. Many floating point operations (flops) cause numerical precision problems and propagation of errors. C. System may be ill-conditioned: det[A]  0. D. No solution or an infinite # of solutions: det[A] = 0. Possible Remedies: A. Carry more significant figures (double precision). B. Pivot when the diagonal is close to zero. C. Scale to reduce round-off error. Gauss elimination (example)

E. T. S. I. Caminos, Canales y Puertos24 PIVOTING A. Row pivoting (Partial Pivoting) - In any good routine, at each step i, find max k | a ki | for k = i, i+1, i+2,..., n Move corresponding row to pivot position. (i) Avoids zero a ii (ii) Keeps numbers small & minimizes round-off, (iii) Uses an equation with large | a ki | to find x i  Maintains diagonal dominance.  Row pivoting does not affect the order of the variables.  Included in any good Gaussian Elimination routine. Gauss elimination (pivoting)

E. T. S. I. Caminos, Canales y Puertos25 B. Column pivoting - Reorder remaining variables x j for j = i,...,n so get largest | a ji | Column pivoting changes the order of the unknowns, x i, and thus leads to complexity in the algorithm. Not usually done. C. Complete or Full pivoting Performing both row pivoting and column pivoting. (If [A] is symmetric, needed to preserve symmetry.) Gauss elimination (pivoting)

E. T. S. I. Caminos, Canales y Puertos26 How to fool pivoting: Multiply the third equation by 100 and then performing pivoting will yield: Forward elimination then yields (2-digit arithmetic): Backsolution yields: x 3 = (vs , e t = 4.4%) x 2 = (vs , e t = 50.0%) x 1 = (vs , e t = 100%) The order of the rows is still poor!! Gauss elimination (pivoting)

E. T. S. I. Caminos, Canales y Puertos27 SCALING A. Express all equations (and variables) in comparable units so all elements of [A] are about the same size. B. If that fails, and max j |a ij | varies widely across the rows, replace each row i by: a ij  This makes the largest coefficient |a ij | of each equation equal to 1 and the largest element of [A] equal to 1 or -1 NOTE: Routines generally do not scale automatically; scaling can cause round-off error too! SOLUTIONS Don't actually scale, but use hypothetical scaling factors to determine what pivoting is necessary. Scale only by powers of 2: no roundoff or division required. Gauss elimination (scaling)

E. T. S. I. Caminos, Canales y Puertos28 If the units of x 1 were expressed in µg instead of mg the matrix might read: How to fool scaling: A poor choice of units can undermine the value of scaling. Begin with our original example: Scaling then yields: Which equation is used to determine x 1 ? Why bother to scale ? Gauss elimination (scaling)

E. T. S. I. Caminos, Canales y Puertos29 OPERATION COUNTING In numerical scientific calculations, the number of multiplies & divides often determines CPU time. (This represents the numerical effort!) One floating point multiply or divide (plus any associated adds or subtracts) is called a FLOP. (The adds/subtracts use little time compared to the multiplies/divides.) FLOP = FLoating point OPeration. Examples:a * x + b a / x – b Gauss elimination (operation counting)

E. T. S. I. Caminos, Canales y Puertos30 Useful identities in counting FLOPS: O(m n ) means that there are terms of order m n and lower. Gauss elimination (operation counting)

E. T. S. I. Caminos, Canales y Puertos31 Simple Example of Operation Counting: DO i = 1 to n Y(i) = X(i)/i – 1 ENDDO X(i) and Y(i) are arrays whose values change when i changes. In each iteration X(i)/i – 1 represents one FLOP because it requires one division (& one subtraction). The DO loop extends over i from 1 to n iterations: Gauss elimination (operation counting)

E. T. S. I. Caminos, Canales y Puertos32 Another Example of Operation Counting: DO i = 1 to n Y(i) = X(i) X(i) + 1 DO j = i to n Z(j) = [ Y(j) / X(i) ] Y(j) + X(i) ENDDO With nested loops, always start from the innermost loop. [Y(j)/X(i)] * Y(j) + X(i) represents 2 FLOPS Gauss elimination (operation counting)