Engineering Computation Part 1 E. T. S. I. Caminos, Canales y Puertos

Introduction Numerical methods are techniques which allow us to formulate mathematical problems to be solved by basic arithmetic operations. They are able to handle large systems of equations, non-linearities, complicated geometries and allow us solving engineering problems which have no analytical solution. The intelligent use of commercial software is favored by the knowledge of numerical methods. They allow us to design ad hoc programs to solve concrete problems. The allow us to become familiar with computers and understand the way they work. They are a good tool to reinforce the mathematical knowledge, because one of their aims consists in transforming complicate problems into simple arithmetic operations. E. T. S. I. Caminos, Canales y Puertos

Introduction Roots of equations : They are methods to solve They try to find the values of a variable for it to satisfy one equation. They are very useful in engineering projects, because in many occasions it is not possible to solve the design equations analytically. E. T. S. I. Caminos, Canales y Puertos

Introduction Systems of linear equations: They are methods looking for the set of values that simultaneously satisfy a system of algebraic equations. Calculus of structures, electric circuits, supply networks, fit of curves, etc. E. T. S. I. Caminos, Canales y Puertos

Introduction Optimization: Determine the value x0 leding to the optimal value of f(x). They can be subject to constraints. E. T. S. I. Caminos, Canales y Puertos

Introduction Fitting curves. Fitting techniques can be divided into two groups: Regression. It is used when one has errors in the experimental data. One looks for the curve showing the trend of the data. Interpolation. It is used to fit tabulated data and predict intermediate values or extrapolated data. E. T. S. I. Caminos, Canales y Puertos

Introduction Integration: Determine the area below a given curve. It has many applications in engineering. Calculus of centers of gravity. Calculus of areas, volumes, etc. It can also be used to solve differential equations. f ( x ) I n t e g r a l b E. T. S. I. Caminos, Canales y Puertos

Introduction Ordinary differential equations : They are very important because many problems can be stated in terms of variations and not in terms of magnitudes. There are two types of problems: Initial value problems, and boundary value problems. E. T. S. I. Caminos, Canales y Puertos

Introduction Partial differential equations: Used for characterizing engineering problems where the behavior of the physical magnitude can be expressed in terms of speed change with respect to two or more variables. Approximation by finite differences or the finite element method. E. T. S. I. Caminos, Canales y Puertos

Mathematical Models A mathematical expression of a given model can be Analytic solution (t=0, v=0): Approximate solution: E. T. S. I. Caminos, Canales y Puertos

Mathematical Models To solve the problem numerically, one replaces the derivative by a divided finite difference, tus transforming the problem into a very simple one containing only simple algebraic operations: E. T. S. I. Caminos, Canales y Puertos

Numerical Differentiation Forward: Centered: How big a step size should we select? One- or two-sided formula: What are the advantages of each? How is optimal step size affected by: - precision of numerical calculations? - precision with which f is computed? - curvature of function f near x=1? - choice of formula? E. T. S. I. Caminos, Canales y Puertos

APPROXIMATION AND ERRORS Numerical Methods Instead of solving for the exact solution we solve math problems with a series of arithmetic operations. Example: dx analytical solution: ln(b) – ln(a) numerical solution e. g., Trapezoidal Rule Error Analysis (a) identify sources of error (b) estimate the magnitude of the error (c) determine how to minimize and control error E. T. S. I. Caminos, Canales y Puertos

Mathematical Models Comparing solutions: Numerical solution, Dt=2seg 2 4 6 8 10 12 14 16 18 20 30 40 50 60 Numerical solution, Dt=1seg Numerical solution, Dt=2seg T (sec) V (m/sec) Exact solution E. T. S. I. Caminos, Canales y Puertos

Approximations and Rounding Errors Unfortunately, computers introduce errors in calculations. However, since many engineering problems have not analytical solution, we are forced to used numerical methods (approximations). The only noption we have is to accept the error and try to reduce it up to a tolerable level. The only way of minimizing the errors is by knowing and understanding why they occur and how we can diminish them. The most frequent errors are: Rounding errors, due to the fact that computers can work only with a finite representation of numbers. Truncation errors, due to differences between the exact and the approximate (numeric) formulations of the mathematical problem being dealt with. Before centering in each one of them, we will see two important concepts on the computer representation of numbers. E. T. S. I. Caminos, Canales y Puertos

Approximations and Rounding Errors Significant figures of a number: The significant figures of a number are those that can be used with confidence. The speedometer and the odometer in the figure estimate up to three and seven significant figures, respectively, 49.5 and 87324.45. This concept has two important implications: 1. An approximation is acceptable when it is exact for a given number of significant figures. 2. There are magnitudes or constants that cannot be represented exactly. E. T. S. I. Caminos, Canales y Puertos

Approximations and Rounding Errors Accuracy closeness of measured/computed values to the "true" value (vs. inaccuracy or bias) Bias systematic deviation from truth, "general trend" Precision closeness of measured/computed values with each other (spread or scatter), relates to the number of significant figures (vs. imprecision or uncertainty) E. T. S. I. Caminos, Canales y Puertos

Approximations and Rounding Errors Accuracy and precision: The errors associated with nd measurements can be characterized observing their accuracy and precision. Accuracy refers to how close the value is to the true value. Precision refers to how close are different measured values using the same method. Numerical methods must be sufficiently exact (without bias) and precise to satisfy the requirements of engineering problems. From now on we will refer to error to refer to the inaccuracy and lack of precision of our predictions. E. T. S. I. Caminos, Canales y Puertos

Approximations and Rounding Errors (a) inaccurat imprecise (b) accurate imprecise (c) Inaccurate precise (d) Accurate precise E. T. S. I. Caminos, Canales y Puertos

Approximations and Rounding Errors Error definitions: True value = approximation + absolute error. Absolute error = true value - approximation . Relative error = absolute error / true value . In real cases not always one can know the true value, thus: In many occasions, the error is calculated as the difference between the previous and the actual approximations. E. T. S. I. Caminos, Canales y Puertos

Approximations and Rounding Errors Thus, the stopping criterium of a numerical method can be: It is convenient to relate the errors with the number of significant figures.If the following relation holds, one can be sure that at least n significant figures are correct. E. T. S. I. Caminos, Canales y Puertos

Approximations and Rounding Errors Numerical systems: A numerical system is a convention to represent quantities. Since we have 10 fingers in our hands, our most familiis the numerical system which basis is 10. It uses 10 different digits. However, computers, due to the memory structure can only store two digits: 0 and 1. Thus, they use the binary system of numeric representation. E. T. S. I. Caminos, Canales y Puertos

Round-off Errors Background: How are numbers stored in a computer? The fundamental unit, a "word," consists of a string of "bits" (binary digits). Because computers are made up of gates or switches which are either closed or open, we work in binary or base 2 system. Example: An 8 bit word representation of the integer "35" is 00100011 or 0 0 1 0 0 0 1 1 ± 26 25 24 23 22 21 20 + 0x26 1x25 0x24 0x23 0x22 1x21 1x20 = 35 32 2 1 Note: We can only represent a finite # of numbers; for our case: –127 to +127 (127 = 27 – 1) or a total of 255 numbers (including 0) E. T. S. I. Caminos, Canales y Puertos

Approximations and Rounding Errors Representation of integer numbers: To represent base 10 numbers in binary form the signed magnitude method is used. The first digit stores the sign (0, positive and 1, negative). The remaining bits are used to store the number. A computer working with words of 16 bits can store integer numbers in the range -32768 to 32767. E. T. S. I. Caminos, Canales y Puertos

Approximations and Rounding Errors Floating point representation: This representation is used for fractional quantities. It has the fraction part, called mantissa, and an integer part, called exponent or characteristic. The mantissa is usually normalized, so that the value of m is limited: E. T. S. I. Caminos, Canales y Puertos

Approximations and Rounding Errors Hipothetical set: We assume a hipothetical set of floating point numbers for a machine using 7 bits as word. The smallest number that can be represented is 0.0625, and the largest is 7. E. T. S. I. Caminos, Canales y Puertos

Approximations and Rounding Errors E. T. S. I. Caminos, Canales y Puertos

Round-off Errors EXAMPLE: “Our” Base 10 Machine: This "Machine" carries three significant decimal digits and one exponent digit. ± 0.DDDE ± e with: 0 < e < 9 100 < DDD < 999 We represent - 0.02778 in our machine: - 0.277E-1 w/ chopping t = 0.29% - 0.278E-1 w/ rounding t = 0.07% (most computers round, but some chop to save time) E. T. S. I. Caminos, Canales y Puertos

Round-off Errors Consequences of Computer Representation of #'s: Limited Range of quantities can be represented because there are a finite # of digits in both the mantissa and the exponent. smallest positive # 0.100E-9 or 10-10 largest positive # 0.999E+9 almost 109 Finite number of floating-point values can be represented within the above range. ± (0.100 to 0.999) E-9 to E+9 2 x 900 #'s x 19 #'s = 34,200 #'s plus zero ( 0.000 E±e ) Thus, 34,201 #'s can be represented within the above range (zero appears only once). E. T. S. I. Caminos, Canales y Puertos

Consequences of Computer Representation of #'s The interval between successive represented numbers (quantization error) is not uniform, and increases as the numbers grow in magnitude. - "quantization" error, Dx, is proportional to the magnitude x. | x | = 0.01 | x | = 10.0 - normalizing | Dx | by | x | gives an approximation of "machine epsilon" - p, with both chopping and rounding, is represented by 0.314E+1 with:"quantizing error" = E. T. S. I. Caminos, Canales y Puertos

Approximations and Rounding Errors Conclusions: There is a limited range to represent quantities. The limits correspond to “underflow” and “overflow”. There is a finite number of quantities that can be represented on a given range. E. T. S. I. Caminos, Canales y Puertos

Approximations and Rounding Errors The interval between numbers increases as the numbers grow in magnitude. This means that the errors are proportional to the magnitude of the number to be represented. Analyzing the relative error one gets: where e is the machine epsilon, that is the worst possible relative error. with chopping With rounding E. T. S. I. Caminos, Canales y Puertos

Machine Epsilon: Smallest number such that: 1 + e > 1 Machine Epsilon on "Our Base 10 Machine" Try 0.01 - - - > 0.100 E+1 + 0.100 E–1 = 0.101 E=1 1.0 + 0.01 = 1.01 Try 0.00999 - - - > 0.100 E+1 + 0.999 E-2 = 0.100 E+1 1.0 + 0.00999 = 1.00999 after chopping ==> 1.00 Therefore, Machine epsilon is 0.01; Quick formula: e = b1- t = 101-3 = 0.01 t = # of sig. figs. in mantissa E. T. S. I. Caminos, Canales y Puertos

Round-off Error due to Arithmetic Operations Trouble happens when you add (or subtract) a small number to a large number: For "our machine": 1,200 + 4 + 6  1,210 Working left-to-right: 1200 + 4 = 1204 which is chopped to 1200 1200 = 0.120E+4 1200 + 6 = 1206 which is chopped to 1200 Reverse the order: 4 + 6 = 10 which is ok 10 + 1200 = 1210 which is ok Remedy: add or subtract small numbers together first. E. T. S. I. Caminos, Canales y Puertos

Round-off Error due to Arithmetic Operations Subtractive Cancellation (subtracting numbers of almost equal size) – too few significant figures left Consider formulas such as : Problems arise when we try to subtract two numbers of almost equal size 32,232 – 32,181 = 51 (Little precision left.) Our Machine ("chopper"): 0.322 E+5 – 0.321 E+5 32200 – 32100 = 100 With rounding: 0.322 E+5 – 0.322 E+5 32200 – 32200 = 0 E. T. S. I. Caminos, Canales y Puertos

Round-off Error due to Arithmetic Operations Smearing Occurs when individual terms are larger than summation itself. Consider the exponential series with x = -10 Consider formulas such as: With 7-decimal-digit accuracy: exact answer = 4.54 10-05 computed answer = – 6.26 10-05 (differs from book value) (45 terms) wrong sign ! Largest intermediate terms are: 9th = –2,755.732 & 10th = 2,755.732 E. T. S. I. Caminos, Canales y Puertos

Modern Computers (IEEE standard) Single Precision: 32 BIT WORDS 24 BITS assigned to mantissa (including the sign bit) 8 BITS to signed exponent Double Precision: 64 BIT WORDS 56 BITS assigned to mantissa (including the sign bit) 8 BITS to signed exponent (not changed) 223 = 8,400,000 or almost 7 full decimal digits for single 255 = 4x1016 or almost 17 decimal digits for double 27 – 1 = 127; 2+127 ~ 2x1038 for both single and double E. T. S. I. Caminos, Canales y Puertos

Truncation Error Error caused by the nature of the numerical technique employed to approximate the solution. Example: Maclaurin series expansion of ex If we use a truncated version of the series: Then the Truncation Error is: E. T. S. I. Caminos, Canales y Puertos

Approximations and Rounding Errors Precautions: Sums of large and small numbers: due to equaling the exponent. They are common in sums of infinite series where the individual terms are very small when compared with the accumulated sum. This error can be reduced by summing first the small terms and using double precision. Cancellation of the subtraction: The subtraction of very similar numbers. Smearing: The individual terms are larger than the total sum. Inner products: They are prone to rounding errors. Thus, it is convenient to use double precision in this type of calculations. E. T. S. I. Caminos, Canales y Puertos

Taylor Series Expansion Basic Idea: Predict the value of a function, ƒ, at a point xi+1 based on the value of the function and all of its derivatives, ƒ, ƒ', ƒ",… at a neighboring point xi Given xi, ƒ(xi), ƒ'(xi), ƒ"(xi), ... ƒn+1(xi), we can predict or approximate ƒ(xi+1) E. T. S. I. Caminos, Canales y Puertos

Taylor Series Expansion General Form (Eq. 4.7 in C&C): h = "step size" = xi+1 – xi Rn = remainder to account for all other terms with xi    xi+1 = O (hn+1) with x not exactly known "on the order of hn+1 " Note: f(x) must be a function with n+1 continuous derivatives E. T. S. I. Caminos, Canales y Puertos

Taylor Series Expansion  0th order T.S. approx. (n = 0): f(xi+1) = f(xi) + O (h1)  1st order T.S. approx. (n = 1): f(xi+1) = f(xi) + hf '(xi) + O (h2)  2nd order T.S. approx. (n = 2):  nth order T.S. approximation will be exact for an nth order polynomial E. T. S. I. Caminos, Canales y Puertos

Taylor Series Expansion f(x ) f(xi ) Zero order f(xi+1 ) f(xi ) First order f(xi+1 ) f(xi )+f '(xi )h Second order f(xi+1 ) f(xi )+f '(xi )h+ )+f "(xi )h2/2! True f(xi+1 ) x xi xi+1 h E. T. S. I. Caminos, Canales y Puertos

Numerical Differentiation from Taylor Series Expansion Objective: Evaluate the derivatives of function, ƒ(xi), without doing it analytically. When would we want to do this? 1. function is too complicated to differentiate analytically: 2. function is not defined by an equation, i.e., given a set of data points (xi, ƒ(xi)), i=1,…,n i 0 1 2 3 4 xi 1.0 3.0 5.0 7.0 9.0 ƒ(xi) 2.3 4.1 5.5 5.7 5.9 E. T. S. I. Caminos, Canales y Puertos

Numerical Differentiation from Taylor Series Expansion First derivative with backward difference. E. T. S. I. Caminos, Canales y Puertos

Numerical Differentiation from Taylor Series Expansion Backward Difference Approx.:  First Derivative: Letting h = xi - xi-1 first backward difference E. T. S. I. Caminos, Canales y Puertos

Example of 1st Backward FDD Using data below calculate ƒ'(x1) : i 0 1 2 3 4 xi 1.0 3.0 5.0 7.0 9.0 ƒ(xi) 2.3 4.1 5.5 5.7 5.9 First Backward Finite-Divided-Difference at x1: f ' (x1)  0.9 { + O (h) } E. T. S. I. Caminos, Canales y Puertos

Backward Difference Approximation Second Derivative: + O([xi-2– xi]3) with h = xi– xi-1 and 2h = xi – xi-2 The 2nd order approximation to ƒ(xi-2) becomes: ƒ(xi-2) = ƒ(xi) – 2hƒ'(xi) + 2h2 ƒ"(xi) +O (h3) [1] 2nd order approximation to ƒ(xi-1): [2] E. T. S. I. Caminos, Canales y Puertos

Backward Difference Approximation Subtracting 2*[2] from [1] yields: f(xi-2) – 2f(xi-1) = –f(xi) + h2f"(xi) + O (h3) Rearranging: h2ƒ"(xi) = f(xi) – 2f(xi-1) + f(xi-2) + O (h3) Second backward difference E. T. S. I. Caminos, Canales y Puertos

Example of 2nd Backward FDD Using data below calculate ƒ"(x2) : i 0 1 2 3 4 xi 1.0 3.0 5.0 7.0 9.0 ƒ(xi) 2.3 4.1 5.5 5.7 5.9 Second Backward Finite-Divided-Difference at x2: f " (5.0)  - 0.1 { + O (h) } E. T. S. I. Caminos, Canales y Puertos

Other Forms of Numerical Differentiation What points are used for each form? Backward: …, ƒ(xi-2), ƒ(xi-1), ƒ(xi), ƒ(xi+1), ƒ(xi+2), … Forward: Centered: E. T. S. I. Caminos, Canales y Puertos

Taylor Series and Truncation errors - Higher order divided differences. - Second finite central divided difference E. T. S. I. Caminos, Canales y Puertos

Other Forms of Numerical Differentiation Forward: Centered: E. T. S. I. Caminos, Canales y Puertos

Taylor Series and Truncation errors Use of the Taylor series to calculate derivatives. First derivative with forward difference. E. T. S. I. Caminos, Canales y Puertos

Taylor Series and Truncation errors First derivative with central differences. E. T. S. I. Caminos, Canales y Puertos

Taylor Series and Truncation errors Questions: • Which is a better approximation? Forward, Centered, or Backward? • Why? • When would you use which? Note: We also can get higher order forward, centered, and backward difference derivative approximations [C&C Chapter 23, tabulated in Figs. 23.1-3] E. T. S. I. Caminos, Canales y Puertos

Example Combining Roundoff and Truncation Error Determine h to minimize the total error of a forward finite-divided difference approximation for: Truncation Error: xi    xi+1 Round-off Error: with e = machine epsilon. As a result: E. T. S. I. Caminos, Canales y Puertos

Example Combining Roundoff and Truncation Error  Total error  = truncation error + roundoff error  E = | Total Error |  + NOTE: Truncation error decreases as h decreases Round-off error increases as h decreases E. T. S. I. Caminos, Canales y Puertos

Example Combining Roundoff and Truncation Error E. T. S. I. Caminos, Canales y Puertos

Example Combining Roundoff and Truncation Error To minimize total error E with respect to h, set the first derivative to zero: Solve for h and approximate f "() as f "(xi): E. T. S. I. Caminos, Canales y Puertos

Example Combining Roundoff and Truncation Error Linear Application: Determine h that will minimize total error for calculating f’(x) for f(x) =  x at x = 1 Using the first forward-divided-difference approximation with error O(h) and a 5-decimal-digit machine: e = b1-t = 101-5 = 10-4 = 0.0001 f '(x) = ; f "(x) = 0 E. T. S. I. Caminos, Canales y Puertos

Example Combining Roundoff and Truncation Error f(x+h)= {exact: 3.1415} h  (x+h) f(x+h)-f(x) [f(x+h)-f(x)]/h 0 3.1415 0.000001 3.1415 0 0 0.00001 3.1416 0.0001 10 0.0001 3.1419 0.0004 4.0 0.001 3.1447 0.0032 3.2 0.01 3.1730 0.0315 3.15 0.1 3. 4557 0.3142 3.142 1 6.2831 3.1416 3.146 Underlined digits are subject to round-off error. They are likely to be in error by ± one or two units. This does not cause much problem when h = 1, but causes large errors in the final result when h < 10-4. E. T. S. I. Caminos, Canales y Puertos

Example Combining Roundoff and Truncation Error Nonlinear Application: Determine h for minimizing the total error for computing f’(x) for ƒ(x) = ex at x = 3 Using the first forward-divided-difference approximation with error O(h) and a 5-decimal-digit machine: e = b1-t = 101-5 = 10-4 = 0.0001 f(x) = f '(x) = f "(x) = ex = 20.0855; or about 0.01 E. T. S. I. Caminos, Canales y Puertos

Example Combining Roundoff and Truncation Error full precision h f(x+h)=ex+h f(x+h)-f(x) [f(x+h)-f(x)] [f(x+h)-f(x)] h h 0 20.085 {exact = 20.085} 0.00001 20.085 0.0 0 20.086 0.0001 20.087 0.002 20 20.086 0.001 20.105 0.020 20 20.096 0.01 20.287 0.202 20.2 20.18 0.1 22.198 2.113 21.13 21.12 1 54.598 34.513 34.523 34.512 Roundoff Truncation Underlined digits subject to roundoff error. Bold digits in error due to truncation. E. T. S. I. Caminos, Canales y Puertos

Additional Error Terminology: Error Propagation Errors which appear because we are basing current calculations on previous calculations which also incurred some form of error Stability and Condition Number [see C&C 4.2.3] Numerically Unstable: Computations which are so sensitive to round-off errors that errors grow uncontrollably during calculations. Condition: sensitivity to such uncertainty; "well conditioned" vs. "ill conditioned" Condition Number: measure of the condition; i.e., extent to which uncertainty in x is amplified by ƒ(x) C.N.  1 ===> "well-conditioned" C.N. >> 1 ===> "ill-conditioned" E. T. S. I. Caminos, Canales y Puertos

Condition Number Begin with Taylor Series: where is an approximation of x. Relative error of f(x) Relative error of x The condition number is the ratio of the two relative errors: E. T. S. I. Caminos, Canales y Puertos

Condition Number Condition Number CN contrasts the uncertainty in x with the uncertainty in f(x) C.N.  1 ===> "well-conditioned", i.e., The error in f(x) is similar to the error in x. C.N. >> 1 ===> "ill-conditioned", i.e., the error in f(x) is amplified and small errors in x can produce large errors in f(x). E. T. S. I. Caminos, Canales y Puertos

Condition Number Condition Number Example: Compute the condition number for f(x) = tan(x) f(x) = tan(x) = sin(x)/cos(x) f'(x) = sec2(x) = 1/cos2(x) CN = x/[cos(x) sin(x)] This becomes large (ill-conditioned) when the denominator approaches zero, i.e., when x  0,  /2, , … E. T. S. I. Caminos, Canales y Puertos