1. Engineering Computation
Part 1
E. T. S. I. Caminos, Canales y Puertos

2. Introduction
Numerical methods are techniques which allow us to formulate mathematical problems to be solved by basic arithmetic operations.
They are able to handle large systems of equations, non-linearities, complicated geometries and allow us solving engineering problems which have no analytical solution.
The intelligent use of commercial software is favored by the knowledge of numerical methods.
They allow us to design ad hoc programs to solve concrete problems.
The allow us to become familiar with computers and understand the way they work.
They are a good tool to reinforce the mathematical knowledge, because one of their aims consists in transforming complicate problems into simple arithmetic operations.
E. T. S. I. Caminos, Canales y Puertos

3. Introduction Roots of equations : They are methods to solve
They try to find the values of a variable for it to satisfy one equation.
They are very useful in engineering projects, because in many occasions it is not possible to solve the design equations analytically.
E. T. S. I. Caminos, Canales y Puertos

4. Introduction Systems of linear equations:
They are methods looking for the set of values that simultaneously satisfy a system of algebraic equations.
Calculus of structures, electric circuits, supply networks, fit of curves, etc.
E. T. S. I. Caminos, Canales y Puertos

5. Introduction Optimization:
Determine the value x0 leding to the optimal value of f(x).
They can be subject to constraints.
E. T. S. I. Caminos, Canales y Puertos

6. Introduction
Fitting curves. Fitting techniques can be divided into two groups:
Regression. It is used when one has errors in the experimental data. One looks for the curve showing the trend of the data.
Interpolation. It is used to fit tabulated data and predict intermediate values or extrapolated data.
E. T. S. I. Caminos, Canales y Puertos

7. Introduction Integration: Determine the area below a given curve.
It has many applications in engineering. Calculus of centers of gravity. Calculus of areas, volumes, etc.
It can also be used to solve differential equations. f ( x ) I n t e g r a l b
E. T. S. I. Caminos, Canales y Puertos

8. Introduction Ordinary differential equations :
They are very important because many problems can be stated in terms of variations and not in terms of magnitudes.
There are two types of problems: Initial value problems, and boundary value problems.
E. T. S. I. Caminos, Canales y Puertos

9. Introduction Partial differential equations:
Used for characterizing engineering problems where the behavior of the physical magnitude can be expressed in terms of speed change with respect to two or more variables.
Approximation by finite differences or the finite element method.
E. T. S. I. Caminos, Canales y Puertos

10. Mathematical Models A mathematical expression of a given model can be
Analytic solution (t=0, v=0):
Approximate solution:
E. T. S. I. Caminos, Canales y Puertos

11. Mathematical Models
To solve the problem numerically, one replaces the derivative by a divided finite difference, tus transforming the problem into a very simple one containing only simple algebraic operations:
E. T. S. I. Caminos, Canales y Puertos

12. Numerical Differentiation
Forward:
Centered:
How big a step size should we select?
One- or two-sided formula:
What are the advantages of each?
How is optimal step size affected by:
- precision of numerical calculations?
- precision with which f is computed?
- curvature of function f near x=1?
- choice of formula?
E. T. S. I. Caminos, Canales y Puertos

13. APPROXIMATION AND ERRORS
Numerical Methods
Instead of solving for the exact solution we solve math problems with a series of arithmetic operations.
Example: dx
analytical solution: ln(b) – ln(a)
numerical solution e. g., Trapezoidal Rule
Error Analysis
(a) identify sources of error
(b) estimate the magnitude of the error
(c) determine how to minimize and control error
E. T. S. I. Caminos, Canales y Puertos

14. Mathematical Models Comparing solutions: Numerical solution, Dt=2seg2 4 6 8 10 12 14 16 18 20 30 40 50 60
Numerical solution, Dt=1seg
Numerical solution, Dt=2seg
T (sec)
V (m/sec)
Exact solution
E. T. S. I. Caminos, Canales y Puertos

15. Approximations and Rounding Errors
Unfortunately, computers introduce errors in calculations. However, since many engineering problems have not analytical solution, we are forced to used numerical methods (approximations). The only noption we have is to accept the error and try to reduce it up to a tolerable level.
The only way of minimizing the errors is by knowing and understanding why they occur and how we can diminish them.
The most frequent errors are:
Rounding errors, due to the fact that computers can work only with a finite representation of numbers.
Truncation errors, due to differences between the exact and the approximate (numeric) formulations of the mathematical problem being dealt with.
Before centering in each one of them, we will see two important concepts on the computer representation of numbers.
E. T. S. I. Caminos, Canales y Puertos

16. Approximations and Rounding Errors
Significant figures of a number:
The significant figures of a number are those that can be used with confidence.
The speedometer and the odometer in the figure estimate up to three and seven significant figures, respectively, 49.5 and
This concept has two important implications:
1. An approximation is acceptable when it is exact for a given number of significant figures.
2. There are magnitudes or constants that cannot be represented exactly.
E. T. S. I. Caminos, Canales y Puertos

17. Approximations and Rounding Errors
Accuracy closeness of measured/computed values to the "true" value (vs. inaccuracy or bias)
Bias systematic deviation from truth, "general trend"
Precision closeness of measured/computed values with each other (spread or scatter), relates to the number of significant figures (vs. imprecision or uncertainty)
E. T. S. I. Caminos, Canales y Puertos

18. Approximations and Rounding Errors
Accuracy and precision:
The errors associated with nd measurements can be characterized observing their accuracy and precision.
Accuracy refers to how close the value is to the true value.
Precision refers to how close are different measured values using the same method.
Numerical methods must be sufficiently exact (without bias) and precise to satisfy the requirements of engineering problems. From now on we will refer to error to refer to the inaccuracy and lack of precision of our predictions.
E. T. S. I. Caminos, Canales y Puertos

19. Approximations and Rounding Errors
(a) inaccurat imprecise
(b) accurate imprecise
(c) Inaccurate precise
(d) Accurate precise
E. T. S. I. Caminos, Canales y Puertos

20. Approximations and Rounding Errors
Error definitions:
True value = approximation + absolute error.
Absolute error = true value - approximation .
Relative error = absolute error / true value .
In real cases not always one can know the true value, thus:
In many occasions, the error is calculated as the difference between the previous and the actual approximations.
E. T. S. I. Caminos, Canales y Puertos

21. Approximations and Rounding Errors
Thus, the stopping criterium of a numerical method can be:
It is convenient to relate the errors with the number of significant figures.If the following relation holds, one can be sure that at least n significant figures are correct.
E. T. S. I. Caminos, Canales y Puertos

22. Approximations and Rounding Errors
Numerical systems:
A numerical system is a convention to represent quantities. Since we have 10 fingers in our hands, our most familiis the numerical system which basis is 10. It uses 10 different digits.
However, computers, due to the memory structure can only store two digits: 0 and 1. Thus, they use the binary system of numeric representation.
E. T. S. I. Caminos, Canales y Puertos

23. Round-off Errors Background: How are numbers stored in a computer?
The fundamental unit, a "word," consists of a string of "bits" (binary digits).
Because computers are made up of gates or switches which are either closed or open, we work in binary or base 2 system.
Example: An 8 bit word representation of the integer "35" is or ±
+ 0x26 1x25 0x24 0x23 0x22 1x21 1x20 = 35
Note: We can only represent a finite # of numbers; for our case:
– to (127 = 27 – 1)
or a total of 255 numbers (including 0)
E. T. S. I. Caminos, Canales y Puertos

24. Approximations and Rounding Errors
Representation of integer numbers:
To represent base 10 numbers in binary form the signed magnitude method is used. The first digit stores the sign (0, positive and 1, negative). The remaining bits are used to store the number.
A computer working with words of 16 bits can store integer numbers in the range to
E. T. S. I. Caminos, Canales y Puertos

25. Approximations and Rounding Errors
Floating point representation:
This representation is used for fractional quantities. It has the fraction part, called mantissa, and an integer part, called exponent or characteristic.
The mantissa is usually normalized, so that the value of m is limited:
E. T. S. I. Caminos, Canales y Puertos

26. Approximations and Rounding Errors
Hipothetical set:
We assume a hipothetical set of floating point numbers for a machine using 7 bits as word.
The smallest number that can be represented is , and the largest is 7.
E. T. S. I. Caminos, Canales y Puertos

27. Approximations and Rounding Errors
E. T. S. I. Caminos, Canales y Puertos

28. Round-off Errors EXAMPLE: “Our” Base 10 Machine:
This "Machine" carries three significant decimal digits and one exponent digit.
± 0.DDDE ± e
with: < e < 9
100 < DDD < 999
We represent in our machine:
E-1 w/ chopping t = 0.29%
E-1 w/ rounding t = 0.07%
(most computers round, but some chop to save time)
E. T. S. I. Caminos, Canales y Puertos

29. Round-off Errors Consequences of Computer Representation of #'s:
Limited Range of quantities can be represented because there are a finite # of digits in both the mantissa and the exponent.
smallest positive # E-9 or
largest positive # E+9 almost 109
Finite number of floating-point values can be represented within the above range.
± (0.100 to 0.999) E-9 to E+9
x #'s x #'s = 34,200 #'s
plus zero ( E±e )
Thus, 34,201 #'s can be represented within the above range (zero appears only once).
E. T. S. I. Caminos, Canales y Puertos

30. Consequences of Computer Representation of #'s
The interval between successive represented numbers (quantization error) is not uniform, and increases as the numbers grow in magnitude.
- "quantization" error, Dx, is proportional to the magnitude x.
| x | =
| x | = 10.0
- normalizing | Dx | by | x | gives an approximation of "machine epsilon"
- p, with both chopping and rounding, is represented by 0.314E+1 with:"quantizing error" =
E. T. S. I. Caminos, Canales y Puertos

31. Approximations and Rounding Errors
Conclusions:
There is a limited range to represent quantities. The limits correspond to “underflow” and “overflow”.
There is a finite number of quantities that can be represented on a given range.
E. T. S. I. Caminos, Canales y Puertos

32. Approximations and Rounding Errors
The interval between numbers increases as the numbers grow in magnitude.
This means that the errors are proportional to the magnitude of the number to be represented. Analyzing the relative error one gets:
where e is the machine epsilon, that is the worst possible relative error.
with chopping
With rounding
E. T. S. I. Caminos, Canales y Puertos

33. Machine Epsilon: Smallest number such that: 1 + e > 1
Machine Epsilon on "Our Base 10 Machine"
Try >
0.100 E E–1 = E=1
= 1.01
Try >
0.100 E E-2 = E+1 =
after chopping ==> 1.00
Therefore, Machine epsilon is 0.01;
Quick formula: e = b1- t = = 0.01
t = # of sig. figs. in mantissa
E. T. S. I. Caminos, Canales y Puertos

34. Round-off Error due to Arithmetic Operations
Trouble happens when you add (or subtract) a small number to a large number:
For "our machine": 1,  1,210
Working left-to-right:
= which is chopped to 1200
= 0.120E+4
= 1206 which is chopped to 1200
Reverse the order:
4 + 6 = 10 which is ok
= which is ok
Remedy: add or subtract small numbers together first.
E. T. S. I. Caminos, Canales y Puertos

35. Round-off Error due to Arithmetic Operations
Subtractive Cancellation (subtracting numbers of almost equal size) – too few significant figures left
Consider formulas such as :
Problems arise when we try to subtract two numbers of almost equal size
32,232 – 32,181 = 51 (Little precision left.)
Our Machine ("chopper"):
0.322 E+5 – E+5
32200 – = 100
With rounding:
0.322 E+5 – E+5
32200 – = 0
E. T. S. I. Caminos, Canales y Puertos

36. Round-off Error due to Arithmetic Operations
Smearing
Occurs when individual terms are larger than summation itself. Consider the exponential series with x = -10
Consider formulas such as:
With 7-decimal-digit accuracy:
exact answer =
computed answer = – (differs from book value)
(45 terms) wrong sign !
Largest intermediate terms are:
9th = –2, & 10th = 2,
E. T. S. I. Caminos, Canales y Puertos

37. Modern Computers (IEEE standard)
Single Precision:
32 BIT WORDS
24 BITS assigned to mantissa (including the sign bit)
8 BITS to signed exponent
Double Precision:
64 BIT WORDS
56 BITS assigned to mantissa (including the sign bit)
8 BITS to signed exponent (not changed)
223 = 8,400,000 or almost 7 full decimal digits for single
255 = 4x1016 or almost 17 decimal digits for double
27 – 1 = 127; ~ 2x for both single and double
E. T. S. I. Caminos, Canales y Puertos

38. Truncation Error
Error caused by the nature of the numerical technique employed to approximate the solution.
Example:
Maclaurin series expansion of ex
If we use a truncated version of the series:
Then the Truncation Error is:
E. T. S. I. Caminos, Canales y Puertos

39. Approximations and Rounding Errors
Precautions:
Sums of large and small numbers: due to equaling the exponent. They are common in sums of infinite series where the individual terms are very small when compared with the accumulated sum. This error can be reduced by summing first the small terms and using double precision.
Cancellation of the subtraction: The subtraction of very similar numbers.
Smearing: The individual terms are larger than the total sum.
Inner products: They are prone to rounding errors. Thus, it is convenient to use double precision in this type of calculations.
E. T. S. I. Caminos, Canales y Puertos

40. Taylor Series Expansion
Basic Idea:
Predict the value of a function, ƒ, at a point xi+1 based on the value of the function and all of its derivatives, ƒ, ƒ', ƒ",… at a neighboring point xi
Given xi, ƒ(xi), ƒ'(xi), ƒ"(xi), ... ƒn+1(xi),
we can predict or approximate ƒ(xi+1)
E. T. S. I. Caminos, Canales y Puertos

41. Taylor Series Expansion
General Form (Eq. 4.7 in C&C):
h = "step size" = xi+1 – xi
Rn = remainder to account for all other terms
with xi    xi+1
= O (hn+1) with x not exactly known "on the order of hn+1 "
Note: f(x) must be a function with n+1 continuous derivatives
E. T. S. I. Caminos, Canales y Puertos

42. Taylor Series Expansion
 0th order T.S. approx. (n = 0): f(xi+1) = f(xi) + O (h1)
 1st order T.S. approx. (n = 1): f(xi+1) = f(xi) + hf '(xi) + O (h2)
 2nd order T.S. approx. (n = 2):
 nth order T.S. approximation will be exact for an nth order polynomial
E. T. S. I. Caminos, Canales y Puertos

43. Taylor Series Expansion
f(x )
f(xi )
Zero order
f(xi+1 ) f(xi )
First order
f(xi+1 ) f(xi )+f '(xi )h
Second order
f(xi+1 ) f(xi )+f '(xi )h+ )+f "(xi )h2/2!
True
f(xi+1 ) x xi
xi+1 h
E. T. S. I. Caminos, Canales y Puertos

44. Numerical Differentiation from Taylor Series Expansion
Objective:
Evaluate the derivatives of function, ƒ(xi), without doing it analytically.
When would we want to do this?
1. function is too complicated to differentiate analytically:
2. function is not defined by an equation,
i.e., given a set of data points (xi, ƒ(xi)), i=1,…,n i xi
ƒ(xi)
E. T. S. I. Caminos, Canales y Puertos

45. Numerical Differentiation from Taylor Series Expansion
First derivative with backward difference.
E. T. S. I. Caminos, Canales y Puertos

46. Numerical Differentiation from Taylor Series Expansion
Backward Difference Approx.:
 First Derivative:
Letting h = xi - xi-1
first backward difference
E. T. S. I. Caminos, Canales y Puertos

47. Example of 1st Backward FDD
Using data below calculate ƒ'(x1) : i xi
ƒ(xi)
First Backward Finite-Divided-Difference at x1:
f ' (x1)  { + O (h) }
E. T. S. I. Caminos, Canales y Puertos

48. Backward Difference Approximation
Second Derivative:
+ O([xi-2– xi]3)
with h = xi– xi-1 and 2h = xi – xi-2
The 2nd order approximation to ƒ(xi-2) becomes:
ƒ(xi-2) = ƒ(xi) – 2hƒ'(xi) + 2h2 ƒ"(xi) +O (h3) [1]
2nd order approximation to ƒ(xi-1):
[2]
E. T. S. I. Caminos, Canales y Puertos

49. Backward Difference Approximation
Subtracting 2*[2] from [1] yields:
f(xi-2) – 2f(xi-1) = –f(xi) + h2f"(xi) + O (h3)
Rearranging:
h2ƒ"(xi) = f(xi) – 2f(xi-1) + f(xi-2) + O (h3)
Second backward difference
E. T. S. I. Caminos, Canales y Puertos

50. Example of 2nd Backward FDD
Using data below calculate ƒ"(x2) : i xi
ƒ(xi)
Second Backward Finite-Divided-Difference at x2:
f " (5.0)  { + O (h) }
E. T. S. I. Caminos, Canales y Puertos

51. Other Forms of Numerical Differentiation
What points are used for each form?
Backward:
…, ƒ(xi-2), ƒ(xi-1), ƒ(xi), ƒ(xi+1), ƒ(xi+2), …
Forward:
Centered:
E. T. S. I. Caminos, Canales y Puertos

52. Taylor Series and Truncation errors
- Higher order divided differences.
- Second finite central divided difference
E. T. S. I. Caminos, Canales y Puertos

53. Other Forms of Numerical Differentiation
Forward:
Centered:
E. T. S. I. Caminos, Canales y Puertos

54. Taylor Series and Truncation errors
Use of the Taylor series to calculate derivatives.
First derivative with forward difference.
E. T. S. I. Caminos, Canales y Puertos

55. Taylor Series and Truncation errors
First derivative with central differences.
E. T. S. I. Caminos, Canales y Puertos

56. Taylor Series and Truncation errors
Questions:
• Which is a better approximation?
Forward, Centered, or Backward?
• Why?
• When would you use which?
Note:
We also can get higher order forward, centered, and backward difference derivative approximations
[C&C Chapter 23, tabulated in Figs ]
E. T. S. I. Caminos, Canales y Puertos

57. Example Combining Roundoff and Truncation Error
Determine h to minimize the total error of a forward finite-divided difference approximation for:
Truncation Error:
xi    xi+1
Round-off Error:
with e = machine epsilon.
As a result:
E. T. S. I. Caminos, Canales y Puertos

58. Example Combining Roundoff and Truncation Error
 Total error  = truncation error + roundoff error 
E = | Total Error |  +
NOTE: Truncation error decreases as h decreases
Round-off error increases as h decreases
E. T. S. I. Caminos, Canales y Puertos

59. Example Combining Roundoff and Truncation Error
E. T. S. I. Caminos, Canales y Puertos

60. Example Combining Roundoff and Truncation Error
To minimize total error E with respect to h, set the first derivative to zero:
Solve for h and approximate f "() as f "(xi):
E. T. S. I. Caminos, Canales y Puertos

61. Example Combining Roundoff and Truncation Error
Linear Application:
Determine h that will minimize total error for calculating f’(x) for
f(x) =  x
at x = 1
Using the first forward-divided-difference approximation with
error O(h) and a 5-decimal-digit machine:
e = b1-t = = 10-4 =
f '(x) = ; f "(x) = 0
E. T. S. I. Caminos, Canales y Puertos

62. Example Combining Roundoff and Truncation Error
f(x+h)= {exact: }
h  (x+h) f(x+h)-f(x) [f(x+h)-f(x)]/h
Underlined digits are subject to round-off error. They are likely to be in error by ± one or two units. This does not cause much problem when h = 1, but causes large errors in the final result when h < 10-4.
E. T. S. I. Caminos, Canales y Puertos

63. Example Combining Roundoff and Truncation Error
Nonlinear Application:
Determine h for minimizing the total error for computing f’(x) for
ƒ(x) = ex at x = 3
Using the first forward-divided-difference approximation with
error O(h) and a 5-decimal-digit machine:
e = b1-t = = 10-4 =
f(x) = f '(x) = f "(x) = ex = ;
or about 0.01
E. T. S. I. Caminos, Canales y Puertos

64. Example Combining Roundoff and Truncation Error
full precision
h f(x+h)=ex+h f(x+h)-f(x) [f(x+h)-f(x)] [f(x+h)-f(x)]
h h
{exact = }
Roundoff Truncation
Underlined digits subject to roundoff error.
Bold digits in error due to truncation.
E. T. S. I. Caminos, Canales y Puertos

65. Additional Error Terminology:
Error Propagation
Errors which appear because we are basing current calculations on previous calculations which also incurred some form of error
Stability and Condition Number [see C&C 4.2.3]
Numerically Unstable: Computations which are so sensitive to round-off errors that errors grow uncontrollably during calculations.
Condition: sensitivity to such uncertainty; "well conditioned" vs. "ill conditioned"
Condition Number: measure of the condition; i.e., extent to which uncertainty in x is amplified by ƒ(x)
C.N.  1 ===> "well-conditioned"
C.N. >> 1 ===> "ill-conditioned"
E. T. S. I. Caminos, Canales y Puertos

66. Condition Number Begin with Taylor Series:
where is an approximation of x.
Relative error of f(x) Relative error of x
The condition number is the ratio of the two relative errors:
E. T. S. I. Caminos, Canales y Puertos

67. Condition Number Condition Number
CN contrasts the uncertainty in x with the uncertainty in f(x)
C.N.  1 ===> "well-conditioned", i.e., The error in f(x) is similar to the error in x.
C.N. >> 1 ===> "ill-conditioned", i.e., the error in f(x) is amplified and small errors in x can produce large errors in f(x).
E. T. S. I. Caminos, Canales y Puertos

68. Condition Number Condition Number
Example: Compute the condition number for f(x) = tan(x)
f(x) = tan(x) = sin(x)/cos(x)
f'(x) = sec2(x) = 1/cos2(x)
CN = x/[cos(x) sin(x)]
This becomes large (ill-conditioned) when the denominator approaches zero, i.e., when x  0,  /2, , …
E. T. S. I. Caminos, Canales y Puertos