1. Approximation of Functions
Approximation problems divided into five parts:
Least square approximation of discrete functions or Regression
Least square approximation of continuous function using various basis polynomials
Orthogonal basis functions
Approximation of periodic functions
Interpolation

2. Norm and Seminorm
A real valued function is called a norm on a vector space if it is defined everywhere on the space and satisfies the following conditions:
𝑓 >0 for𝑓≠0; 𝑓 =0 iff 𝑓=0 ∀𝑥∈ 𝑎,𝑏
𝛼𝑓 = 𝛼 𝑓 for a real 𝛼
𝑓+𝑔 ≤ 𝑓 + 𝑔
Lp-Norm: 𝑓 𝑝 = 𝑎 𝑏 𝑓 𝑥 𝑝 𝑑𝑥 1 𝑝
p = 2: Euclidean or L2 norm, 𝑓 2 = 𝑎 𝑏 𝑓 𝑥 2 𝑑𝑥 1 2
p → ∞: 𝑓 ∞ = max 𝑥∈ 𝑎,𝑏 𝑓 𝑥
Weighted Euclidean Norm: 𝑓 2,𝑤 = 𝑎 𝑏 𝑓 𝑥 2 𝑤 𝑥 𝑑𝑥 1 2
Euclidean seminorm: tab 𝑓 2, 𝐺 = 𝑗=0 𝑛 𝑓 𝑥 𝑗

3. Inner Product
The inner product of two real-valued continuous functions f(x) and g(x) is denoted by 𝑓, 𝑔 and is defined as:
𝑓, 𝑔 = 𝑎 𝑏 𝑓 𝑥 𝑔 𝑥 𝑤 𝑥 𝑑𝑥 Continuous Case 𝑖=0 𝑛 𝑓 𝑥 𝑖 𝑔 𝑥 𝑖 𝑤 𝑖 (Discrete Case)
Properties of Inner Product:
Commutativity: 𝑓, 𝑔 = 𝑔, 𝑓
Linearity: (𝑐 1 𝑓+ 𝑐 2 𝑔),𝜑 = 𝑐 1 𝑓, 𝜑 + 𝑐 2 𝑔, 𝜑
Positivity: 𝑓, 𝑓 ≥0
𝑓, 𝑓 = 𝑓 =[ 𝑎 𝑏 𝑓 𝑥 2 𝑑𝑥 ]2

4. Basis Functions: Linear Independence
A sequence of (n + 1) functions 𝜑 0 , 𝜑 1 , 𝜑 2 ,⋯ 𝜑 𝑛 are linearly independent if,
𝑗=0 𝑛 𝑐 𝑗 𝜑 𝑗 =0 ⟹ 𝑐 𝑗 =0 ∀𝑗
This sequence of functions builds (or spans) an (n + 1)-dimensional linear subspace
From the Definition of Norm:
𝑗=0 𝑛 𝑐 𝑗 𝜑 𝑗 =0 is true only if 𝑐 𝑗 =0 ∀𝑗
Linearity of inner product:
𝑗=0 𝑛 𝑐 𝑗 𝜑 𝑗 , 𝜑 𝑘 = 𝑗=0 𝑛 𝑐 𝑗 𝜑 𝑗 𝑥 , 𝜑 𝑘 𝑥

5. Orthogonal Functions
Two real-valued continuous functions f(x) and g(x) are said to be orthogonal if 𝑓, 𝑔 =0
A finite or infinite sequence of functions 𝜑 0 , 𝜑 1 , 𝜑 2 ,⋯ 𝜑 𝑛 ,⋯ make an orthogonal system if 𝜑 𝑖 , 𝜑 𝑗 =0 for all i ≠ j and 𝜑 𝑖 ≠0 for all i.
In addition, if 𝜑 𝑖 =1, the sequence of functions is called an orthonormal system
Pythagorean Theorem for functions: 𝑓, 𝑔 =0⟹ 𝑓+𝑔 2 = 𝑓 𝑔 2
𝑓+𝑔 2 = 𝑓+𝑔 , 𝑓+𝑔 = 𝑓, 𝑓 + 𝑓, 𝑔 + 𝑔, 𝑓 + 𝑔, 𝑔 = 𝑓 𝑔 2
Generalized for orthogonal system: 𝑗=0 𝑛 𝑐 𝑗 𝜑 𝑗 2 = 𝑗=0 𝑛 𝑐 𝑗 𝜑 𝑗 2

6. Least Square Problem
Let f(x) be a continuous real-valued function in (a, b) that is to be approximated by p(x), a linear combination of a system of (n + 1) linearly independent functions 𝜑 0 , 𝜑 1 , 𝜑 2 ,⋯ 𝜑 𝑛 as shown below:
𝑝 𝑥 = 𝑗=0 𝑛 𝑐 𝑗 𝜑 𝑗 𝑥
Determine the coefficients 𝑐 𝑗 = 𝑐 𝑗 ∗ such that, a weighted Euclidean norm or seminorm of error p(x) – f(x) becomes as small as possible.
𝑝 𝑥 −𝑓 𝑥 2 = 𝑎 𝑏 𝑝 𝑥 −𝑓 𝑥 2 𝑑𝑥 is minimum when 𝑐 𝑗 = 𝑐 𝑗 ∗
𝑝 𝑥 −𝑓 𝑥 2 = 𝑖=1 𝑚 𝑝 𝑥 𝑖 −𝑓 𝑥 𝑖 is minimum when 𝑐 𝑗 = 𝑐 𝑗 ∗
Least Square Solution: 𝑝 ∗ 𝑥 = 𝑗=0 𝑛 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥

7. Schematic of Least Square Solution
f(x)
Space Spanned by a system of (n + 1) linearly independent functions 𝜑 0 , 𝜑 1 , 𝜑 2 ,⋯ 𝜑 𝑛
𝑝 𝑥 = 𝑗=0 𝑛 𝑐 𝑗 𝜑 𝑗
𝑝 ∗ 𝑥 = 𝑗=0 𝑛 𝑐 𝑗 ∗ 𝜑 𝑗
Solution: 𝑓 𝑥 − 𝑝 ∗ 𝑥 , 𝜑 𝑘 =0 for k = 0, 1, 2, … n

8. Least Square Solution: Proof
When 𝜑 0 , 𝜑 1 , 𝜑 2 ,⋯ 𝜑 𝑛 are linearly independent, the least square problem has a unique solution:
𝑝 ∗ 𝑥 = 𝑗=0 𝑛 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥
where, p*(x) – f(x) is orthogonal to all φj’s, j = 0, 1, 2, … n.
We need to prove:
𝑝 ∗ 𝑥 −𝑓 𝑥 2 is minimum when p*(x) – f(x) is orthogonal to all φj’s, j = 0, 1, 2, … n
existence and uniqueness of the least square solution

9. Least Square Solution: Proof
Let 𝑐 0 , 𝑐 1 , 𝑐 2 ,⋯ 𝑐 𝑛 be another sequence of coefficients with 𝑐 𝑗 ≠ 𝑐 𝑗 ∗ for at least one j, then 𝑗=0 𝑛 𝑐 𝑗 𝜑 𝑗 𝑥 −𝑓 𝑥 = 𝑗=0 𝑛 𝑐 𝑗 − 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥 + 𝑝 ∗ 𝑥 −𝑓 𝑥 If p*(x) – f(x) is orthogonal to all φj’s, it is also orthogonal to their linear combination 𝑗=0 𝑛 𝑐 𝑗 − 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥 . According to Pythagorean theorem, we have: 𝑗=0 𝑛 𝑐 𝑗 𝜑 𝑗 𝑥 −𝑓 𝑥 2 = 𝑗=0 𝑛 𝑐 𝑗 − 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥 2 + 𝑝 ∗ 𝑥 −𝑓 𝑥 2 > 𝑝 ∗ 𝑥 −𝑓 𝑥 2 Therefore, if p*(x) – f(x) is orthogonal to all φj’s, then p*(x) is the solution of the least square problem.

10. Least Square Solution: Normal Equations
If 𝜑 0 , 𝜑 1 , 𝜑 2 ,⋯ 𝜑 𝑛 are linearly independent, solution to the least square problem is:
𝑝 ∗ 𝑥 −𝑓 𝑥 , 𝜑 𝑘 𝑥 =0 𝑘=0, 1, 2,⋯𝑛
where, 𝑝 ∗ 𝑥 = 𝑗=0 𝑛 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥
Therefore,
𝑗=0 𝑛 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥 −𝑓 𝑥 , 𝜑 𝑘 𝑥 =0 𝑘=0, 1, 2,⋯𝑛
𝑗=0 𝑛 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥 , 𝜑 𝑘 𝑥 = 𝑓 𝑥 , 𝜑 𝑘 𝑥 𝑘=0, 1, 2,⋯𝑛
Normal Equations!

11. Least Square Solution: Normal Equations
𝑗=0 𝑛 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥 , 𝜑 𝑘 𝑥 = 𝑓 𝑥 , 𝜑 𝑘 𝑥 ; 𝑘=0, 1, 2,⋯𝑛
𝑐 0 ∗ 𝜑 0 , 𝜑 0 + 𝑐 1 ∗ 𝜑 1 , 𝜑 0 + 𝑐 2 ∗ 𝜑 2 , 𝜑 0 +⋯+ 𝑐 𝑛 ∗ 𝜑 𝑛 , 𝜑 0 = 𝑓, 𝜑 𝑘=0
𝑐 0 ∗ 𝜑 0 , 𝜑 1 + 𝑐 1 ∗ 𝜑 1 , 𝜑 1 + 𝑐 2 ∗ 𝜑 2 , 𝜑 1 +⋯+ 𝑐 𝑛 ∗ 𝜑 𝑛 , 𝜑 1 = 𝑓, 𝜑 𝑘=1
𝑐 0 ∗ 𝜑 0 , 𝜑 2 + 𝑐 1 ∗ 𝜑 1 , 𝜑 2 + 𝑐 2 ∗ 𝜑 2 , 𝜑 2 +⋯+ 𝑐 𝑛 ∗ 𝜑 𝑛 , 𝜑 2 = 𝑓, 𝜑 𝑘=2
⋮ ⋮ ⋮ ⋯ ⋮ = ⋮
𝑐 0 ∗ 𝜑 0 , 𝜑 𝑛 + 𝑐 1 ∗ 𝜑 1 , 𝜑 𝑛 + 𝑐 2 ∗ 𝜑 2 , 𝜑 𝑛 +⋯+ 𝑐 𝑛 ∗ 𝜑 𝑛 , 𝜑 𝑛 = 𝑓, 𝜑 𝑛 𝑘=𝑛
Moreover, if 𝜑 0 , 𝜑 1 , 𝜑 2 ,⋯ 𝜑 𝑛 is an orthogonal system:
𝑐 𝑘 ∗ = 𝑓, 𝜑 𝑘 𝜑 𝑘 , 𝜑 𝑘 𝑘=0, 1, 2,⋯𝑛

12. Least Square Solution: Existence and Uniqueness
𝑗=0 𝑛 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥 , 𝜑 𝑘 𝑥 = 𝑓 𝑥 , 𝜑 𝑘 𝑥 ; 𝑘=0, 1, 2,⋯𝑛
𝑐 0 ∗ 𝜑 0 , 𝜑 0 + 𝑐 1 ∗ 𝜑 1 , 𝜑 0 + 𝑐 2 ∗ 𝜑 2 , 𝜑 0 +⋯+ 𝑐 𝑛 ∗ 𝜑 𝑛 , 𝜑 0 = 𝑓, 𝜑 𝑘=0
𝑐 0 ∗ 𝜑 0 , 𝜑 1 + 𝑐 1 ∗ 𝜑 1 , 𝜑 1 + 𝑐 2 ∗ 𝜑 2 , 𝜑 1 +⋯+ 𝑐 𝑛 ∗ 𝜑 𝑛 , 𝜑 1 = 𝑓, 𝜑 𝑘=1
𝑐 0 ∗ 𝜑 0 , 𝜑 2 + 𝑐 1 ∗ 𝜑 1 , 𝜑 2 + 𝑐 2 ∗ 𝜑 2 , 𝜑 2 +⋯+ 𝑐 𝑛 ∗ 𝜑 𝑛 , 𝜑 2 = 𝑓, 𝜑 𝑘=2
⋮ ⋮ ⋮ ⋯ ⋮ = ⋮
𝑐 0 ∗ 𝜑 0 , 𝜑 𝑛 + 𝑐 1 ∗ 𝜑 1 , 𝜑 𝑛 + 𝑐 2 ∗ 𝜑 2 , 𝜑 𝑛 +⋯+ 𝑐 𝑛 ∗ 𝜑 𝑛 , 𝜑 𝑛 = 𝑓, 𝜑 𝑛 𝑘=𝑛
Solution to the normal equations exist and is unique unless the following homogenous system has a nontrivial solution for 𝑐 0 ∗ , 𝑐 1 ∗ ,⋯ 𝑐 𝑛 ∗ , i.e., 𝑐 𝑗 ∗ ≠0 for at least one j:
𝑗=0 𝑛 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥 , 𝜑 𝑘 𝑥 = 0

13. Least Square Solution: Existence and Uniqueness
Solution to the normal equations exist and is unique unless the following homogenous system has a nontrivial solution for 𝑐 0 ∗ , 𝑐 1 ∗ ,⋯ 𝑐 𝑛 ∗ , i.e., 𝑐 𝑗 ∗ ≠0 for at least one j: 𝑗=0 𝑛 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥 , 𝜑 𝑘 𝑥 = 0 This would lead to 𝑗=0 𝑛 𝑐 𝑗 ∗ 𝜑 𝑗 2 = 𝑗=0 𝑛 𝑐 𝑗 ∗ 𝜑 𝑗 , 𝑘=0 𝑛 𝑐 𝑘 ∗ 𝜑 𝑘 = 𝑘=0 𝑛 𝑗=0 𝑛 𝜑 𝑗 , 𝜑 𝑘 𝑐 𝑗 ∗ 𝑐 𝑘 ∗ = 𝑘=0 𝑛 0. 𝑐 𝑘 ∗ =0 which contradicts that the 𝜑 0 , 𝜑 1 , 𝜑 2 ,⋯ 𝜑 𝑛 are linearly independent.

14. Least Square Solution: Example (Continuous)
Approximate the function f(x) = 1/(1 + x2) for x in [0, 1] using a straight line.
The basis functions are: 𝜑 0 𝑥 = 𝜑 0 =1; 𝜑 1 𝑥 = 𝜑 1 =𝑥; 𝑝 ∗ 𝑥 = 𝑗=0 1 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥
Normal Equation: 𝑗=0 𝑛 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥 , 𝜑 𝑘 𝑥 = 𝑓 𝑥 , 𝜑 𝑘 𝑥 ; 𝑘=0, 1, 2,⋯𝑛
𝑐 0 ∗ 𝜑 0 , 𝜑 0 + 𝑐 1 ∗ 𝜑 1 , 𝜑 0 = 𝑓, 𝜑 0
𝑐 0 ∗ 𝜑 0 , 𝜑 1 + 𝑐 1 ∗ 𝜑 1 , 𝜑 1 = 𝑓, 𝜑 1
𝜑 0 , 𝜑 0 = 𝑑𝑥 =1; 𝜑 0 , 𝜑 1 = 𝜑 1 , 𝜑 0 = 0 1 𝑥 1 𝑑𝑥 = 1 2 =0.5
𝜑 1 , 𝜑 1 = 0 1 𝑥 𝑥 𝑑𝑥 = 1 3 ; 𝑓, 𝜑 0 = 𝑥 𝑑𝑥 = 𝜋 4
𝑓, 𝜑 1 = 𝑥 2 𝑥 𝑑𝑥 = ln 2 2
𝑓, 𝑔 = 𝑎 𝑏 𝑓 𝑥 𝑔 𝑥 𝑤 𝑥 𝑑𝑥 Continuous Case

15. Least Square Solution: Example (Continuous)
𝑐 0 ∗ 𝜑 0 , 𝜑 0 + 𝑐 1 ∗ 𝜑 1 , 𝜑 0 = 𝑓, 𝜑 0 𝑐 0 ∗ 𝜑 0 , 𝜑 1 + 𝑐 1 ∗ 𝜑 1 , 𝜑 1 = 𝑓, 𝜑 /3 𝑐 0 ∗ 𝑐 1 ∗ = 𝜋/4 ln 2 /2 𝑐 0 ∗ = 𝜋/4 0.5 ln 2 /2 1/ /3 =1.062 𝑐 1 ∗ = 1 𝜋/4 0.5 ln 2 / /3 =− 𝑝 ∗ 𝑥 = 𝑗=0 1 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥 =1.062−0.5535𝑥

16. Least Square Solution: Example (Continuous)
Approximate the function f(x) = 1/(1 + x2) for x in [0, 1] using a 2nd order polynomial. The basis functions are: 𝜑 0 =1; 𝜑 1 =𝑥; 𝜑 2 = 𝑥 2 ; 𝑝 ∗ 𝑥 = 𝑗=0 2 𝑐 𝑗 ∗ 𝜑 𝑗 𝑐 0 ∗ 𝜑 0 , 𝜑 0 + 𝑐 1 ∗ 𝜑 1 , 𝜑 0 + 𝑐 2 ∗ 𝜑 2 , 𝜑 0 = 𝑓, 𝜑 0 𝑐 0 ∗ 𝜑 0 , 𝜑 1 + 𝑐 1 ∗ 𝜑 1 , 𝜑 1 + 𝑐 2 ∗ 𝜑 2 , 𝜑 1 = 𝑓, 𝜑 1 𝑐 0 ∗ 𝜑 0 , 𝜑 2 + 𝑐 1 ∗ 𝜑 1 , 𝜑 2 + 𝑐 2 ∗ 𝜑 2 , 𝜑 2 = 𝑓, 𝜑 2 Additional inner products to be evaluated are: 𝜑 0 , 𝜑 2 = 𝜑 2 , 𝜑 0 = 0 1 𝑥 2 1 𝑑𝑥 = 1 3 𝜑 1 , 𝜑 2 = 𝜑 2 , 𝜑 1 = 0 1 𝑥 2 𝑥 𝑑𝑥 = 1 4 =0.25; 𝜑 2 , 𝜑 2 = 0 1 𝑥 2 𝑥 2 𝑑𝑥 = 1 5 =0.2 𝑓, 𝜑 2 = 𝑥 2 𝑥 2 𝑑𝑥 =1− π 4

17. Least Square Solution: Example (Continuous)
𝑐 0 ∗ 𝜑 0 , 𝜑 0 + 𝑐 1 ∗ 𝜑 1 , 𝜑 0 + 𝑐 2 ∗ 𝜑 2 , 𝜑 0 = 𝑓, 𝜑 0 𝑐 0 ∗ 𝜑 0 , 𝜑 1 + 𝑐 1 ∗ 𝜑 1 , 𝜑 1 + 𝑐 2 ∗ 𝜑 2 , 𝜑 1 = 𝑓, 𝜑 1 𝑐 0 ∗ 𝜑 0 , 𝜑 2 + 𝑐 1 ∗ 𝜑 1 , 𝜑 2 + 𝑐 2 ∗ 𝜑 2 , 𝜑 2 = 𝑓, 𝜑 / / / 𝑐 0 ∗ 𝑐 1 ∗ 𝑐 2 ∗ = 𝜋/4 ln 2 /2 1−𝜋/4 Solving by Gauss Elimination: 𝑐 0 ∗ =1.030; 𝑐 1 ∗ =−0.3605; 𝑐 2 ∗ =− 𝑝 ∗ 𝑥 = 𝑗=0 2 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥 =1.03−0.3605𝑥−0.193 𝑥 2

18. Least Square Solution: Example (Continuous)
How do you judge how good the fit is or how do you compare fit of two polynomials?
Estimate 𝑝 ∗ 𝑥 −𝑓 𝑥 ∞ . Two options:
Analytical (+ numerical)
Numerical (+ visual)

19. Discrete Data
(n + 1) observations or data pairs [(x0, y0), (x1, y1), (x2, y2) … (xn, yn)]
(m + 1) basis functions: 𝜑 0 , 𝜑 1 , 𝜑 2 ,⋯ 𝜑 𝑚
Approximating polynomial: 𝑝 𝑥 = 𝑗=0 𝑚 𝑐 𝑗 𝜑 𝑗 𝑥
𝑐 0 𝜑 0 𝑥 0 + 𝑐 1 𝜑 1 𝑥 0 + 𝑐 2 𝜑 2 𝑥 0 +⋯+ 𝑐 𝑚 𝜑 𝑚 𝑥 0 = 𝑦 0
𝑐 0 𝜑 0 𝑥 1 + 𝑐 1 𝜑 1 𝑥 1 + 𝑐 2 𝜑 2 𝑥 1 +⋯+ 𝑐 𝑚 𝜑 𝑚 𝑥 1 = 𝑦 1
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
𝑐 0 𝜑 0 𝑥 𝑛 + 𝑐 1 𝜑 1 𝑥 𝑛 + 𝑐 2 𝜑 2 𝑥 𝑛 +⋯+ 𝑐 𝑚 𝜑 𝑚 𝑥 𝑛 = 𝑦 𝑛
n equations, m unknowns:
m < n: over-determined system, least square regression
m = n: unique solution, interpolation
m > n: under-determined system

20. Least Square Solution: Example (Discrete Data)
Variation of Ultimate Shear Strength (y) with curing Temperature (x) for a certain rubber compound was reported (J. Quality Technology, 1971, pp ) as: Fit a linear and a quadratic regression model to the data. For the linear model, the basis functions and the polynomial are: 𝜑 0 𝑥 = 𝜑 0 =1; 𝜑 1 𝑥 = 𝜑 1 =𝑥; 𝑝 ∗ 𝑥 = 𝑗=0 1 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥 𝑐 0 ∗ 𝜑 0 , 𝜑 0 + 𝑐 1 ∗ 𝜑 1 , 𝜑 0 = 𝑓, 𝜑 0 𝑐 0 ∗ 𝜑 0 , 𝜑 1 + 𝑐 1 ∗ 𝜑 1 , 𝜑 1 = 𝑓, 𝜑 1 Given vectors: x = [138, 140, 144.5, 146, 148, 151.5, 153.5, 157] and y = f(x) = [770, 800, 840, 810, 735, 640, 590, 560] No. of data points, m = 8. Denote elements of x and y as xi and yi, respectively.
x, in oC
138
140
144.5
146
148
151.5
153.5
157
y, in psi
770
800
840
810
735
640
590
560

21. Least Square Solution: Example (Discrete Data)
𝜑 0 , 𝜑 0 = 𝑖=1 𝑚 1 1 =8; 𝜑 1 , 𝜑 1 = 𝑖=1 𝑚 𝑥 𝑖 𝑥 𝑖 = 𝑖=1 𝑚 𝑥 𝑖 2 = 𝜑 1 , 𝜑 0 = 𝜑 0 , 𝜑 1 = 𝑖=1 𝑚 1 𝑥 𝑖 = 𝑖=1 𝑚 𝑥 𝑖 = 𝑓, 𝜑 0 = 𝑖=1 𝑚 𝑦 𝑖 1 =5745; 𝑓, 𝜑 1 = 𝑖=1 𝑚 𝑦 𝑖 𝑥 𝑖 = 𝑐 0 ∗ 𝜑 0 , 𝜑 0 + 𝑐 1 ∗ 𝜑 1 , 𝜑 0 = 𝑓, 𝜑 0 𝑐 0 ∗ 𝜑 0 , 𝜑 1 + 𝑐 1 ∗ 𝜑 1 , 𝜑 1 = 𝑓, 𝜑 𝑐 0 ∗ 𝑐 1 ∗ = ⟹ 𝑐 0 ∗ = ; 𝑐 1 ∗ =− 𝑝 ∗ 𝑥 = 𝑗=0 1 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥 =2773.5− 𝑥

22. Least Square Solution: Example (Discrete Data)
For the quadratic model, the basis functions and the polynomial are: 𝜑 0 =1; 𝜑 1 =𝑥; 𝜑 2 = 𝑥 2 ; 𝑝 ∗ 𝑥 = 𝑗=0 2 𝑐 𝑗 ∗ 𝜑 𝑗 𝑐 0 ∗ 𝜑 0 , 𝜑 0 + 𝑐 1 ∗ 𝜑 1 , 𝜑 0 + 𝑐 2 ∗ 𝜑 2 , 𝜑 0 = 𝑓, 𝜑 0 𝑐 0 ∗ 𝜑 0 , 𝜑 1 + 𝑐 1 ∗ 𝜑 1 , 𝜑 1 + 𝑐 2 ∗ 𝜑 2 , 𝜑 1 = 𝑓, 𝜑 1 𝑐 0 ∗ 𝜑 0 , 𝜑 2 + 𝑐 1 ∗ 𝜑 1 , 𝜑 2 + 𝑐 2 ∗ 𝜑 2 , 𝜑 2 = 𝑓, 𝜑 2 Additional inner products to be evaluated are: 𝜑 0 , 𝜑 2 = 𝜑 2 , 𝜑 0 = 𝑖=1 𝑚 1 𝑥 𝑖 2 = 𝜑 1 , 𝜑 2 = 𝜑 2 , 𝜑 1 = 𝑖=1 𝑚 𝑥 𝑖 𝑥 𝑖 2 = 𝑖=1 𝑚 𝑥 𝑖 3 = 𝜑 2 , 𝜑 2 = 𝑖=1 𝑚 𝑥 𝑖 2 𝑥 𝑖 2 = 𝑖=1 𝑚 𝑥 𝑖 4 = 𝑓, 𝜑 2 = 𝑖=1 𝑚 𝑦 𝑖 𝑥 𝑖 2 =

23. Least Square Solution: Example (Discrete Data)
𝑐 0 ∗ 𝜑 0 , 𝜑 0 + 𝑐 1 ∗ 𝜑 1 , 𝜑 0 + 𝑐 2 ∗ 𝜑 2 , 𝜑 0 = 𝑓, 𝜑 0 𝑐 0 ∗ 𝜑 0 , 𝜑 1 + 𝑐 1 ∗ 𝜑 1 , 𝜑 1 + 𝑐 2 ∗ 𝜑 2 , 𝜑 1 = 𝑓, 𝜑 1 𝑐 0 ∗ 𝜑 0 , 𝜑 2 + 𝑐 1 ∗ 𝜑 1 , 𝜑 2 + 𝑐 2 ∗ 𝜑 2 , 𝜑 2 = 𝑓, 𝜑 𝑐 0 ∗ 𝑐 1 ∗ 𝑐 2 ∗ = Solving by Gauss Elimination: 𝑐 0 ∗ =− ; 𝑐 1 ∗ = ; 𝑐 2 ∗ =− 𝑝 ∗ 𝑥 = 𝑗=0 2 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥 =− 𝑥− 𝑥 2

24. Least Square Solution: Example (Discrete Data)
How do you judge how good the fit is or how do you compare fit of two polynomials?
Square of errors!

25. Least Square Solution: Example (Discrete Data)
Denote: 𝑦 𝑖 =𝑓 𝑥 𝑖 , 𝑦 𝑖 = 𝑝 ∗ 𝑥 𝑖
𝜇 𝑦 = 𝑖=0 𝑚 𝑦 𝑖 𝑚+1 , 𝜎 𝑡 2 = 𝑖=0 𝑚 𝑦 𝑖 − 𝜇 𝑦 2
𝜀 2 = 𝑖=0 𝑚 𝑦 𝑖 − 𝑦 𝑖 2
R2 = 0.73
Coefficient of regression r or R is given by:
𝑟 2 = 𝜎 𝑡 2 − 𝜀 2 𝜎 𝑡 2
R2 = 0.88

26. Additional Points
You can do multiple regression using the same frame work.
Linearize some non-linear equations
Evaluate Integrals using Numerical Methods for Integration for functions that are not easy to integrate using analytical means!

27. ESO 208A: Computational Methods in Engineering Orthogonal Basis, Periodic Functions

28. Least Square Solution: Normal Equations
𝑗=0 𝑛 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥 , 𝜑 𝑘 𝑥 = 𝑓 𝑥 , 𝜑 𝑘 𝑥 ; 𝑘=0, 1, 2,⋯𝑛
If 𝜑 0 , 𝜑 1 , 𝜑 2 ,⋯ 𝜑 𝑛 is an orthogonal system:
𝑐 𝑘 ∗ = 𝑓, 𝜑 𝑘 𝜑 𝑘 , 𝜑 𝑘 𝑘=0, 1, 2,⋯𝑛
Let us explore orthogonal systems of basis functions!

29. Orthogonal Polynomials: Tchebycheff
Consider the Equality:
cos 𝑛+1 𝜑 + cos 𝑛−1 𝜑 =2 cos 𝜑 cos 𝑛𝜑 𝑛≥1
n = 1: cos 2𝜑 =2 cos 2 𝜑 −1
n = 2: cos 3𝜑 =2 cos 𝜑 cos 2𝜑 − cos 𝜑 =4 cos 3 𝜑 −3 cos 𝜑
Define: 𝑥= cos 𝜑 ,𝜑∈ 0,𝜋 and 𝑇 𝑛 𝑥 = cos 𝑛𝜑 = cos 𝑛 cos −1 𝑥
Recursion Formula: 𝑇 𝑛+1 𝑥 =2𝑥 𝑇 𝑛 𝑥 − 𝑇 𝑛−1 𝑥
Example: 𝑇 0 𝑥 =1; 𝑇 1 𝑥 =𝑥; 𝑇 2 𝑥 =2 𝑥 2 −1
Symmetry: 𝑇 𝑛 −𝑥 = −1 𝑛 𝑇 𝑛 𝑥
Leading Coefficient: 2n-1 for n ≥ 1 and 1 for n = 0
Tn(x) constitutes an orthogonal family of polynomials in [-1, 1]!

30. Orthogonal Polynomials: Tchebycheff
Tn(x) has n zeros in [-1, 1] called the Tchebycheff abscissae:
𝑥 𝑘 = cos 2𝑘+1 𝑛 𝜋 𝑘=0, 1, 2 ⋯ 𝑛−1
Follows from cos 𝑛𝜑 =0 for 𝜑= 2𝑘+1 𝑛 𝜋 2
Tn(x) has n+1 extrema in [-1, 1]:
𝑥 𝑗 = cos 𝑗𝜋 𝑛 ; 𝑇 𝑛 𝑥 𝑗 = −1 𝑗 𝑗=0, 1, 2 ⋯𝑛
Follows from cos 𝑛𝜑 has maxima at 𝜑= 𝑗𝜋 𝑛

31. Orthogonal Polynomials: Tchebycheff
Orthogonality (continuous):
𝑇 𝑚 𝑥 , 𝑇 𝑛 𝑥 = 0 𝜋 cos 𝑚𝜑 cos 𝑛𝜑 𝑑𝜑
= −1 1 𝑇 𝑚 𝑥 𝑇 𝑛 𝑥 − 𝑥 2 𝑑𝑥
= if 𝑚≠𝑛 𝜋 2 if 𝑚=𝑛 ≠0 𝜋 if 𝑚=𝑛=0
For arbitrary f(x), a ≤ x ≤ b: 𝑥= 𝑏+𝑎 2 + 𝑏−𝑎 2 𝜉 −1≤𝜉≤1

32. Orthogonal Polynomials: Tchebycheff
Orthogonality (discrete): For 0 ≤ m ≤ N and 0 ≤ n ≤ N 𝑇 𝑚 𝑥 , 𝑇 𝑛 𝑥 = 𝑘=0 𝑁 𝑇 𝑚 𝑥 𝑘 𝑇 𝑛 𝑥 𝑘 = 0 if 𝑚≠𝑛 1 2 𝑁+1 if 𝑚=𝑛 ≠0 𝑁+1 if 𝑚=𝑛=0 where, {xk} are the zeros of TN+1(x)

33. Orthogonal Polynomials: Legendre
Solution of the Legendre’s equation (for n non-negative:
𝑑 𝑑𝑥 1− 𝑥 2 𝑑𝜑 𝑑𝑥 +𝑛 𝑛+1 𝜑=0 −1≤𝑥≤1
Solutions are an orthogonal set of polynomials given by:
𝑃 0 𝑥 =1; 𝑃 𝑛 𝑥 = 1 2 𝑛 𝑛! 𝑑 𝑛 𝑑𝑥 𝑛 𝑥 2 −1 𝑛
Bonnet’s recursive relation for n ≥ 2:
𝑃 𝑛 𝑥 = 2𝑛−1 𝑛 𝑥 𝑃 𝑛−1 𝑥 − 𝑛−1 𝑛 𝑃 𝑛−2 𝑥
Examples:
𝑃 0 𝑥 =1; 𝑃 1 𝑥 =𝑥; 𝑃 2 𝑥 = 𝑥 2 −1 ; 𝑃 3 𝑥 = 𝑥 3 −3𝑥

34. Orthogonal Polynomials: Legendre
Symmetry: 𝑃 𝑛 −𝑥 = −1 𝑛 𝑃 𝑛 𝑥
Orthogonality (continuous):
𝑃 𝑚 𝑥 , 𝑃 𝑛 𝑥 = if 𝑚≠𝑛 2 2𝑛+1 if 𝑚=𝑛
𝜔 𝑥 =1; 𝑃 𝑛 𝑥 ≤1
For discrete data: equidistant data points.

35. Orthonormal Polynomials: Gram
For Equidistant Discrete Data:
For −1≤𝑥≤1, a net of (n + 1) equidistant points are given by:
𝑥 𝑘 =−1+ 2𝑘 𝑛 for 𝑘=0, 1, 2, ⋯𝑛
On this net, the orthonormal set of polynomials 𝐺 𝑚 𝑥 𝑚=0 𝑛 are given by:
𝐺 −1 𝑥 =0; 𝐺 0 𝑥 = 1 𝑛+1 ; 𝐺 𝑚+1 𝑥 = 𝛼 𝑚 𝑥 𝐺 𝑚 𝑥 − 𝛼 𝑚 𝛼 𝑚−1 𝐺 𝑚−1 𝑥
𝛼 𝑚 = 𝑛 𝑚 𝑚+1 2 −1 𝑛+1 2 − 𝑚 𝑚=0, 1, 2,⋯𝑛−1
Example (for n = 5):
𝐺 0 𝑥 = ; 𝐺 1 𝑥 = 5𝑥 ; 𝐺 2 𝑥 = 𝑥 2 −

36. Orthonormal Polynomials: Gram
Orthogonality:
𝐺 𝑖 𝑥 , 𝐺 𝑗 𝑥 = 𝑘=0 𝑛 𝐺 𝑖 𝑥 𝑘 𝐺 𝑗 𝑥 𝑘 = 0 if 𝑖≠𝑗 1 if 𝑖=𝑗
When m << n1/2, Gm(x) are very similar to the Legendre polynomials
When n << m1/2, Gm(x) have very large oscillations between the net points, large maximum norm in [-1, 1]
When fitting a polynomial to equidistant data, one should never choose m larger than ~ 2n1/2

37. Least Square Solution: Example (Continuous)
Approximate the function f(x) = 1/(1 + x2) for x in [0, 1] using a 2nd order polynomial using Legendre polynomials. For Legendre polynomials, use x = (z + 1)/2 such that for x in [0, 1], z is in [-1, 1] The function is: f(z) = 4/(5 + 2z + z2) The basis functions are: 𝜑 0 =1; 𝜑 1 =𝑧; 𝜑 2 = 𝑧 2 −1 ; 𝑝 ∗ 𝑧 = 𝑗=0 2 𝑐 𝑗 ∗ 𝜑 𝑗 𝑓, 𝜑 0 = − 𝑧+ 𝑧 2 𝑑𝑧 = π 2 = 𝑓, 𝜑 1 = −1 1 4𝑧 5+2𝑧+ 𝑧 2 𝑑𝑧 =2 ln 2 − π 2 =− 𝑓, 𝜑 2 = − 𝑧 2 −1 5+2𝑧+ 𝑧 2 𝑑𝑧 =12−6 ln 2 − 5π 2 =−0.1286× 10 −1 𝑐 0 ∗ = =0.7854; 𝑐 1 ∗ = − /3 =−0.2768; 𝑐 2 ∗ = −0.1286× 10 −1 2/5 =−0.3216× 10 −1

38. Least Square Solution: Example (Continuous)
𝑐 0 ∗ = =0.7854; 𝑐 1 ∗ = − /3 =−0.2768; 𝑐 2 ∗ = −0.1286× 10 −1 2/5 =−0.3216× 10 −1 𝑝 ∗ 𝑧 = 𝑗=0 2 𝑐 𝑗 ∗ 𝜑 𝑗 𝑧 =0.7854−0.2768𝑧−0.3216× 10 −1 × 𝑧 2 −1 =0.8015−0.2768𝑧−0.4824× 10 −1 𝑧 2 If you now use, z = 2x – 1 𝑝 ∗ 𝑥 =0.8015− 𝑥−1 −0.4824× 10 −1 2𝑥−1 2 =1.030−0.3605𝑥−0.193 𝑥 2 Least square polynomial is unique! It does not depend on the basis!