1. Approximation of Functions
Approximation problems divided into five parts:
Least square approximation of discrete functions or Regression
Least square approximation of continuous function using various basis polynomials
Orthogonal basis functions
Approximation of periodic functions
Interpolation

2. Norm and Seminorm
A real valued function is called a norm on a vector space if it is defined everywhere on the space and satisfies the following conditions:
𝑓 >0 for𝑓≠0; 𝑓 =0 iff 𝑓=0 ∀𝑥∈ 𝑎,𝑏
𝛼𝑓 = 𝛼 𝑓 for a real 𝛼
𝑓+𝑔 ≤ 𝑓 + 𝑔
Lp-Norm: 𝑓 𝑝 = 𝑎 𝑏 𝑓 𝑥 𝑝 𝑑𝑥 1 𝑝
p = 2: Euclidean or L2 norm, 𝑓 2 = 𝑎 𝑏 𝑓 𝑥 2 𝑑𝑥 1 2
p → ∞: 𝑓 ∞ = max 𝑥∈ 𝑎,𝑏 𝑓 𝑥
Weighted Euclidean Norm: 𝑓 2,𝑤 = 𝑎 𝑏 𝑓 𝑥 2 𝑤 𝑥 𝑑𝑥 1 2
Euclidean seminorm: tab 𝑓 2, 𝐺 = 𝑗=0 𝑛 𝑓 𝑥 𝑗

3. Inner Product
The inner product of two real-valued continuous functions f(x) and g(x) is denoted by 𝑓, 𝑔 and is defined as:
𝑓, 𝑔 = 𝑎 𝑏 𝑓 𝑥 𝑔 𝑥 𝑤 𝑥 𝑑𝑥 Continuous Case 𝑖=0 𝑛 𝑓 𝑥 𝑖 𝑔 𝑥 𝑖 𝑤 𝑖 (Discrete Case)
Properties of Inner Product:
Commutativity: 𝑓, 𝑔 = 𝑔, 𝑓
Linearity: (𝑐 1 𝑓+ 𝑐 2 𝑔),𝜑 = 𝑐 1 𝑓, 𝜑 + 𝑐 2 𝑔, 𝜑
Positivity: 𝑓, 𝑓 ≥0
𝑓, 𝑓 = 𝑓 =[ 𝑎 𝑏 𝑓 𝑥 2 𝑑𝑥 ]2

4. Basis Functions: Linear Independence
A sequence of (n + 1) functions 𝜑 0 , 𝜑 1 , 𝜑 2 ,⋯ 𝜑 𝑛 are linearly independent if,
𝑗=0 𝑛 𝑐 𝑗 𝜑 𝑗 =0 ⟹ 𝑐 𝑗 =0 ∀𝑗
This sequence of functions builds (or spans) an (n + 1)-dimensional linear subspace
From the Definition of Norm:
𝑗=0 𝑛 𝑐 𝑗 𝜑 𝑗 =0 is true only if 𝑐 𝑗 =0 ∀𝑗
Linearity of inner product:
𝑗=0 𝑛 𝑐 𝑗 𝜑 𝑗 , 𝜑 𝑘 = 𝑗=0 𝑛 𝑐 𝑗 𝜑 𝑗 𝑥 , 𝜑 𝑘 𝑥

5. Orthogonal Functions
Two real-valued continuous functions f(x) and g(x) are said to be orthogonal if 𝑓, 𝑔 =0
A finite or infinite sequence of functions 𝜑 0 , 𝜑 1 , 𝜑 2 ,⋯ 𝜑 𝑛 ,⋯ make an orthogonal system if 𝜑 𝑖 , 𝜑 𝑗 =0 for all i ≠ j and 𝜑 𝑖 ≠0 for all i.
In addition, if 𝜑 𝑖 =1, the sequence of functions is called an orthonormal system
Pythagorean Theorem for functions: 𝑓, 𝑔 =0⟹ 𝑓+𝑔 2 = 𝑓 𝑔 2
𝑓+𝑔 2 = 𝑓+𝑔 , 𝑓+𝑔 = 𝑓, 𝑓 + 𝑓, 𝑔 + 𝑔, 𝑓 + 𝑔, 𝑔 = 𝑓 𝑔 2
Generalized for orthogonal system: 𝑗=0 𝑛 𝑐 𝑗 𝜑 𝑗 2 = 𝑗=0 𝑛 𝑐 𝑗 𝜑 𝑗 2

6. Least Square Problem
Let f(x) be a continuous real-valued function in (a, b) that is to be approximated by p(x), a linear combination of a system of (n + 1) linearly independent functions 𝜑 0 , 𝜑 1 , 𝜑 2 ,⋯ 𝜑 𝑛 as shown below:
𝑝 𝑥 = 𝑗=0 𝑛 𝑐 𝑗 𝜑 𝑗 𝑥
Determine the coefficients 𝑐 𝑗 = 𝑐 𝑗 ∗ such that, a weighted Euclidean norm or seminorm of error p(x) – f(x) becomes as small as possible.
𝑝 𝑥 −𝑓 𝑥 2 = 𝑎 𝑏 𝑝 𝑥 −𝑓 𝑥 2 𝑑𝑥 is minimum when 𝑐 𝑗 = 𝑐 𝑗 ∗
𝑝 𝑥 −𝑓 𝑥 2 = 𝑖=1 𝑚 𝑝 𝑥 𝑖 −𝑓 𝑥 𝑖 is minimum when 𝑐 𝑗 = 𝑐 𝑗 ∗
Least Square Solution: 𝑝 ∗ 𝑥 = 𝑗=0 𝑛 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥

7. Schematic of Least Square Solution
f(x)
Space Spanned by a system of (n + 1) linearly independent functions 𝜑 0 , 𝜑 1 , 𝜑 2 ,⋯ 𝜑 𝑛
𝑝 𝑥 = 𝑗=0 𝑛 𝑐 𝑗 𝜑 𝑗
𝑝 ∗ 𝑥 = 𝑗=0 𝑛 𝑐 𝑗 ∗ 𝜑 𝑗
Solution: 𝑓 𝑥 − 𝑝 ∗ 𝑥 , 𝜑 𝑘 =0 for k = 0, 1, 2, … n

8. Least Square Solution: Proof
When 𝜑 0 , 𝜑 1 , 𝜑 2 ,⋯ 𝜑 𝑛 are linearly independent, the least square problem has a unique solution:
𝑝 ∗ 𝑥 = 𝑗=0 𝑛 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥
where, p*(x) – f(x) is orthogonal to all φj’s, j = 0, 1, 2, … n.
We need to prove:
𝑝 ∗ 𝑥 −𝑓 𝑥 2 is minimum when p*(x) – f(x) is orthogonal to all φj’s, j = 0, 1, 2, … n
existence and uniqueness of the least square solution

9. Least Square Solution: Proof
Let 𝑐 0 , 𝑐 1 , 𝑐 2 ,⋯ 𝑐 𝑛 be another sequence of coefficients with 𝑐 𝑗 ≠ 𝑐 𝑗 ∗ for at least one j, then 𝑗=0 𝑛 𝑐 𝑗 𝜑 𝑗 𝑥 −𝑓 𝑥 = 𝑗=0 𝑛 𝑐 𝑗 − 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥 + 𝑝 ∗ 𝑥 −𝑓 𝑥 If p*(x) – f(x) is orthogonal to all φj’s, it is also orthogonal to their linear combination 𝑗=0 𝑛 𝑐 𝑗 − 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥 . According to Pythagorean theorem, we have: 𝑗=0 𝑛 𝑐 𝑗 𝜑 𝑗 𝑥 −𝑓 𝑥 2 = 𝑗=0 𝑛 𝑐 𝑗 − 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥 2 + 𝑝 ∗ 𝑥 −𝑓 𝑥 2 > 𝑝 ∗ 𝑥 −𝑓 𝑥 2 Therefore, if p*(x) – f(x) is orthogonal to all φj’s, then p*(x) is the solution of the least square problem.

10. Least Square Solution: Normal Equations
If 𝜑 0 , 𝜑 1 , 𝜑 2 ,⋯ 𝜑 𝑛 are linearly independent, solution to the least square problem is:
𝑝 ∗ 𝑥 −𝑓 𝑥 , 𝜑 𝑘 𝑥 =0 𝑘=0, 1, 2,⋯𝑛
where, 𝑝 ∗ 𝑥 = 𝑗=0 𝑛 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥
Therefore,
𝑗=0 𝑛 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥 −𝑓 𝑥 , 𝜑 𝑘 𝑥 =0 𝑘=0, 1, 2,⋯𝑛
𝑗=0 𝑛 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥 , 𝜑 𝑘 𝑥 = 𝑓 𝑥 , 𝜑 𝑘 𝑥 𝑘=0, 1, 2,⋯𝑛
Normal Equations!

11. Least Square Solution: Normal Equations
𝑗=0 𝑛 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥 , 𝜑 𝑘 𝑥 = 𝑓 𝑥 , 𝜑 𝑘 𝑥 ; 𝑘=0, 1, 2,⋯𝑛
𝑐 0 ∗ 𝜑 0 , 𝜑 0 + 𝑐 1 ∗ 𝜑 1 , 𝜑 0 + 𝑐 2 ∗ 𝜑 2 , 𝜑 0 +⋯+ 𝑐 𝑛 ∗ 𝜑 𝑛 , 𝜑 0 = 𝑓, 𝜑 𝑘=0
𝑐 0 ∗ 𝜑 0 , 𝜑 1 + 𝑐 1 ∗ 𝜑 1 , 𝜑 1 + 𝑐 2 ∗ 𝜑 2 , 𝜑 1 +⋯+ 𝑐 𝑛 ∗ 𝜑 𝑛 , 𝜑 1 = 𝑓, 𝜑 𝑘=1
𝑐 0 ∗ 𝜑 0 , 𝜑 2 + 𝑐 1 ∗ 𝜑 1 , 𝜑 2 + 𝑐 2 ∗ 𝜑 2 , 𝜑 2 +⋯+ 𝑐 𝑛 ∗ 𝜑 𝑛 , 𝜑 2 = 𝑓, 𝜑 𝑘=2
⋮ ⋮ ⋮ ⋯ ⋮ = ⋮
𝑐 0 ∗ 𝜑 0 , 𝜑 𝑛 + 𝑐 1 ∗ 𝜑 1 , 𝜑 𝑛 + 𝑐 2 ∗ 𝜑 2 , 𝜑 𝑛 +⋯+ 𝑐 𝑛 ∗ 𝜑 𝑛 , 𝜑 𝑛 = 𝑓, 𝜑 𝑛 𝑘=𝑛
Moreover, if 𝜑 0 , 𝜑 1 , 𝜑 2 ,⋯ 𝜑 𝑛 is an orthogonal system:
𝑐 𝑘 ∗ = 𝑓, 𝜑 𝑘 𝜑 𝑘 , 𝜑 𝑘 𝑘=0, 1, 2,⋯𝑛

12. Least Square Solution: Existence and Uniqueness
𝑗=0 𝑛 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥 , 𝜑 𝑘 𝑥 = 𝑓 𝑥 , 𝜑 𝑘 𝑥 ; 𝑘=0, 1, 2,⋯𝑛
𝑐 0 ∗ 𝜑 0 , 𝜑 0 + 𝑐 1 ∗ 𝜑 1 , 𝜑 0 + 𝑐 2 ∗ 𝜑 2 , 𝜑 0 +⋯+ 𝑐 𝑛 ∗ 𝜑 𝑛 , 𝜑 0 = 𝑓, 𝜑 𝑘=0
𝑐 0 ∗ 𝜑 0 , 𝜑 1 + 𝑐 1 ∗ 𝜑 1 , 𝜑 1 + 𝑐 2 ∗ 𝜑 2 , 𝜑 1 +⋯+ 𝑐 𝑛 ∗ 𝜑 𝑛 , 𝜑 1 = 𝑓, 𝜑 𝑘=1
𝑐 0 ∗ 𝜑 0 , 𝜑 2 + 𝑐 1 ∗ 𝜑 1 , 𝜑 2 + 𝑐 2 ∗ 𝜑 2 , 𝜑 2 +⋯+ 𝑐 𝑛 ∗ 𝜑 𝑛 , 𝜑 2 = 𝑓, 𝜑 𝑘=2
⋮ ⋮ ⋮ ⋯ ⋮ = ⋮
𝑐 0 ∗ 𝜑 0 , 𝜑 𝑛 + 𝑐 1 ∗ 𝜑 1 , 𝜑 𝑛 + 𝑐 2 ∗ 𝜑 2 , 𝜑 𝑛 +⋯+ 𝑐 𝑛 ∗ 𝜑 𝑛 , 𝜑 𝑛 = 𝑓, 𝜑 𝑛 𝑘=𝑛
Solution to the normal equations exist and is unique unless the following homogenous system has a nontrivial solution for 𝑐 0 ∗ , 𝑐 1 ∗ ,⋯ 𝑐 𝑛 ∗ , i.e., 𝑐 𝑗 ∗ ≠0 for at least one j:
𝑗=0 𝑛 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥 , 𝜑 𝑘 𝑥 = 0

13. Least Square Solution: Existence and Uniqueness
Solution to the normal equations exist and is unique unless the following homogenous system has a nontrivial solution for 𝑐 0 ∗ , 𝑐 1 ∗ ,⋯ 𝑐 𝑛 ∗ , i.e., 𝑐 𝑗 ∗ ≠0 for at least one j: 𝑗=0 𝑛 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥 , 𝜑 𝑘 𝑥 = 0 This would lead to 𝑗=0 𝑛 𝑐 𝑗 ∗ 𝜑 𝑗 2 = 𝑗=0 𝑛 𝑐 𝑗 ∗ 𝜑 𝑗 , 𝑘=0 𝑛 𝑐 𝑘 ∗ 𝜑 𝑘 = 𝑘=0 𝑛 𝑗=0 𝑛 𝜑 𝑗 , 𝜑 𝑘 𝑐 𝑗 ∗ 𝑐 𝑘 ∗ = 𝑘=0 𝑛 0. 𝑐 𝑘 ∗ =0 which contradicts that the 𝜑 0 , 𝜑 1 , 𝜑 2 ,⋯ 𝜑 𝑛 are linearly independent.

14. Least Square Solution: Example (Continuous)
Approximate the function f(x) = 1/(1 + x2) for x in [0, 1] using a straight line.
The basis functions are: 𝜑 0 𝑥 = 𝜑 0 =1; 𝜑 1 𝑥 = 𝜑 1 =𝑥; 𝑝 ∗ 𝑥 = 𝑗=0 1 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥
Normal Equation: 𝑗=0 𝑛 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥 , 𝜑 𝑘 𝑥 = 𝑓 𝑥 , 𝜑 𝑘 𝑥 ; 𝑘=0, 1, 2,⋯𝑛
𝑐 0 ∗ 𝜑 0 , 𝜑 0 + 𝑐 1 ∗ 𝜑 1 , 𝜑 0 = 𝑓, 𝜑 0
𝑐 0 ∗ 𝜑 0 , 𝜑 1 + 𝑐 1 ∗ 𝜑 1 , 𝜑 1 = 𝑓, 𝜑 1
𝜑 0 , 𝜑 0 = 𝑑𝑥 =1; 𝜑 0 , 𝜑 1 = 𝜑 1 , 𝜑 0 = 0 1 𝑥 1 𝑑𝑥 = 1 2 =0.5
𝜑 1 , 𝜑 1 = 0 1 𝑥 𝑥 𝑑𝑥 = 1 3 ; 𝑓, 𝜑 0 = 𝑥 𝑑𝑥 = 𝜋 4
𝑓, 𝜑 1 = 𝑥 2 𝑥 𝑑𝑥 = ln 2 2
𝑓, 𝑔 = 𝑎 𝑏 𝑓 𝑥 𝑔 𝑥 𝑤 𝑥 𝑑𝑥 Continuous Case

15. Least Square Solution: Example (Continuous)
𝑐 0 ∗ 𝜑 0 , 𝜑 0 + 𝑐 1 ∗ 𝜑 1 , 𝜑 0 = 𝑓, 𝜑 0 𝑐 0 ∗ 𝜑 0 , 𝜑 1 + 𝑐 1 ∗ 𝜑 1 , 𝜑 1 = 𝑓, 𝜑 /3 𝑐 0 ∗ 𝑐 1 ∗ = 𝜋/4 ln 2 /2 𝑐 0 ∗ = 𝜋/4 0.5 ln 2 /2 1/ /3 =1.062 𝑐 1 ∗ = 1 𝜋/4 0.5 ln 2 / /3 =− 𝑝 ∗ 𝑥 = 𝑗=0 1 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥 =1.062−0.5535𝑥

16. Least Square Solution: Example (Continuous)
Approximate the function f(x) = 1/(1 + x2) for x in [0, 1] using a 2nd order polynomial. The basis functions are: 𝜑 0 =1; 𝜑 1 =𝑥; 𝜑 2 = 𝑥 2 ; 𝑝 ∗ 𝑥 = 𝑗=0 2 𝑐 𝑗 ∗ 𝜑 𝑗 𝑐 0 ∗ 𝜑 0 , 𝜑 0 + 𝑐 1 ∗ 𝜑 1 , 𝜑 0 + 𝑐 2 ∗ 𝜑 2 , 𝜑 0 = 𝑓, 𝜑 0 𝑐 0 ∗ 𝜑 0 , 𝜑 1 + 𝑐 1 ∗ 𝜑 1 , 𝜑 1 + 𝑐 2 ∗ 𝜑 2 , 𝜑 1 = 𝑓, 𝜑 1 𝑐 0 ∗ 𝜑 0 , 𝜑 2 + 𝑐 1 ∗ 𝜑 1 , 𝜑 2 + 𝑐 2 ∗ 𝜑 2 , 𝜑 2 = 𝑓, 𝜑 2 Additional inner products to be evaluated are: 𝜑 0 , 𝜑 2 = 𝜑 2 , 𝜑 0 = 0 1 𝑥 2 1 𝑑𝑥 = 1 3 𝜑 1 , 𝜑 2 = 𝜑 2 , 𝜑 1 = 0 1 𝑥 2 𝑥 𝑑𝑥 = 1 4 =0.25; 𝜑 2 , 𝜑 2 = 0 1 𝑥 2 𝑥 2 𝑑𝑥 = 1 5 =0.2 𝑓, 𝜑 2 = 𝑥 2 𝑥 2 𝑑𝑥 =1− π 4

17. Least Square Solution: Example (Continuous)
𝑐 0 ∗ 𝜑 0 , 𝜑 0 + 𝑐 1 ∗ 𝜑 1 , 𝜑 0 + 𝑐 2 ∗ 𝜑 2 , 𝜑 0 = 𝑓, 𝜑 0 𝑐 0 ∗ 𝜑 0 , 𝜑 1 + 𝑐 1 ∗ 𝜑 1 , 𝜑 1 + 𝑐 2 ∗ 𝜑 2 , 𝜑 1 = 𝑓, 𝜑 1 𝑐 0 ∗ 𝜑 0 , 𝜑 2 + 𝑐 1 ∗ 𝜑 1 , 𝜑 2 + 𝑐 2 ∗ 𝜑 2 , 𝜑 2 = 𝑓, 𝜑 / / / 𝑐 0 ∗ 𝑐 1 ∗ 𝑐 2 ∗ = 𝜋/4 ln 2 /2 1−𝜋/4 Solving by Gauss Elimination: 𝑐 0 ∗ =1.030; 𝑐 1 ∗ =−0.3605; 𝑐 2 ∗ =− 𝑝 ∗ 𝑥 = 𝑗=0 2 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥 =1.03−0.3605𝑥−0.193 𝑥 2

18. Least Square Solution: Example (Continuous)
How do you judge how good the fit is or how do you compare fit of two polynomials?
Estimate 𝑝 ∗ 𝑥 −𝑓 𝑥 ∞ . Two options:
Analytical (+ numerical)
Numerical (+ visual)

19. Discrete Data
(n + 1) observations or data pairs [(x0, y0), (x1, y1), (x2, y2) … (xn, yn)]
(m + 1) basis functions: 𝜑 0 , 𝜑 1 , 𝜑 2 ,⋯ 𝜑 𝑚
Approximating polynomial: 𝑝 𝑥 = 𝑗=0 𝑚 𝑐 𝑗 𝜑 𝑗 𝑥
𝑐 0 𝜑 0 𝑥 0 + 𝑐 1 𝜑 1 𝑥 0 + 𝑐 2 𝜑 2 𝑥 0 +⋯+ 𝑐 𝑚 𝜑 𝑚 𝑥 0 = 𝑦 0
𝑐 0 𝜑 0 𝑥 1 + 𝑐 1 𝜑 1 𝑥 1 + 𝑐 2 𝜑 2 𝑥 1 +⋯+ 𝑐 𝑚 𝜑 𝑚 𝑥 1 = 𝑦 1
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
𝑐 0 𝜑 0 𝑥 𝑛 + 𝑐 1 𝜑 1 𝑥 𝑛 + 𝑐 2 𝜑 2 𝑥 𝑛 +⋯+ 𝑐 𝑚 𝜑 𝑚 𝑥 𝑛 = 𝑦 𝑛
n equations, m unknowns:
m < n: over-determined system, least square regression
m = n: unique solution, interpolation
m > n: under-determined system

20. Least Square Solution: Example (Discrete Data)
Variation of Ultimate Shear Strength (y) with curing Temperature (x) for a certain rubber compound was reported (J. Quality Technology, 1971, pp ) as: Fit a linear and a quadratic regression model to the data. For the linear model, the basis functions and the polynomial are: 𝜑 0 𝑥 = 𝜑 0 =1; 𝜑 1 𝑥 = 𝜑 1 =𝑥; 𝑝 ∗ 𝑥 = 𝑗=0 1 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥 𝑐 0 ∗ 𝜑 0 , 𝜑 0 + 𝑐 1 ∗ 𝜑 1 , 𝜑 0 = 𝑓, 𝜑 0 𝑐 0 ∗ 𝜑 0 , 𝜑 1 + 𝑐 1 ∗ 𝜑 1 , 𝜑 1 = 𝑓, 𝜑 1 Given vectors: x = [138, 140, 144.5, 146, 148, 151.5, 153.5, 157] and y = f(x) = [770, 800, 840, 810, 735, 640, 590, 560] No. of data points, m = 8. Denote elements of x and y as xi and yi, respectively.
x, in oC
138
140
144.5
146
148
151.5
153.5
157
y, in psi
770
800
840
810
735
640
590
560

21. Least Square Solution: Example (Discrete Data)
𝜑 0 , 𝜑 0 = 𝑖=1 𝑚 1 1 =8; 𝜑 1 , 𝜑 1 = 𝑖=1 𝑚 𝑥 𝑖 𝑥 𝑖 = 𝑖=1 𝑚 𝑥 𝑖 2 = 𝜑 1 , 𝜑 0 = 𝜑 0 , 𝜑 1 = 𝑖=1 𝑚 1 𝑥 𝑖 = 𝑖=1 𝑚 𝑥 𝑖 = 𝑓, 𝜑 0 = 𝑖=1 𝑚 𝑦 𝑖 1 =5745; 𝑓, 𝜑 1 = 𝑖=1 𝑚 𝑦 𝑖 𝑥 𝑖 = 𝑐 0 ∗ 𝜑 0 , 𝜑 0 + 𝑐 1 ∗ 𝜑 1 , 𝜑 0 = 𝑓, 𝜑 0 𝑐 0 ∗ 𝜑 0 , 𝜑 1 + 𝑐 1 ∗ 𝜑 1 , 𝜑 1 = 𝑓, 𝜑 𝑐 0 ∗ 𝑐 1 ∗ = ⟹ 𝑐 0 ∗ = ; 𝑐 1 ∗ =− 𝑝 ∗ 𝑥 = 𝑗=0 1 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥 =2773.5− 𝑥

22. Least Square Solution: Example (Discrete Data)
For the quadratic model, the basis functions and the polynomial are: 𝜑 0 =1; 𝜑 1 =𝑥; 𝜑 2 = 𝑥 2 ; 𝑝 ∗ 𝑥 = 𝑗=0 2 𝑐 𝑗 ∗ 𝜑 𝑗 𝑐 0 ∗ 𝜑 0 , 𝜑 0 + 𝑐 1 ∗ 𝜑 1 , 𝜑 0 + 𝑐 2 ∗ 𝜑 2 , 𝜑 0 = 𝑓, 𝜑 0 𝑐 0 ∗ 𝜑 0 , 𝜑 1 + 𝑐 1 ∗ 𝜑 1 , 𝜑 1 + 𝑐 2 ∗ 𝜑 2 , 𝜑 1 = 𝑓, 𝜑 1 𝑐 0 ∗ 𝜑 0 , 𝜑 2 + 𝑐 1 ∗ 𝜑 1 , 𝜑 2 + 𝑐 2 ∗ 𝜑 2 , 𝜑 2 = 𝑓, 𝜑 2 Additional inner products to be evaluated are: 𝜑 0 , 𝜑 2 = 𝜑 2 , 𝜑 0 = 𝑖=1 𝑚 1 𝑥 𝑖 2 = 𝜑 1 , 𝜑 2 = 𝜑 2 , 𝜑 1 = 𝑖=1 𝑚 𝑥 𝑖 𝑥 𝑖 2 = 𝑖=1 𝑚 𝑥 𝑖 3 = 𝜑 2 , 𝜑 2 = 𝑖=1 𝑚 𝑥 𝑖 2 𝑥 𝑖 2 = 𝑖=1 𝑚 𝑥 𝑖 4 = 𝑓, 𝜑 2 = 𝑖=1 𝑚 𝑦 𝑖 𝑥 𝑖 2 =

23. Least Square Solution: Example (Discrete Data)
𝑐 0 ∗ 𝜑 0 , 𝜑 0 + 𝑐 1 ∗ 𝜑 1 , 𝜑 0 + 𝑐 2 ∗ 𝜑 2 , 𝜑 0 = 𝑓, 𝜑 0 𝑐 0 ∗ 𝜑 0 , 𝜑 1 + 𝑐 1 ∗ 𝜑 1 , 𝜑 1 + 𝑐 2 ∗ 𝜑 2 , 𝜑 1 = 𝑓, 𝜑 1 𝑐 0 ∗ 𝜑 0 , 𝜑 2 + 𝑐 1 ∗ 𝜑 1 , 𝜑 2 + 𝑐 2 ∗ 𝜑 2 , 𝜑 2 = 𝑓, 𝜑 𝑐 0 ∗ 𝑐 1 ∗ 𝑐 2 ∗ = Solving by Gauss Elimination: 𝑐 0 ∗ =− ; 𝑐 1 ∗ = ; 𝑐 2 ∗ =− 𝑝 ∗ 𝑥 = 𝑗=0 2 𝑐 𝑗 ∗ 𝜑 𝑗 𝑥 =− 𝑥− 𝑥 2

24. Least Square Solution: Example (Discrete Data)
How do you judge how good the fit is or how do you compare fit of two polynomials?
Square of errors!

25. Least Square Solution: Example (Discrete Data)
Denote: 𝑦 𝑖 =𝑓 𝑥 𝑖 , 𝑦 𝑖 = 𝑝 ∗ 𝑥 𝑖
𝜇 𝑦 = 𝑖=0 𝑚 𝑦 𝑖 𝑚+1 , 𝜎 𝑡 2 = 𝑖=0 𝑚 𝑦 𝑖 − 𝜇 𝑦 2
𝜀 2 = 𝑖=0 𝑚 𝑦 𝑖 − 𝑦 𝑖 2
R2 = 0.73
Coefficient of regression r or R is given by:
𝑟 2 = 𝜎 𝑡 2 − 𝜀 2 𝜎 𝑡 2
R2 = 0.88

26. Additional Points
You can do multiple regression using the same frame work.
Linearize some non-linear equations
Evaluate Integrals using Numerical Methods for Integration for functions that are not easy to integrate using analytical means!