1. Least Squares Approximations
Prepared by Vince Zaccone
For Campus Learning Assistance Services at UCSB

2. Least Squares Approximations
Suppose you have an inconsistent linear system (i.e. there are no solutions).
How do you find a vector that comes close to solving the system?
We will use a method called “Least-Squares” Approximation.
We have the following system:
𝐴 𝑥 = 𝑏
Unfortunately there are no exact solutions.
We will look for a solution of the related system:
𝐴 𝑥 = 𝑏
In this equation, is the orthogonal projection of onto the column space of A.
In other words, is the vector that A maps to that is “closest” to . 𝑏 𝑏 𝑏 𝑏
Prepared by Vince Zaccone
For Campus Learning Assistance Services at UCSB

3. Least Squares Approximations
𝐴 𝑥 = 𝑏
We start with this system, which is inconsistent.
𝐴 𝑥 = 𝑏
Instead, we will look for a solution of this related system.
A least-squares solution will satisfy the equation
𝐴 𝑇 𝐴 𝑥 = 𝐴 𝑇 𝑏
An example is on the next slide.
Prepared by Vince Zaccone
For Campus Learning Assistance Services at UCSB

4. Least Squares Approximations
Here is an example:
We have the following system:
1 2 − 𝑥 = 3 −1 5
Prepared by Vince Zaccone
For Campus Learning Assistance Services at UCSB

5. Least Squares Approximations
Here is an example:
We have the following system:
1 2 − 𝑥 = 3 −1 5
A least-squares solution will satisfy the equation
𝐴 𝑇 𝐴 𝑥 = 𝐴 𝑇 𝑏
Prepared by Vince Zaccone
For Campus Learning Assistance Services at UCSB

6. Least Squares Approximations
Here is an example:
We have the following system:
1 2 − 𝑥 = 3 −1 5
A least-squares solution will satisfy the equation
𝐴 𝑇 𝐴 𝑥 = 𝐴 𝑇 𝑏
𝐴 𝑇 𝑏 = 1 − ∙ 3 −1 5 = 9 12
Prepared by Vince Zaccone
For Campus Learning Assistance Services at UCSB

7. Least Squares Approximations
Here is an example:
We have the following system:
1 2 − 𝑥 = 3 −1 5
A least-squares solution will satisfy the equation
𝐴 𝑇 𝐴 𝑥 = 𝐴 𝑇 𝑏
𝐴 𝑇 𝑏 = 1 − ∙ 3 −1 5 = 9 12
This is our least-squares solution
𝑥 = → 𝑥 =
Prepared by Vince Zaccone
For Campus Learning Assistance Services at UCSB

8. Least Squares Approximations
Here is an example:
We have the following system:
1 2 − 𝑥 = 3 −1 5
A least-squares solution will satisfy the equation
𝐴 𝑇 𝐴 𝑥 = 𝐴 𝑇 𝑏
This matrix came out diagonal because the columns of the original matrix were orthogonal
This is our least-squares solution
𝑥 = → 𝑥 =
Prepared by Vince Zaccone
For Campus Learning Assistance Services at UCSB

9. Least Squares Approximations
Here is an example:
We have the following system:
1 2 − 𝑥 = 3 −1 5
𝑥 =
This is our least-squares solution
We can find the “error” by comparing this with the original system.
Least-Squares Error = 𝑏 −𝐴 𝑥
𝑏 −𝐴 𝑥 =𝑒𝑟𝑟𝑜𝑟
Prepared by Vince Zaccone
For Campus Learning Assistance Services at UCSB

10. Least Squares Approximations
Here is an example:
We have the following system:
1 2 − 𝑥 = 3 −1 5
𝑥 =
This is our least-squares solution
We can find the “error” by comparing this with the original system.
Least-Squares Error = 𝑏 −𝐴 𝑥
𝑏 −𝐴 𝑥 =𝑒𝑟𝑟𝑜𝑟
𝑏 −𝐴 𝑥 = 3 −1 5 − 4 −1 4 = −1 0 1
𝑏 −𝐴 𝑥 = 2
The error in any approximation to the original system will be at least this large.
Prepared by Vince Zaccone
For Campus Learning Assistance Services at UCSB