1. Chapter 10 Real Inner Products and Least-Square (cont.)
In this handout:
Section 10.5: Least-Squares

2. A common problem in business, science, engineering is to collect data and analyze them to predict future events.
If such data are plotted, they constitute a scatter diagram which may provide useful insight into the underlying relationship between system variables.
The data below appears to follow a straight line relationship. The problem is to determine the equation of the straight line that best fits the data. x y

3. Consider an arbitrary straight line, y = b0 + b1 x , to be fitted through these data points. For each data point, the error is the difference between the y-value of the point and the y-value obtained from the straight line approximation.

4. The least-squares straight line
Definition 1: The least-squares error E is the sum of the squares of the individual errors. That is,
Definition 2: The least-squares straight line is the line that minimizes the least-squares error.
We want to find the equation of the least-squares straight line:
y = mx + c
We seek values of m and c that minimizes the least-squares error.

5. The normal equations For each of N data points, the error is
We want the values for m and c that minimize
This occurs when
Or, upon simplifying, when
The last two equations are the normal equations for a least-squares fit in two variables. Examples on the board.

6. Matrix representation of the normal equations
Ideally, we would like to choose m and c so that yi = mxi + c for all data pairs (xi, yi), i=1,2,…,N.
That is, we want the values for m and c that solve the system
or, equivalently, the matrix equation

7. Matrix representation of the normal equations
This system has the standard form Ax=b where x = [m c],
b = [y1 y2 … yN], and A has two columns [x1 x2 … xN] and [1 1 … 1].
Ax=b has a solution if and only if the data falls on a straight line.
If not, then the system is inconsistent, and we seek a solution that minimizes the least-square error:
The least-square solution which is given by the normal equations
has the following matrix form

8. The least-squares solution for any linear system
The concept of linear-squares can be generalized to any linear system Ax=b.
We are primarily interested in cases where the system is inconsistent. This generally occurs when A has more rows than columns.
Measurement errors are inevitable in observational and experimental sciences.
Errors can be smoothed out by averaging over many cases, i.e., taking more measurements than are strictly necessary to determine parameters of system.
Resulting system is overdetermined (more rows than columns), so usually there is no exact solution.
The least-squares is an approximate solution to this kind of systems.

9. The least-squares solution for any linear system
We seek the vector that minimizes the least-squares error defined by
Theorem 1: If x has the property that Ax-b is orthogonal to the columns of A, then x minimizes the least-squares error.
As a consequence to Theorem 1,
x is the least-squares solution to Ax=b
if and only if
x is the solution to
This set of normal equations is guaranteed to have a unique solution whenever the columns of A are linearly independent.
The solution can be found using the techniques of previous chapters.

10. The least-squares solution of a linear system (example)
Find the least squares solution of the linear system Ax = b given by
x1 – x2 = 4
3x1 + 2x2 = 1
-2x1 + 4x2 = 3
Solution:

11. The least-squares solution of a linear system (example)
Solution (cont.):
We have
so the normal system ATAx = ATb in this case is
Solving this system yields the least squares solution
x1 = 17/95, x2 = 143/285