Ordinary Least-Squares

Outline Linear regression Geometry of least-squares Discussion of the Gauss-Markov theorem Ordinary Least-Squares

One-dimensional regression Ordinary Least-Squares

One-dimensional regression Find a line that represent the ”best” linear relationship: Ordinary Least-Squares

One-dimensional regression Problem: the data does not go through a line Ordinary Least-Squares

One-dimensional regression Problem: the data does not go through a line Find the line that minimizes the sum: Ordinary Least-Squares

One-dimensional regression Problem: the data does not go through a line Find the line that minimizes the sum: We are looking for that minimizes Ordinary Least-Squares

Matrix notation and Using the following notations Ordinary Least-Squares

Matrix notation and Using the following notations we can rewrite the error function using linear algebra as: Ordinary Least-Squares

Matrix notation and Using the following notations we can rewrite the error function using linear algebra as: Ordinary Least-Squares

Multidimentional linear regression Using a model with m parameters Ordinary Least-Squares

Multidimentional linear regression Using a model with m parameters Ordinary Least-Squares

Multidimentional linear regression Using a model with m parameters Ordinary Least-Squares

Multidimentional linear regression Using a model with m parameters and n measurements Ordinary Least-Squares

Multidimentional linear regression Using a model with m parameters and n measurements Ordinary Least-Squares

Ordinary Least-Squares

Ordinary Least-Squares

parameter 1 Ordinary Least-Squares

parameter 1 measurement n Ordinary Least-Squares

Minimizing Ordinary Least-Squares

Minimizing Ordinary Least-Squares

Minimizing is flat at Ordinary Least-Squares

Minimizing is flat at Ordinary Least-Squares

Minimizing is flat at does not go down around Ordinary Least-Squares

Minimizing is flat at does not go down around Ordinary Least-Squares

Positive semi-definite In 1-D In 2-D Ordinary Least-Squares

Minimizing Ordinary Least-Squares

Minimizing Ordinary Least-Squares

Minimizing Ordinary Least-Squares

Minimizing Always true Ordinary Least-Squares

Minimizing The normal equation Always true Ordinary Least-Squares

Geometric interpretation Ordinary Least-Squares

Geometric interpretation b is a vector in Rn Ordinary Least-Squares

Geometric interpretation b is a vector in Rn The columns of A define a vector space range(A) Ordinary Least-Squares

Geometric interpretation b is a vector in Rn The columns of A define a vector space range(A) Ax is an arbitrary vector in range(A) Ordinary Least-Squares

Geometric interpretation b is a vector in Rn The columns of A define a vector space range(A) Ax is an arbitrary vector in range(A) Ordinary Least-Squares

Geometric interpretation is the orthogonal projection of b onto range(A) Ordinary Least-Squares

The normal equation: Ordinary Least-Squares

The normal equation: Existence: has always a solution Ordinary Least-Squares

The normal equation: Existence: has always a solution Uniqueness: the solution is unique if the columns of A are linearly independent Ordinary Least-Squares

The normal equation: Existence: has always a solution Uniqueness: the solution is unique if the columns of A are linearly independent Ordinary Least-Squares

Under-constrained problem Ordinary Least-Squares

Under-constrained problem Ordinary Least-Squares

Under-constrained problem Ordinary Least-Squares

Under-constrained problem Poorly selected data One or more of the parameters are redundant Ordinary Least-Squares

Under-constrained problem Poorly selected data One or more of the parameters are redundant Add constraints Ordinary Least-Squares

How good is the least-squares criteria? Optimality: the Gauss-Markov theorem Ordinary Least-Squares

How good is the least-squares criteria? Optimality: the Gauss-Markov theorem Let and be two sets of random variables and define: Ordinary Least-Squares

How good is the least-squares criteria? Optimality: the Gauss-Markov theorem Let and be two sets of random variables and define: If Ordinary Least-Squares

How good is the least-squares criteria? Optimality: the Gauss-Markov theorem Let and be two sets of random variables and define: If Then is the best unbiased linear estimator Ordinary Least-Squares

b ei a no errors in ai Ordinary Least-Squares

b b ei ei a a no errors in ai errors in ai Ordinary Least-Squares

b a homogeneous errors Ordinary Least-Squares

b b a a homogeneous errors non-homogeneous errors Ordinary Least-Squares

b a no outliers Ordinary Least-Squares

outliers b b a a no outliers outliers Ordinary Least-Squares