1. Ordinary Least-Squares

2. Outline Linear regression Geometry of least-squares
Discussion of the Gauss-Markov theorem
Ordinary Least-Squares

3. One-dimensional regression
Ordinary Least-Squares

4. One-dimensional regression
Find a line that represent the
”best” linear relationship:
Ordinary Least-Squares

5. One-dimensional regression
Problem: the data does not go through a line
Ordinary Least-Squares

6. One-dimensional regression
Problem: the data does not go through a line
Find the line that minimizes the sum:
Ordinary Least-Squares

7. One-dimensional regression
Problem: the data does not go through a line
Find the line that minimizes the sum:
We are looking for that minimizes
Ordinary Least-Squares

8. Matrix notation and Using the following notations
Ordinary Least-Squares

9. Matrix notation and Using the following notations
we can rewrite the error function using linear algebra as:
Ordinary Least-Squares

10. Matrix notation and Using the following notations
we can rewrite the error function using linear algebra as:
Ordinary Least-Squares

11. Multidimentional linear regression
Using a model with m parameters
Ordinary Least-Squares

12. Multidimentional linear regression
Using a model with m parameters
Ordinary Least-Squares

13. Multidimentional linear regression
Using a model with m parameters
Ordinary Least-Squares

14. Multidimentional linear regression
Using a model with m parameters
and n measurements
Ordinary Least-Squares

15. Multidimentional linear regression
Using a model with m parameters
and n measurements
Ordinary Least-Squares

16. Ordinary Least-Squares

17. Ordinary Least-Squares

18. parameter 1
Ordinary Least-Squares

19. parameter 1
measurement n
Ordinary Least-Squares

20. Minimizing
Ordinary Least-Squares

21. Minimizing
Ordinary Least-Squares

22. Minimizing
is flat at
Ordinary Least-Squares

23. Minimizing
is flat at
Ordinary Least-Squares

24. Minimizing
is flat at
does not go down
around
Ordinary Least-Squares

25. Minimizing
is flat at
does not go down
around
Ordinary Least-Squares

26. Positive semi-definite
In 1-D
In 2-D
Ordinary Least-Squares

27. Minimizing
Ordinary Least-Squares

28. Minimizing
Ordinary Least-Squares

29. Minimizing
Ordinary Least-Squares

30. Minimizing
Always true
Ordinary Least-Squares

31. Minimizing
The normal equation
Always true
Ordinary Least-Squares

32. Geometric interpretation
Ordinary Least-Squares

33. Geometric interpretation
b is a vector in Rn
Ordinary Least-Squares

34. Geometric interpretation
b is a vector in Rn
The columns of A define a vector space range(A)
Ordinary Least-Squares

35. Geometric interpretation
b is a vector in Rn
The columns of A define a vector space range(A)
Ax is an arbitrary vector in range(A)
Ordinary Least-Squares

36. Geometric interpretation
b is a vector in Rn
The columns of A define a vector space range(A)
Ax is an arbitrary vector in range(A)
Ordinary Least-Squares

37. Geometric interpretation
is the orthogonal projection of b onto range(A)
Ordinary Least-Squares

38. The normal equation:
Ordinary Least-Squares

39. The normal equation: Existence: has always a solution
Ordinary Least-Squares

40. The normal equation: Existence: has always a solution
Uniqueness: the solution is unique if the columns of A are linearly independent
Ordinary Least-Squares

41. The normal equation: Existence: has always a solution
Uniqueness: the solution is unique if the columns of A are linearly independent
Ordinary Least-Squares

42. Under-constrained problem
Ordinary Least-Squares

43. Under-constrained problem
Ordinary Least-Squares

44. Under-constrained problem
Ordinary Least-Squares

45. Under-constrained problem
Poorly selected data
One or more of the
parameters are redundant
Ordinary Least-Squares

46. Under-constrained problem
Poorly selected data
One or more of the
parameters are redundant
Add constraints
Ordinary Least-Squares

47. How good is the least-squares criteria?
Optimality: the Gauss-Markov theorem
Ordinary Least-Squares

48. How good is the least-squares criteria?
Optimality: the Gauss-Markov theorem
Let and be two sets of random variables
and define:
Ordinary Least-Squares

49. How good is the least-squares criteria?
Optimality: the Gauss-Markov theorem
Let and be two sets of random variables
and define: If
Ordinary Least-Squares

50. How good is the least-squares criteria?
Optimality: the Gauss-Markov theorem
Let and be two sets of random variables
and define: If
Then is the
best unbiased linear estimator
Ordinary Least-Squares

51. b ei a
no errors in ai
Ordinary Least-Squares

52. b b ei ei a a
no errors in ai
errors in ai
Ordinary Least-Squares

53. b a
homogeneous errors
Ordinary Least-Squares

54. b b a a homogeneous errors non-homogeneous errors
Ordinary Least-Squares

55. b a
no outliers
Ordinary Least-Squares

56. outliers b b a a
no outliers
outliers
Ordinary Least-Squares