Linear Algebra Curve Fitting

Last Class: Curve Fitting

Sum-of-Squares Error Function

0 th Order Polynomial

1 st Order Polynomial

3 rd Order Polynomial

9 th Order Polynomial

Over-fitting Root-Mean-Square (RMS) Error:

Polynomial Coefficients

Data Set Size: 9 th Order Polynomial

Data Set Size: 9 th Order Polynomial

Regularization Penalize large coefficient values

Regularization:

Regularization: vs.

Polynomial Coefficients

Methodology Regression Machine Training Regression Machine Update weights w and degree M s.t. min error Regression

Questions ● How to find the curve that best fits the points? ● i.e., how to solve the minimization problem? ● Answer 1: use gnuplot or any other pre-built software (homework) ● Answer 2: recall linear algebra (will see with more details in upcoming class) ● Why solve this particular minimization problem? e.g., why solve this minimization problem rather than doing linear interpolation? Principled answer follows...

Linear Algebra ● To find curve that best fits points: apply basic concepts from linear algebra ● What is linear algebra? ● The study of vector spaces ● What are vector spaces? ● Set of elements that can be ● Summed ● Multiplied by scalar Operation is well defined

Linear Algebra=Study of Vectors ● Examples of vectors ● [0,1,0,2] ● ● Signals ● Signals are vectors!

Dimension of Vector Space ● Basis = set of element that span space ● Dimension of line = 1 ● basis = (1,0) ● Dimension of plane = 2 ● basis = {(1,0), (0,1)} ● Dimension of space = 3 ● basis = {(1,0,0), (0,1,0), (0,0,1)} ● Dimension of Space = # Elements Basis

Functions are Vectors I ● Vectors Functions ● Size Size ● x=[x1,x2] y=f(t) ● |x|^2=x1^2+x2^2 |y|^2 = ● Inner product Inner Product ● y=[y1,y2] z=g(t) ● x y = x1 y1 + x2 y2

Signals are Vectors II ● Vectors Functions ● x=[x1,x2] y=f(t) ● BasisBasis ● {[1,0],[1,0]} {1, sin wt, cos wt, sin 2wt, cos 2wt,...} ● Components Components ● x1 coefficients of basis x2 functions

A Very Special Basis ● Vectors Functions ● x=[x1,x2] y=f(t) ● BasisBasis ● {[1,0],[1,0]} {1, sin wt, cos wt, sin 2wt, cos 2wt,...} ● Components Components ● x1 coefficients of basis x2 functions Basis in which each element Is a vector, formed by set of M samples of a function (Subspace of R^M)

Representations Coeff. Basis Representation traditional function sampled function [1,2] [0,1] [1,0] {1, sin(wt), … } [0,2,0] {[1,1,1,1], [1,2,3,4], [1,4,9,16]}

Approximating a Vector I ● e = g - c x ● g' x = |g| |x| cos  ● |x|^2 = x' x ● |g| cos  c |x| e g c x x e1 g c1 x x e2 g c2 x x

Approximating a Vector II ● e = g - c x ● g' x = |g| |x| cos  ● |x|^2 = x' x ● |g| cos  c |x| ● Major result: c = g' x (x' x)^(-1) e g c x x

Approximating a Vector III ● e = g - c x ● g' x = |g| |x| cos  ● |x|^2 = x' x ● |g| cos  c |x| ● c =g'x / |x|^2 ● Major result: c = g' x ((x' x)^(-1))' e g c x Coefficient to compute Given target point (basis matrix coords) Given basis matrix x

Approximating a Vector IV ● e = g - x c ● g' x = |g| |x| cos  ● |x|^2 = x' x ● g' x  x c ● Major result: c = g' x ((x' x)^(-1))' c=vector of computed coefficients g=vector of target points x=basis matrix - each column is a basis elem. - each column is a polynomial evaluated at desired points i-th line of matrix x

Approximating a Vector IV ● e = g - x c ● g' x = |g| |x| cos  ● |x|^2 = x' x ● g' x  x c ● Major result: c = g' x ((x' x)^(-1))' Coefficient to compute Given target point (vector space V) Given basis matrix (elements in basis matrix are in vector space U contained in V) c=vector of computed coefficients g=vector of target points x=basis matrix - each column is a basis elem. - each column is a polynomial evaluated at desired points

Least Square Error Solution! ● As most machine learning problems, least square is equivalent to basis conversion! ● Major result: c = g' x ((x' x)^(-1))' c=vector of computed coefficients g=vector of target points x=basis matrix - each column is a basis elem. - each column is a polynomial evaluated at desired points Coefficients to compute (point in R^M) Given points (element in R^N)

Least Square Error Solution! Space of points in R^N Space of functions in R^M Machine learning algorithm maps to closest

Regularized Least Square ● Major result: c = g' x (( I+x' x)^(-1))' c=vector of computed coefficients g=vector of target points x=basis matrix - each column is a basis elem. - each column is a polynomial evaluated at desired points

Least Squares and Gauss ● How to solve the least square problem? ● Carl Friedrich Gauss solution in 1794 (age 18) ● Why solving the least square problem? ● Carl Friedrich Gauss solution in 1822 (age 46) ● Least square solution is optimal in the sense that it is the best linear unbiased estimator of the coefficients of the polynomials ● Assumptions: errors have zero mean and equal variances ●