Scientific Computing Linear Least Squares

Interpolation vs Approximation Recall: Given a set of (x,y) data points, Interpolation is the process of finding a function (usually a polynomial) that passes through these data points.

Interpolation vs Approximation Given a set of (x,y) data points, Approximation is the process of finding a function (usually a line or a polynomial) that comes the “closest” to the data points. Data has “noise” – cannot find interpolating line.

General Least Squares Idea Given data and a class of functions F, the goal is to find the “best” function f in F that approximates the data. Consider the data to be two vectors of length n + 1. That is, x = [x 0 x 1 … x n ] t and y = [y 0 y 1 … y n ] t

General Least Squares Idea Definition: The error, or residual, of a given function with respect to this data is the vector r = y - f(x): That is r = [r 0 r 1 … r n ] t where r i = y i - f(x i ) We want to find a function f such that the error is made as small as possible. How do we measure the size of the error? With vector norms.

Vector Norms A vector norm is a quantity that measures how large a vector is (the magnitude of the vector). Our previous examples for vectors in R n : Manhattan Euclidean Chebyshev For Least Squares, the best norm to use will be the 2- norm.

Ordinary Least Squares Definition The least-squares best approximating function to a set of data, x, y from a class of functions, F, is the function f * in F that minimizes the 2-norm of the error. That is, if f * is the least squares best approximating function, then This is often called the ordinary least squares method of approximating data.

Linear Least Squares Linear Least Squares: We assume that the class of functions is the class of all possible lines. By the method in the last slide, we then want to find a linear function f(x) = ax + b that minimizes the 2- norm of the vector y-f(x), i.e. that minimizes:

Linear Least Squares Linear Least Squares: To minimize it is enough to minimize the term inside the square root. So, if f(x) = ax+b, we need to minimize over all possible values of a and b. From calculus, we know that the minimum will occur where the partial derivatives with respect to a and b are zero.

Linear Least Squares Linear Least Squares: These equations can be written as: These last two equations are called the Normal Equations for the best line fit to the data.

Linear Least Squares Linear Least Squares: Note that these are two equations in the unknowns a and b. Let Then, the solution is

Matrix Formulation of Linear Least Squares Linear Least Squares: Want to minimize Let Then, we want to find a vector c that minimizes the length squared of the error vector Ac-y (or y – Ac) That is, we want to minimize This is equivalent to minimizing the Euclidean distance from y to Ac.

Matrix Formulation of Linear Least Squares Linear Least Squares: Find a vector c to minimize the Euclidian distance from y to Ac. Equivalently, minimize ||y–Ac|| 2 or (y–Ac) t (y–Ac) If we take all partials of this expression (with respect to c 0, c 1 ) and set these equal to zero, we get

Matrix Formulation of Linear Least Squares The equation A t Ac = A t y is also called the Normal Equation for the linear best fit. The equations one gets from A t Ac = A t y are exactly the same equations we had before: The solution c=[a b] for A t Ac = A t y gives the constants for the line ax+b.

Matlab Example % Example of how to find the linear least squares fit to noisy data x = 1:.1:6; % x values y =.1*x + 1; % linear y-values ynoisy = y +.1*randn(size(x)); % add noise to y values plot(x, ynoisy,'.') % Plot the noisy data hold on % So we can plot more data later % Find d11, d12, d21, d22, e1, e2 D=[sum(x.^2), sum(x); sum(x), length(x)]; e1 = x*ynoisy'; e2 = sum(ynoisy); % Solve for a and b det = D(1,1)*D(2,2) - D(1,2)*D(2,1); a = (D(2,2)*e1 - D(1,2)*e2)/det; b = (D(1,1)*e2 - D(2,1)*e1)/det; % Create a vector of y-values for the linear best fit fit_y = a.*x + b; plot(x, fit_y,'-') % Plot the best fit line

Matlab Example A t Ac = A t y % Example of how to find the linear least squares fit to noisy data x = 1:.1:6; % x values y =.1*x + 1; % linear y-values ynoisy = y +.1*randn(size(x)); % add noise to y values plot(x, ynoisy,'.') % Plot the noisy data hold on % So we can more data later % Create matrix A A = zeros(length(x), 2); A(:,1) = x; A(:,2) = ones(length(x),1); D = A'*A; e = A'*ynoisy'; % Solve for constants a and b c= D \ e; fit_y = c(1).*x + c(2); plot(x, fit_y,'O')