ELG5377 Adaptive Signal Processing Lecture 13: Method of Least Squares

Introduction Given a sequence of observations x(1), x(2), …, x(N) which occur at time t 1, t 2, …, t N. –The requirement is to construct a curve that is used to fit these points in some optimum fashion. –Let us denote this curve as f(t i ). –The objective is to minimize the squares of the differences between f(t i ) and x(i). J = Σ(f(t i )-x(i)) 2. The method of least squares can be viewed as an alternative to Wiener filters –Wiener filters are based on ensemble averages –MLS is deterministic in approach and is based on time averages.

Statement of the Linear Least-Squares Estimation Problem Consider a physical phenomena that is characterized by two sets of variables –d(i) and x(i). The variable d(i) is observed at time t i in response to the subset of variables x(i), x(i-1), … x(i-M+1). The response d(i) is modelled by –where w ok are unknown parameters of the model and e o (i) represents a measurement error. –The measurement error is an unobservable random variable that is introduced to the model to account for its inaccuracy. (1)

Statement of the Linear Least-Squares Estimation Problem 2 It is customary to assume that the measurement error is white with 0 mean and variance  2. The implication of this assumption is that –where the values of x(i), x(i-1), …, x(i-M+1) are all known. –Hence the mean of the response d(i) is uniquely determined by the model.

Statement of the Linear Least-Squares Estimation Problem 3 The problem we have to solve is to estimate the unknown parameters w ok of the multiple linear regression model of (1) given the two observable sets of variables x(i) and d(i) for i= 1, 2, …, N. To do this we use the linear transversal filter shown below as the model of interest, whose output is y(i) and we use d(i) as the desired response. … w0*w0* w1*w1* w M-1 * + + y(i)y(i) d(i)d(i) e(i)e(i) - x(i)x(i) x(i-1)x(i-M+1)

Statement of the Linear Least-Squares Estimation Problem 4 Must chose tap weights so as to minimize this cost function

Data Windowing Typically, we are given data for i= 1 to N, where N > M. It is only at time M, where d(i) is a function of known data. Also, for i > N, d(i) has unknown data in its equation. Covariance method –No assumptions on unknown data, therefore i 1 = M and i 2 = N. Autocorrelation method –i 1 = 1 and i 2 = N+M-1. We assume that x(i) = 0 for i N. Prewindowing and postwindowing. We will use covariance method.

Orthogonality Principle Revisited (2)

Orthogonality Principle Revisited 2 Let e(i) = e min (i) if we select w 0, w 1, …, w M-1 such that J is minimized. Then from (2) –The minimum-error time series is orthogonal to the time series x(i-k) applied to tap k of a transversal filter of length M for k = 1, 2, …,M-1 when the filter is operating in its least-squares condition. Let y min (i) be the output of the filter when it is operating in its least squares condition. d(i) = y min (i)+e min (i).

Energy of Desired

Normal Equations and Linear Least-Squares Filters Let the filter described by the tap weights, w 0, w 1, …, w M-1 be operating in its least-squares condition. –Therefore k = 0, 1, …, M-1

Normal Equations and Linear Least-Squares Filters 2

Normal Equations and Linear Least-Squares Filters: Matrix formulation

Example x(1)=2, x(2) = 1, x(3) = -0.5, x(4) = 1.2 d(2) = 0.5, d(3) = 1, d(4) = 0 Find the two-tap LS filter.