Linear Least-Squares Approximation Ellen and Jason

Problem the Algorithm Solves Finds a function that most clearly passes through a set of points Algorithm is used for – Summarizing data – Predicting data

Standard variable names & expressions p =  X k q =  Y k r =  X k Y k s =  X k ) 2 d = (m + 1) s - (p) 2 a = [(m + 1) r - pq] / d b =[sq – pr] / d y = ax + b

Standard Terms & Definitions l 1 Approximation:  ax k + b – y k  l 2 Approximation:  a, b  ax k + b – y k  2

Principle behind algorithm/How it was derived  (y i – ax i - b) 2 This equation gives us the error.

Principle behind algorithm/How it was derived Now we take the derivative with respect to m, and with respect to b. df/dm =  2(y i - mx i - b)(-x i ) =  -2x i y i + 2mx i 2 + 2bx i df/db =  2(Y i - mx i - b)(-1) =  -2y i + 2mx i + 2b

Principle behind algorithm/How it was derived Now we want to set the equations to 0 because we are trying to find the minimum error. On the graph you can see that the minimum is at a point where the derivative would equal 0.  -2y i + 2mx i + 2b =  -2y i ) + m  (2x i ) + b  (2)  -2x i y i + 2mx i 2 + 2bx i  (-2x i y i ) + m  (2x i 2 ) + b  (2x i )

Principle behind algorithm/How it was derived  (-2x i y i ) + m  (2x i 2 ) + b  (2x i ) = 0 m  (2x i 2 ) + b  (2x i ) =  (2x i y i ) m  (x i 2 ) + b  (x i ) =  (x i y i )  -2y i ) + m  (2x i ) + b  (2) = 0 m  (2x i ) + b  (2) =  (2y i ) m  (x i ) + b =  (y i )

Principle behind algorithm/How it was derived Am + Bb = E Cm + Db = F To get rid of variables A, C and m we multiply the first equation by C and the second equation by -A C(Am +Bb = E) -A(Cm + Db = F) => ACm + BCb = EC -ACm - ADb = -AF (BC – AD)b = EC – AF b = (EC-AF)/(BC-AD)

Principle behind algorithm/How it was derived Am + Bb = E Cm + Db = F Now, in order to get rid of b, we multiply the first equation by D and the second equation by (-B) D(Am +Bb = E) -B(Cm + Db = F) => ADm + BDb = ED -BCm - BDb = -BF (AD - BC)m = ED - BF m = (AD - BC)/(ED - BF)

Example Find the linear least-squares solution for the table of values p = = 17 q = = 25 r = (1*2) + (3*6) + (4*9) + (9*8) = 128 s = (1 2 ) + (3 2 ) + (4 2 ) + (9 2 ) = 107 d =(3+1)*107 - (17 2 ) = 139 a = [(3+1)*128- (17*25)] / 139 = 87/139 b = [(107*25) - (17*128)] / 139 = 499/139 Y = ax+b Y = (87/139)x + (499/139)

Example: Graph

Example: Error Point y - (ax +b) Point 1: [2 - (0.6259* )] 2 = Point 2: [6 - (0.6259* )] 2 = Point 3: [9 - (0.6259* )] 2 = Point 4: [8 - (0.6259* )] 2 = Error:

Advantages Least Squares RegressionLeast Absolute Deviation Regressions Not Very RobustRobust Stable SolutionUnstable Solution Always One SolutionPossibly Multiple Solutions

Disadvantages Limitations in shapes – May not be effective for data that is nonlinear. For example, a linear function would not represent the sets of points in this graph very well.