Models for Robust Estimation and Identification Shivkumar Chandrasekaran (UCSB) Keith Evan Schubert (CSUSB)

Simple Problem Consider the problem Ax=b The resulting x value is

Simple Problem 2 Consider the problem Ax=b The resulting x value is

What’s Up? The condition number (sensitivity to perturbations) is about 400. Perturbation is 0.01, so 0.01*400=4. Components of x can vary by this much!

What Can We Do? Rather than solve it the standard way x=A\b x=(ATA)-1ATb Consider the following: x=(ATA+I)-1ATb  =.01 Then:

Lucky Guess? -1 -0.5 0.5 1 1.5 2 y x

Does It Always Work? No Consider  x0 Consider -si2 (si is singular value of A) x± Picking the wrong value can get junk

Methods with the Same Form Name Cost Function  = Least Squares ||Ax-b|| Total Least Squares ||[A b]-[C d]||F s.t.: ||Cx-d||=0 sn+1 Tikhonov ||Ax-b||2 + ||Lx||2 LTL Min Max ||Ax-b|| + h||x|| h||Ax-b||/||x|| Min Min ||Ax-b|| - h||x|| -h||Ax-b||/||x|| Backward Error 1 (||Ax-b||+h||x||)/||A|| ||x|| -||Ax-b||2/||x||2

Singular Value Decomposition For any matrix A: A=USVT UUT=UTU=I VVT=VTV=I S diagonal with s1≥s2≥... ≥sn≥

Using the SVD For rectangular Matrices Define Then b1=U1Tb b2=U2Tb z=vTx Then ||Ax-b||2 = ||Sz-b1||2 + ||b2||2 ATA+I = V(S2+I)VT

Obtaining a Secular Equation Clear Denominator Substitute x SVD of A Combine Top Block Evaluate Norms Combine Terms

Solution by Secular Equation Calculate SVD of A O(mn2+n3), usually m>>n Precalculate key quantities (b1,b2,S2) O(n2) Solve by any root finder to find  Bisection Newton’s Method O(np), p is number of iterations to solution Substitute into x=V(S2+I)-1Sb1 Overall O(mn2+n3+np) Can be sped up by “economy version” of SVD no U2 calculated, get b22 by b2=b12+b22

Why Backward? Forward errors Backward errors Explicitly account for each error source (x+d1)(y+d2)=xy+(yd1+xd2+d1d2) Backward errors Check that algorithm acting on data will give a solution that is “near” to the actual system acting on a nearby set of data I.E. Algorithm with good data should do about as well as a perfect calculation on ok data

Picture Please! Inherent Condition in A b Perfect Calculations b* Nearby Data (x*) Inherent Condition in A b* Actual Data (x) Perfect Calculations b Algorithm Errors due to algorithm best

Backward Error Criterion: minimize ||Ax-b||/(||A|| ||x||+||b||) Normal Equations Solution:

Non Convex

Finding The Root

Sketch of Proof If b1,n=0 then Else 2n2 Not in (n2, n-12) by perturbation analysis Not n-12 by Hessian condition

b1,n=0 Normal Equation Hessian Condition requires

(n2, n-12)

n-12 The determinant of the Hessian must be positive After lots of algebra the determinant of the Hessian is

Informal Algorithm Get (A,b) svd(A)  [u1 u2],,v U1b  b1 Use rootfinder (bisection, Newton, etc.) to get  in [-sn2,0] vT(2- I)-1  b1  x

What You Get

Least Squares

Total Least Squares

Tikhonov

Backward Error

Original

Comparison

Final Thoughts BE is always optimistic in that it presumes that the real system is “better” Even with this it is “robust” There is a perturbed version of this algorithm which can be either optimistic or pessimistic