1. Models for Robust Estimation and Identification
Shivkumar Chandrasekaran (UCSB)
Keith Evan Schubert (CSUSB)

2. Simple Problem
Consider the problem Ax=b
The resulting x value is

3. Simple Problem 2
Consider the problem Ax=b
The resulting x value is

4. What’s Up?
The condition number (sensitivity to perturbations) is about 400.
Perturbation is 0.01, so 0.01*400=4.
Components of x can vary by this much!

5. What Can We Do? Rather than solve it the standard way
x=A\b
x=(ATA)-1ATb
Consider the following:
x=(ATA+I)-1ATb
 =.01
Then:

6. Lucky Guess? -1
-0.5
0.5 1
1.5 2 y x

7. Does It Always Work? No Consider  x0
Consider -si2 (si is singular value of A)
x±
Picking the wrong value can get junk

8. Methods with the Same Form
Name
Cost Function
 =
Least Squares
||Ax-b||
Total Least Squares
||[A b]-[C d]||F
s.t.: ||Cx-d||=0
sn+1
Tikhonov
||Ax-b||2 + ||Lx||2
LTL
Min Max
||Ax-b|| + h||x||
h||Ax-b||/||x||
Min Min
||Ax-b|| - h||x||
-h||Ax-b||/||x||
Backward Error 1
(||Ax-b||+h||x||)/||A|| ||x||
-||Ax-b||2/||x||2

9. Singular Value Decomposition
For any matrix A:
A=USVT
UUT=UTU=I
VVT=VTV=I
S diagonal with
s1≥s2≥... ≥sn≥

10. Using the SVD For rectangular Matrices Define Then b1=U1Tb b2=U2Tb
z=vTx
Then
||Ax-b||2 = ||Sz-b1||2 + ||b2||2
ATA+I = V(S2+I)VT

11. Obtaining a Secular Equation
Clear Denominator
Substitute x
SVD of A
Combine Top Block
Evaluate Norms
Combine Terms

12. Solution by Secular Equation
Calculate SVD of A
O(mn2+n3), usually m>>n
Precalculate key quantities (b1,b2,S2)
O(n2)
Solve by any root finder to find 
Bisection
Newton’s Method
O(np), p is number of iterations to solution
Substitute into x=V(S2+I)-1Sb1
Overall O(mn2+n3+np)
Can be sped up by “economy version” of SVD
no U2 calculated, get b22 by b2=b12+b22

13. Why Backward? Forward errors Backward errors
Explicitly account for each error source
(x+d1)(y+d2)=xy+(yd1+xd2+d1d2)
Backward errors
Check that algorithm acting on data will give a solution that is “near” to the actual system acting on a nearby set of data
I.E. Algorithm with good data should do about as well as a perfect calculation on ok data

14. Picture Please! Inherent Condition in A b Perfect Calculations b*
Nearby
Data (x*)
Inherent Condition
in A b*
Actual
Data (x)
Perfect
Calculations b
Algorithm
Errors due to
algorithm
best

15. Backward Error Criterion: minimize ||Ax-b||/(||A|| ||x||+||b||)
Normal Equations
Solution:

16. Non Convex

17. Finding The Root

18. Sketch of Proof If b1,n=0 then Else 2n2
Not in (n2, n-12) by perturbation analysis
Not n-12 by Hessian condition

19. b1,n=0
Normal Equation
Hessian Condition requires

20. (n2, n-12)

21. n-12 The determinant of the Hessian must be positive
After lots of algebra the determinant of the Hessian is

22. Informal Algorithm Get (A,b) svd(A)  [u1 u2],,v U1b  b1
Use rootfinder (bisection, Newton, etc.) to get  in [-sn2,0]
vT(2- I)-1  b1  x

23. What You Get

24. Least Squares

25. Total Least Squares

26. Tikhonov

27. Backward Error

28. Original

29. Comparison

30. Final Thoughts
BE is always optimistic in that it presumes that the real system is “better”
Even with this it is “robust”
There is a perturbed version of this algorithm which can be either optimistic or pessimistic