1. Backward Error Estimation
CSUSB
IAS
Shivkumar Chandrasekaran (UCSB)
Ernesto Gomez (CSUSB)
Yasha Karant (CSUSB)
Keith Evan Schubert (CSUSB)
The support of NSF under award
is gratefully acknowledged

2. 3 Decimal Digits
Consider the problem, b=Ax
The resulting x value is

3. 2 Decimal Digits
Reconsider the problem b=Ax
The resulting x value is

4. What’s Up?
Sensitivity to perturbations
Components of x can vary by:

5. What Can We Do? Rather than solve it the standard way
x=A\b
x=(ATA)-1ATb
Consider the following:
x=(ATA+I)-1ATb
 =.01
Then:

6. Lucky Guess? -1
-0.5
0.5 1
1.5 2 y x

7. Does It Always Work? No Consider , x=(ATA+I)-1ATb x0
Consider -si2, x=(ATA+I)-1ATb
x±
(si is singular value of A)
Picking the wrong value can get junk

8. Methods with the Same Form
Name
Cost Function
 =
Least Squares
||Ax-b||
Total Least Squares
||[A b]-[C d]||F
s.t.: ||Cx-d||=0
-s2n+1
Tikhonov
||Ax-b||2 + ||Lx||2
LTL
Min Max
||Ax-b|| + h||x||
h||Ax-b||/||x||
Min Min
||Ax-b|| - h||x||
-h||Ax-b||/||x||
Backward Error 3
(||Ax-b||+h||x||)/||A|| ||x||
-||Ax-b||2/||x||2

9. Why Backward? Forward errors Backward errors
Explicitly account for each error source
(x+d1)(y+d2)=xy+(yd1+xd2+d1d2)
Backward errors
Check that algorithm acting on data will give a solution that is “near” to the actual system acting on a nearby set of data
I.E. Algorithm with good data should do about as well as a perfect calculation on ok data

10. Picture Please! Inherent Condition in A b Perfect Calculations b*
Nearby
Data (x*)
Inherent Condition
in A b*
Actual
Data (x)
Perfect
Calculations b
Algorithm
Errors due to
algorithm
best

11. Backward Error
Criterion
Maximize

12. Non Convex
Cost  zn

13. Backward Error
Normal Equations
Hessian
Solution Lies in

14. Backward Error
Solution
Secular Equation
Solution Lies in

15. Finding The Root
g() 

16. Singular Value Decomposition
For any matrix A:
A=USVT
U,V are Orthogonal
UUT=UTU=I
VVT=VTV=I
S diagonal with
s1≥s2≥... ≥sn≥0

17. Using the SVD For rectangular Matrices Define Then b1=U1Tb b2=U2Tb
z=vTx
Then
||Ax-b||2 = ||Sz-b1||2 + ||b2||2
ATA+I = V(S2+I)VT

18. Solution by Secular Equation
Calculate SVD of A
O(mn2+n3), usually m>>n
Precalculate key quantities (b1,b2,S2)
O(n2)
Solve by any root finder to find 
Bisection
Newton’s Method
O(np), p is number of iterations to solution
Substitute into x=V(S2+I)-1Sb1
Overall O(mn2+n3+np)
Can be sped up by “economy version” of SVD
no U2 calculated, get b22 by b2=b12+b22

19. Informal Algorithm Get (A,b) svd(A)  [u1 u2],,v U1b  b1
Use rootfinder (bisection, Newton, etc.) to get  in [-sn2,0]
vT(2- I)-1  b1  x

20. What You Get

21. Least Squares

22. Total Least Squares

23. Tikhonov

24. Backward Error

25. Original

26. Comparison

27. 1-D Images

28. Final Thoughts BE can be optimistic or pessimistic Robust
Applications with uncertainty
Image debluring
Image separation
GPR, seismology
Medical imaging
System ID