1. Confessions of a Numerical Analyst Keith Evan Schubert
Backward Thinking
Confessions of a Numerical Analyst
Keith Evan Schubert

2. Simple Problem
Consider the problem ax=b
The resulting x value is

3. Simple Problem 2
Consider the problem ax=b
The resulting x value is

4. What’s Up?
The condition number (sensitivity to perturbations) is about 400.
A condition number of 1 is perfect.
Perturbation is 0.01, so 0.01*400=4.
Components of x can vary by this much!

5. What Can We Do? Rather than solve it the standard way
X=a\b
X=(ATA)-1atb
Consider the following:
X=(ATA+i)-1atb
 =.01
Then:

6. Lucky Guess?

7. Does It Always Work? No Consider  X0
Consider si2 (si is singular value of A)
X±
Picking the wrong value can get junk

8. Skyline Consider a 1 dimensional picture Use height instead of color
Result looks like the silhouette of a city’s skyline
Have smog which blurs and softens
Don’t know exactly how much blur
Want to get sharp edges

9. Getting Garbage

10. Getting Improvement

11. Why Backward? Forward errors Backward errors
Explicitly account for each error source
(X+d1)(y+d2)=xy+(yd1+xd2+d1d2)
Backward errors
Check that my algorithm acting on data will give me a solution that is “near” to the actual system acting on a nearby set of data
I.E. My algorithm with good data should do about as well as a perfect calculation on ok data

12. Picture Please! Inherent errors in A b Perfect Calculations b*
Errors due to
algorithm
best
My Algorithm
Actual
Data (x)
Nearby
Data (x*)

13. Least Squares Usually we don’t have an invertible matrix
Need to find an estimated solution
Criterion: minimize ||ax-b||
Normal equation
ATA x = ATb
Solution
X = (ATA)-1atb

14. Backward Error Criterion: minimize ||Ax-b||/(||A|| ||x||+||b||)
Normal Equations
Solution:

15. Non Convex

16. Finding The Root

17. Look At Critical Region

18. Informal Algorithm Get (A,b) svd(A)  [u1 u2],,v U1b  b1
Use rootfinder (bisection, Newton, etc.) to get  in [-sn2,0]
vT(2- I)-1  b1  x

19. What You Get

20. Least Squares

21. Total Least Squares

22. Tikhonov

23. Backward Error

24. Original

25. Comparison

26. Final Thoughts
BE is always optimistic in that it presumes that the real system is “better”
Even with this it is “robust”
There is a perturbed version of this algorithm which can be either optimistic or pessimistic
That version is not fully proven