1. Outline Numerical Stability Singular Value Decomposition
Properties on the SVD

2. Numerical Stability, Condition Number
Smaller Perturbation of Coefficients May Cause Significant Solution Variation
Original Function Ax = b
Perturbation of A and b
Perturbation of Vector b
b  b + ∆b, thus solution x  x + ∆x
A(x + ∆x) = b + ∆b, ∆x = A-1∆b
Upper bound: ||∆x||2 ≤ ||A-1||2 ||∆b||2

3. Numerical Stability, Condition Number
Perturbation of Matrix A
A  A+∆A, thus solution x  x+∆x
(A+∆A)(x+∆x) = b, Ax = b and thus
||∆x||2 ≤ ||A-1||2||∆A||2||x + ∆x||2
||∆x||2/||x + ∆x||2 ≤ ||A||2||A-1||2(||∆A||2/||A||2)
Condition Number: ||A||2||A-1||2
Critically Depends on the Norm ||A-1||2

4. Numerical Stability, Condition Number
Condition Number of a 2*2 Matrix
Proportional to det(A-1)
A-1 = adj(A) * det(A-1)
Question: Numerical Stability Could Be Improved via Least Square Method?
Condition Number for the LS solution
cond(AHA) could be very large

5. Singular Value Decomposition
Further Characterize ||A||2||A-1||2
A = UΣVH, U and V unitary matrices,
Σ diagonal matrix with non-negative elements
Equivalent form for the SVD
A = UΣVH  AV = UΣ
A = UΣVH  UHA = ΣVH
Eigenvalue decomposition for AAH
AAH = UΣΣHUH = UΣ2UH

6. SVD Properties
A = UΣVH, U and V unitary matrices, m*n matrix A, rank(A)=r
the first r rows/columns of Σ are not zero
First r columns of U, ortho-normal basis for the column space of A
First r rows of V, ortho-normal basis for the row space of A
Last m-r columns of V, ortho-normal basis for the zero column space of A
Last n-r columns of U, ortho-normal basis for the zero row space of A

7. SVD Properties
First r columns of U, ortho-normal basis for the column space of A
First r rows of VH, ortho-normal basis for the row space of A
Last m-r columns of U, ortho-normal basis for the zero column space of A
Last n-r rows of VH, ortho-normal basis for the zero row space of A

8. SVD Properties MP inverse based on the SVD A = UΣVH, A+ = VΣ+UH
Σ+: Pseudo Inverse of Σ

9. SVD Properties Spectral of Matrix A
||A||spec = maximum singular value of A
F-Norm of A: ||A||F = ||Σ||F
Orthonormal matrix does not change 2-Norm
Determinant of Matrix A:
|det(A)| = |det(U)||det(Σ)||det(VH)| = |det(Σ)|
cond(A) = maximum singular value / minimum singular value

10. SVD Properties
cond(A) = maximum singular value / minimum singular value
||A||2 = maximum singular value
||A-1||2 = inverse of minimum singular value
Minimum singular value
min (xHAHAx), s.t. xHx = 1
Maximum singular value
max (xHAHAx), s.t. xHx = 1

11. SVD Properties Low Rank Matrix approximation
Norm and spectrum
Signal and data processing
Minimum Spectral and F-Norm under Rank-k Matrix

12. SVD Properties Inequalities on the Singular Values
Rank the Singular Values of Matrices A and B
Consider the Singular Values of Matrix A+B

13. SVD Properties Ky-Fan Norm: the largest k singular values of matrix
Special Cases of Ky-Fan Norm
k = 1, the largest singular value
k = N, the trace norm
Ky-Fan Norm Inequalities