Singular Value Decomposition and Numerical Rank

The SVD was established for real square matrices in the 1870’s by Beltrami & Jordan for complex square matrices by Autonne for general rectangular matrices by Eckart & Young (autonne-Eckart-Young theorem) Theorem: Let. Then there exist orthogonal [unitary] matrices and such that where and with

Since, we have. Denoting by we can arrange that. Let be a corresponding set of orthonormal eigenvectors and let Then if we have where Also so that and thus Let. Then from we have Choose any such that is orthogonal. Then and so as desired.

The numbers together with are called the singular values of and they are positive square roots of the eigenvalues (which are non negative) of. The columns of are called the left singular vector of (the orthonormal eigenvectors of ) while the columns of are called the right singular vector of (the orthonormal eigenvectors of ). The matrix has singular values, the positive square roots of the eigenvalues of. The nonzero singular values of and are the same.

It is not generally a good idea to compute the singular values of by the first finding the eigenvalues of, tempting as that is. Ex: Let be a real number with (so that ) Let Then so we compute leading to the (erroneous) conclusion that the rank of is 1. If we could compute in infinite precision, we would have with and thus. The point is that by working with we have unnecessarily introduced into the computation.

It is clear from the definition that the number of nonzero singular values of determines its rank while the question is not nearly clear-cut in the context of computation on a digital computer, it is now generally acknowledged that the singular value decomposition is the only generally reliable method of determining rank numerically look at the “smallest non-zero singular value” of a matrix. Since that computed value is exact for a matrix near, it makes sense to consider the rank of all matrices in some -ball (w,r,t. the spectral norm say) around. The choice of may also be based on measurement errors incurred in estimating the coefficients of or the coefficients may be uncertain because of round off errors incurred in a previous computation to get them. The key quantity in rank determination is ‘ ’.

The smallest nonzero singular value gives a dependable measure of how far (in the sense) a matrix is from matrices of lesser rank. But alone is clearly sensitive to scale so that a better measure is. But so the important quantity is which turns out to be the reciprocal of the number, the so-called conditional number of A w.r.t. pseudo inversion. In the case when A is invertible, is usual spectral condition number w.r.t. inversion. Ref: Stewart “On the pertubation of pseudo-inverses, projections, and linear least squares problems,” SIAM Review, vol.19, pp , In solving the linear system, the condition number gives a measure of how much errors in A and/or may be magnified in the computed solution. Moreover, if, gives a measure of the “nearness” of A to singularity.

In fact measures the nearness of A to singularity in any matrix norm and for certain norms it is easy to construct explicitly a matrix E with and A+E singular. Ex: Consider the matrix is, in fact, very near singular and gets more nearly so as n increases. Adding to every element in the first column of A gives an exactly singular matrix. Rank determination, in the presence of round off error, is a highly nontrivial problem.