Eigen & Singular Value Decomposition قسمتی از درس ریاضی مهندسی پیشرفته (ارشد و دکتری) 1-90-89 دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق دوشنبه، 3/9/89

Recap: Clustering 2 Hierarchical clustering Evaluation Agglomerative clustering techniques Evaluation Term vs. document space clustering Multi-lingual docs Feature selection Labeling دوشنبه، 3/9/89

Eigenvalues & Eigenvectors Eigenvectors (for a square mm matrix S) How many eigenvalues are there at most? Example (right) eigenvector eigenvalue only has a non-zero solution if this is a m-th order equation in λ which can have at most m distinct solutions (roots of the characteristic polynomial) – can be complex even though S is real. دوشنبه، 3/9/89

Matrix-vector multiplication has eigenvalues 3, 2, 0 with corresponding eigenvectors On each eigenvector, S acts as a multiple of the identity matrix: but as a different multiple on each. Any vector (say x= ) can be viewed as a combination of the eigenvectors: x = 2v1 + 4v2 + 6v3 دوشنبه، 3/9/89

Matrix vector multiplication Thus a matrix-vector multiplication such as Sx (S, x as in the previous slide) can be rewritten in terms of the eigenvalues/vectors: Even though x is an arbitrary vector, the action of S on x is determined by the eigenvalues/vectors. Suggestion: the effect of “small” eigenvalues is small. دوشنبه، 3/9/89

Eigenvalues & Eigenvectors For symmetric matrices, eigenvectors for distinct eigenvalues are orthogonal All eigenvalues of a real symmetric matrix are real. All eigenvalues of a positive semidefinite matrix are non-negative دوشنبه، 3/9/89

Plug in these values and solve for eigenvectors. Example Let Then The eigenvalues are 1 and 3 (nonnegative, real). The eigenvectors are orthogonal (and real): Real, symmetric. Plug in these values and solve for eigenvectors. دوشنبه، 3/9/89

Eigen/diagonal Decomposition Let be a square matrix with m linearly independent eigenvectors (a “non-defective” matrix) Theorem: Exists an eigen decomposition (cf. matrix diagonalization theorem) Columns of U are eigenvectors of S Diagonal elements of are eigenvalues of Unique for distinct eigen-values diagonal دوشنبه، 3/9/89

Diagonal decomposition: why/how Let U have the eigenvectors as columns: Then, SU can be written Thus SU=U, or U–1SU= And S=UU–1. دوشنبه، 3/9/89

Diagonal decomposition - example Recall The eigenvectors and form Recall UU–1 =1. Inverting, we have Then, S=UU–1 = دوشنبه، 3/9/89

Let’s divide U (and multiply U–1) by Example continued Let’s divide U (and multiply U–1) by Then, S= Q  (Q-1= QT ) Why? Stay tuned … دوشنبه، 3/9/89

Symmetric Eigen Decomposition If is a symmetric matrix: Theorem: Exists a (unique) eigen decomposition where Q is orthogonal: Q-1= QT Columns of Q are normalized eigenvectors Columns are orthogonal. (everything is real) دوشنبه، 3/9/89

Exercise Examine the symmetric eigen decomposition, if any, for each of the following matrices: دوشنبه، 3/9/89

Singular Value Decomposition دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق دوشنبه، 3/9/89

Underconstrained Least Squares What if you have fewer data points than parameters in your function? Intuitively, can’t do standard least squares Recall that solution takes the form ATAx = ATb When A has more columns than rows, ATA is singular: can’t take its inverse, etc. دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق دوشنبه، 3/9/89

Underconstrained Least Squares More subtle version: more data points than unknowns, but data poorly constrains function Example: fitting to y=ax2+bx+c دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق دوشنبه، 3/9/89

Underconstrained Least Squares Problem: if problem very close to singular, roundoff error can have a huge effect Even on “well-determined” values! Can detect this: Uncertainty proportional to covariance C = (ATA)-1 In other words, unstable if ATA has small values More precisely, care if xT(ATA)x is small for any x Idea: if part of solution unstable, set answer to 0 Avoid corrupting good parts of answer دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق دوشنبه، 3/9/89

Singular Value Decomposition (SVD) Handy mathematical technique that has application to many problems Given any mn matrix A, algorithm to find matrices U, V, and S such that A = U S VT U is mm and Orthonormal S is mn and Diagonal V is nn and Orthonormal دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق دوشنبه، 3/9/89

SVD Treat as black box: code widely available In Matlab: [U,S,V]=svd(A,0) دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق دوشنبه، 3/9/89

SVD The Si are called the singular values of A If A is singular, some of the Si will be 0 In general rank(A) = number of nonzero si SVD is mostly unique (up to permutation of singular values, or if some Si are equal) دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق دوشنبه، 3/9/89

Singular Value Decomposition دوشنبه، 3/9/89

The Singular Value Decomposition VT m x n m x m m x n n x n · = S r = the rank of A = number of linearly independent columns/rows دوشنبه، 3/9/89

The Singular Value Decomposition VT = m x n m x m m x n n x n r = the rank of A = number of linearly independent columns/rows دوشنبه، 3/9/89

SVD Properties U, V give us orthonormal bases for the subspaces of A: 1st r columns of U: Column space of A Last m - r columns of U: Left nullspace of A 1st r columns of V: Row space of A 1st n - r columns of V: Nullspace of A IMPLICATION: Rank(A) = r دوشنبه، 3/9/89

Singular Value Decomposition where u1 …ur are the r orthonormal vectors that are basis of C(A) and v1 …vr are the r orthonormal vectors that are basis of C(AT ) دوشنبه، 3/9/89

Matlab Example >> A = rand(3,5) دوشنبه، 3/9/89

Matlab Example >> [U,S,V] = svd(A) دوشنبه، 3/9/89

(m x m) AAT (n x n) ATA SVD Proof Any m x n matrix A has two symmetric covariant matrices (m x m) AAT (n x n) ATA دوشنبه، 3/9/89

Why is SVD so useful? موارد استفاده: Inverses Pseudo Inverse Eigenvalues and Eigenvectors Matrix equivalent using SVD as Similarity transform Frobenius Norm of a Matrix Matrix Liklihood دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق دوشنبه، 3/9/89

Continued موارد استفاده: Principal Components Analysis (PCA) on: Faces and Recognition Total Least Squares Constrained Optimization دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق دوشنبه، 3/9/89

SVD and Inverses Application #1: inverses A-1=(VT)-1 S-1 U-1 = V S-1 UT Using fact that inverse = transpose for orthogonal matrices Since S is diagonal, S-1 also diagonal with reciprocals of entries of S دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق دوشنبه، 3/9/89

SVD and Inverses A-1=(VT)-1 S-1 U-1 = V S-1 UT This fails when some si are 0 It’s supposed to fail – singular matrix Pseudoinverse: if si=0, set 1/si to 0 (!) “Closest” matrix to inverse Defined for all (even non-square, singular, etc.) matrices Equal to (ATA)-1AT if ATA invertible دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق دوشنبه، 3/9/89

SVD and Least Squares Solving Ax=b by least squares x=pseudoinverse(A) times b Compute pseudoinverse using SVD Lets you see if data is singular Even if not singular, ratio of max to min singular values (condition number) tells you how stable the solution will be Set 1/si to 0 if si is small (even if not exactly 0) دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق دوشنبه، 3/9/89

SVD and Eigenvectors Let A=USVT, and let xi be ith column of V Consider ATA xi: So elements of S are sqrt(eigenvalues) and columns of V are eigenvectors of ATA What we wanted for robust least squares fitting! دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق دوشنبه، 3/9/89

SVD and Matrix Similarity One common equivalent of matrix similarity in linear system of : Can be deduced using . This changes the linear system to : This means of a similarity transform for the system using SVD(A). دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق دوشنبه، 3/9/89

SVD and Matrix Norm One common definition for the norm of a matrix is the Frobenius norm: Frobenius norm can be computed from SVD So changes to a matrix can be evaluated by looking at changes to singular values دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق دوشنبه، 3/9/89

SVD and Matrix Liklihood Suppose you want to find best rank-k approximation to A Answer: set all but the largest k singular values to zero Can form compact representation by eliminating columns of U and V corresponding to zeroed si دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق دوشنبه، 3/9/89

SVD and PCA Principal Components Analysis (PCA): approximating a high-dimensional data set with a lower-dimensional subspace * Data points Second principal component First principal component Original axes دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق دوشنبه، 3/9/89

SVD and PCA Data matrix with points as rows, take SVD Subtract out mean (“whitening”) Columns of Vk are principal components Value of si gives importance of each component دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق دوشنبه، 3/9/89

Physical interpretation Consider a correlation matrix, A Error ellipse with the major axis as the larger eigenvalue and the minor axis as the smaller eigenvalue دوشنبه، 3/9/89

Physical interpretation Orthogonal directions of greatest variance in data Projections along PC1 (Principal Component) discriminate the data most along any one axis Original Variable A Original Variable B PC 1 PC 2 دوشنبه، 3/9/89

دوشنبه، 3/9/89

PCA on Faces: “Eigenfaces” First principal component Average face Other components For all except average, “gray” = 0, “white” > 0, “black” < 0 دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق دوشنبه، 3/9/89

Image Compression using SVD The image is stored as a 256X264 matrix M with entries between 0 and 1 The matrix M has rank 256 Select r X 256 as an approximation to the original M As r in increased from 1 all the way to 256 the reconstruction of M would improve i.e. approximation error would reduce Advantage To send the matrix M, need to send 256X264 = 67584 numbers To send an r = 36 approximation of M, need to send 36 + 36*256 + 36*264 = 18756 numbers 36 singular values 36 left vectors, each having 256 entries 36 right vectors, each having 264 entries Courtesy: http://www.uwlax.edu/faculty/will/svd/compression/index.html دوشنبه، 3/9/89

Using PCA for Recognition Store each person as coefficients of projection onto first few principal components Compute projections of target image, compare to database (“nearest neighbor classifier”) دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق دوشنبه، 3/9/89

Total Least Squares One final least squares application Fitting a line: vertical vs. perpendicular error دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق دوشنبه، 3/9/89

Total Least Squares Distance from point to line: where n is normal vector to line, a is a constant Minimize: دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق دوشنبه، 3/9/89

Total Least Squares First, let’s pretend we know n, solve for a Then دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق دوشنبه، 3/9/89

Total Least Squares So, let’s define and minimize دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق دوشنبه، 3/9/89

Total Least Squares Write as linear system Have An=0 Problem: lots of n are solutions, including n=0 Standard least squares will, in fact, return n=0 دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق دوشنبه، 3/9/89

Constrained Optimization Solution: constrain n to be unit length So, try to minimize |An|2 subject to |n|2=1 Expand in eigenvectors ei of ATA: where the i are eigenvalues of ATA دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق دوشنبه، 3/9/89

Constrained Optimization To minimize subject to set min = 1, all other i = 0 That is, n is eigenvector of ATA with the smallest corresponding eigenvalue دکتر رنجبر نوعی، گروه مهندسی کنترل و ابزار دقیق دوشنبه، 3/9/89

Applications of SVD in Linear Algebra Homogeneous equations, Ax = 0 Minimum-norm solution is x=0 (trivial solution) Impose a constraint, “Constrained” optimization problem Special Case If rank(A)=n-1 (m ¸ n-1, n=0) then x= vn ( is a constant) Genera Case If rank(A)=n-k (m ¸ n-k, n-k+1== n=0) then x=1vn-k+1++kvn with 21++2n=1 Has appeared before Homogeneous solution of a linear system of equations Computation of Homogrpahy using DLT Estimation of Fundamental matrix For proof: Johnson and Wichern, “Applied Multivariate Statistical Analysis”, pg 79 دوشنبه، 3/9/89