1. Eigen & Singular Value Decomposition
قسمتی از درس ریاضی مهندسی پیشرفته (ارشد و دکتری)
دکتر رنجبر نوعی،
گروه مهندسی کنترل و ابزار دقیق
دوشنبه، 3/9/89

2. Recap: Clustering 2 Hierarchical clustering Evaluation
Agglomerative clustering techniques
Evaluation
Term vs. document space clustering
Multi-lingual docs
Feature selection
Labeling
دوشنبه، 3/9/89

3. Eigenvalues & Eigenvectors
Eigenvectors (for a square mm matrix S)
How many eigenvalues are there at most?
Example
(right) eigenvector
eigenvalue
only has a non-zero solution if
this is a m-th order equation in λ which can have at most m distinct solutions (roots of the characteristic polynomial) – can be complex even though S is real.
دوشنبه، 3/9/89

4. Matrix-vector multiplication
has eigenvalues 3, 2, 0 with
corresponding eigenvectors
On each eigenvector, S acts as a multiple of the identity
matrix: but as a different multiple on each.
Any vector (say x= ) can be viewed as a combination of
the eigenvectors: x = 2v1 + 4v2 + 6v3
دوشنبه، 3/9/89

5. Matrix vector multiplication
Thus a matrix-vector multiplication such as Sx (S, x as in the previous slide) can be rewritten in terms of the eigenvalues/vectors:
Even though x is an arbitrary vector, the action of S on x is determined by the eigenvalues/vectors.
Suggestion: the effect of “small” eigenvalues is small.
دوشنبه، 3/9/89

6. Eigenvalues & Eigenvectors
For symmetric matrices, eigenvectors for distinct
eigenvalues are orthogonal
All eigenvalues of a real symmetric matrix are real.
All eigenvalues of a positive semidefinite matrix
are non-negative
دوشنبه، 3/9/89

7. Plug in these values and solve for eigenvectors.
Example
Let
Then
The eigenvalues are 1 and 3 (nonnegative, real).
The eigenvectors are orthogonal (and real):
Real, symmetric.
Plug in these values and solve for eigenvectors.
دوشنبه، 3/9/89

8. Eigen/diagonal Decomposition
Let be a square matrix with m linearly independent eigenvectors (a “non-defective” matrix)
Theorem: Exists an eigen decomposition
(cf. matrix diagonalization theorem)
Columns of U are eigenvectors of S
Diagonal elements of are eigenvalues of
Unique for distinct eigen-values
diagonal
دوشنبه، 3/9/89

9. Diagonal decomposition: why/how
Let U have the eigenvectors as columns:
Then, SU can be written
Thus SU=U, or U–1SU=
And S=UU–1.
دوشنبه، 3/9/89

10. Diagonal decomposition - example
Recall
The eigenvectors and form
Recall
UU–1 =1.
Inverting, we have
Then, S=UU–1 =
دوشنبه، 3/9/89

11. Let’s divide U (and multiply U–1) by
Example continued
Let’s divide U (and multiply U–1) by
Then, S= Q 
(Q-1= QT )
Why? Stay tuned …
دوشنبه، 3/9/89

12. Symmetric Eigen Decomposition
If is a symmetric matrix:
Theorem: Exists a (unique) eigen decomposition
where Q is orthogonal:
Q-1= QT
Columns of Q are normalized eigenvectors
Columns are orthogonal.
(everything is real)
دوشنبه، 3/9/89

13. Exercise
Examine the symmetric eigen decomposition, if any, for each of the following matrices:
دوشنبه، 3/9/89

14. Singular Value Decomposition
دکتر رنجبر نوعی،
گروه مهندسی کنترل و ابزار دقیق
دوشنبه، 3/9/89

15. Underconstrained Least Squares
What if you have fewer data points than parameters in your function?
Intuitively, can’t do standard least squares
Recall that solution takes the form ATAx = ATb
When A has more columns than rows, ATA is singular: can’t take its inverse, etc.
دکتر رنجبر نوعی،
گروه مهندسی کنترل و ابزار دقیق
دوشنبه، 3/9/89

16. Underconstrained Least Squares
More subtle version: more data points than unknowns, but data poorly constrains function
Example: fitting to y=ax2+bx+c
دکتر رنجبر نوعی،
گروه مهندسی کنترل و ابزار دقیق
دوشنبه، 3/9/89

17. Underconstrained Least Squares
Problem: if problem very close to singular, roundoff error can have a huge effect
Even on “well-determined” values!
Can detect this:
Uncertainty proportional to covariance C = (ATA)-1
In other words, unstable if ATA has small values
More precisely, care if xT(ATA)x is small for any x
Idea: if part of solution unstable, set answer to 0
Avoid corrupting good parts of answer
دکتر رنجبر نوعی،
گروه مهندسی کنترل و ابزار دقیق
دوشنبه، 3/9/89

18. Singular Value Decomposition (SVD)
Handy mathematical technique that has application to many problems
Given any mn matrix A, algorithm to find matrices U, V, and S such that
A = U S VT
U is mm and Orthonormal
S is mn and Diagonal
V is nn and Orthonormal
دکتر رنجبر نوعی،
گروه مهندسی کنترل و ابزار دقیق
دوشنبه، 3/9/89

19. SVD
Treat as black box: code widely available In Matlab: [U,S,V]=svd(A,0)
دکتر رنجبر نوعی،
گروه مهندسی کنترل و ابزار دقیق
دوشنبه، 3/9/89

20. SVD The Si are called the singular values of A
If A is singular, some of the Si will be 0
In general rank(A) = number of nonzero si
SVD is mostly unique (up to permutation of singular values, or if some Si are equal)
دکتر رنجبر نوعی،
گروه مهندسی کنترل و ابزار دقیق
دوشنبه، 3/9/89

21. Singular Value Decomposition
دوشنبه، 3/9/89

22. The Singular Value DecompositionVT
m x n m x m m x n n x n = S
r = the rank of A
= number of linearly independent
columns/rows
دوشنبه، 3/9/89

23. The Singular Value DecompositionVT =
m x n m x m m x n n x n
r = the rank of A
= number of linearly independent
columns/rows
دوشنبه، 3/9/89

24. SVD Properties U, V give us orthonormal bases for the subspaces of A:
1st r columns of U: Column space of A
Last m - r columns of U: Left nullspace of A
1st r columns of V: Row space of A
1st n - r columns of V: Nullspace of A
IMPLICATION: Rank(A) = r
دوشنبه، 3/9/89

25. Singular Value Decomposition
where
u1 …ur are the r orthonormal vectors that are basis of C(A) and
v1 …vr are the r orthonormal vectors that are basis of C(AT )
دوشنبه، 3/9/89

26. Matlab Example
>> A = rand(3,5)
دوشنبه، 3/9/89

27. Matlab Example
>> [U,S,V] = svd(A)
دوشنبه، 3/9/89

28. (m x m) AAT (n x n) ATA SVD Proof
Any m x n matrix A has two symmetric covariant matrices
(m x m) AAT
(n x n) ATA
دوشنبه، 3/9/89

29. Why is SVD so useful? موارد استفاده:
Inverses
Pseudo Inverse
Eigenvalues and Eigenvectors
Matrix equivalent using SVD as Similarity transform
Frobenius Norm of a Matrix
Matrix Liklihood
دکتر رنجبر نوعی،
گروه مهندسی کنترل و ابزار دقیق
دوشنبه، 3/9/89

30. Continued موارد استفاده:
Principal Components Analysis (PCA) on:
Faces and
Recognition
Total Least Squares
Constrained Optimization
دکتر رنجبر نوعی،
گروه مهندسی کنترل و ابزار دقیق
دوشنبه، 3/9/89

31. SVD and Inverses Application #1: inverses
A-1=(VT)-1 S-1 U-1 = V S-1 UT
Using fact that inverse = transpose for orthogonal matrices
Since S is diagonal, S-1 also diagonal with reciprocals of entries of S
دکتر رنجبر نوعی،
گروه مهندسی کنترل و ابزار دقیق
دوشنبه، 3/9/89

32. SVD and Inverses A-1=(VT)-1 S-1 U-1 = V S-1 UT
This fails when some si are 0
It’s supposed to fail – singular matrix
Pseudoinverse: if si=0, set 1/si to 0 (!)
“Closest” matrix to inverse
Defined for all (even non-square, singular, etc.) matrices
Equal to (ATA)-1AT if ATA invertible
دکتر رنجبر نوعی،
گروه مهندسی کنترل و ابزار دقیق
دوشنبه، 3/9/89

33. SVD and Least Squares Solving Ax=b by least squares
x=pseudoinverse(A) times b
Compute pseudoinverse using SVD
Lets you see if data is singular
Even if not singular, ratio of max to min singular values (condition number) tells you how stable the solution will be
Set 1/si to 0 if si is small (even if not exactly 0)
دکتر رنجبر نوعی،
گروه مهندسی کنترل و ابزار دقیق
دوشنبه، 3/9/89

34. SVD and Eigenvectors Let A=USVT, and let xi be ith column of V
Consider ATA xi:
So elements of S are sqrt(eigenvalues) and columns of V are eigenvectors of ATA
What we wanted for robust least squares fitting!
دکتر رنجبر نوعی،
گروه مهندسی کنترل و ابزار دقیق
دوشنبه، 3/9/89

35. SVD and Matrix Similarity
One common equivalent of matrix similarity in linear system of :
Can be deduced using
This changes the linear system to :
This means of a similarity transform for the system using SVD(A).
دکتر رنجبر نوعی،
گروه مهندسی کنترل و ابزار دقیق
دوشنبه، 3/9/89

36. SVD and Matrix Norm
One common definition for the norm of a matrix is the Frobenius norm:
Frobenius norm can be computed from SVD
So changes to a matrix can be evaluated by looking at changes to singular values
دکتر رنجبر نوعی،
گروه مهندسی کنترل و ابزار دقیق
دوشنبه، 3/9/89

37. SVD and Matrix Liklihood
Suppose you want to find best rank-k approximation to A
Answer: set all but the largest k singular values to zero
Can form compact representation by eliminating columns of U and V corresponding to zeroed si
دکتر رنجبر نوعی،
گروه مهندسی کنترل و ابزار دقیق
دوشنبه، 3/9/89

38. SVD and PCA
Principal Components Analysis (PCA): approximating a high-dimensional data set with a lower-dimensional subspace *
Data points
Second principal component
First principal component
Original axes
دکتر رنجبر نوعی،
گروه مهندسی کنترل و ابزار دقیق
دوشنبه، 3/9/89

39. SVD and PCA Data matrix with points as rows, take SVD
Subtract out mean (“whitening”)
Columns of Vk are principal components
Value of si gives importance of each component
دکتر رنجبر نوعی،
گروه مهندسی کنترل و ابزار دقیق
دوشنبه، 3/9/89

40. Physical interpretation
Consider a correlation matrix, A
Error ellipse with the major axis as the larger eigenvalue and the minor axis as the smaller eigenvalue
دوشنبه، 3/9/89

41. Physical interpretation
Orthogonal directions of greatest variance in data
Projections along PC1 (Principal Component) discriminate the data most along any one axis
Original Variable A
Original Variable B
PC 1
PC 2
دوشنبه، 3/9/89

42. دوشنبه، 3/9/89

43. PCA on Faces: “Eigenfaces”
First principal component
Average face
Other components
For all except average, “gray” = 0, “white” > 0,
“black” < 0
دکتر رنجبر نوعی،
گروه مهندسی کنترل و ابزار دقیق
دوشنبه، 3/9/89

44. Image Compression using SVD
The image is stored as a 256X264 matrix M with entries between 0 and 1
The matrix M has rank 256
Select r X 256 as an approximation to the original M
As r in increased from 1 all the way to 256 the reconstruction of M would improve i.e. approximation error would reduce
Advantage
To send the matrix M, need to send 256X264 = numbers
To send an r = 36 approximation of M, need to send * *264 = numbers
36 singular values
36 left vectors, each having 256 entries
36 right vectors, each having 264 entries
Courtesy:
دوشنبه، 3/9/89

45. Using PCA for Recognition
Store each person as coefficients of projection onto first few principal components
Compute projections of target image, compare to database (“nearest neighbor classifier”)
دکتر رنجبر نوعی،
گروه مهندسی کنترل و ابزار دقیق
دوشنبه، 3/9/89

46. Total Least Squares One final least squares application
Fitting a line: vertical vs. perpendicular error
دکتر رنجبر نوعی،
گروه مهندسی کنترل و ابزار دقیق
دکتر رنجبر نوعی،
گروه مهندسی کنترل و ابزار دقیق
دوشنبه، 3/9/89

47. Total Least Squares
Distance from point to line: where n is normal vector to line, a is a constant
Minimize:
دکتر رنجبر نوعی،
گروه مهندسی کنترل و ابزار دقیق
دوشنبه، 3/9/89

48. Total Least Squares First, let’s pretend we know n, solve for a Then
دکتر رنجبر نوعی،
گروه مهندسی کنترل و ابزار دقیق
دوشنبه، 3/9/89

49. Total Least Squares So, let’s define and minimize دکتر رنجبر نوعی،
گروه مهندسی کنترل و ابزار دقیق
دوشنبه، 3/9/89

50. Total Least Squares Write as linear system Have An=0
Problem: lots of n are solutions, including n=0
Standard least squares will, in fact, return n=0
دکتر رنجبر نوعی،
گروه مهندسی کنترل و ابزار دقیق
دوشنبه، 3/9/89

51. Constrained Optimization
Solution: constrain n to be unit length
So, try to minimize |An|2 subject to |n|2=1
Expand in eigenvectors ei of ATA: where the i are eigenvalues of ATA
دکتر رنجبر نوعی،
گروه مهندسی کنترل و ابزار دقیق
دوشنبه، 3/9/89

52. Constrained Optimization
To minimize subject to set min = 1, all other i = 0
That is, n is eigenvector of ATA with the smallest corresponding eigenvalue
دکتر رنجبر نوعی،
گروه مهندسی کنترل و ابزار دقیق
دوشنبه، 3/9/89

53. Applications of SVD in Linear Algebra
Homogeneous equations, Ax = 0
Minimum-norm solution is x=0 (trivial solution)
Impose a constraint,
“Constrained” optimization problem
Special Case
If rank(A)=n-1 (m ¸ n-1, n=0) then x= vn ( is a constant)
Genera Case
If rank(A)=n-k (m ¸ n-k, n-k+1== n=0) then x=1vn-k+1++kvn with 21++2n=1
Has appeared before
Homogeneous solution of a linear system of equations
Computation of Homogrpahy using DLT
Estimation of Fundamental matrix
For proof: Johnson and Wichern, “Applied Multivariate Statistical Analysis”, pg 79
دوشنبه، 3/9/89