1. Eigen Decomposition
Based on the slides by Mani Thomas and book by Gilbert Strang.
Modified and extended by Longin Jan Latecki

2. Introduction Eigenvalue decomposition
Physical interpretation of eigenvalue/eigenvectors

3. A(x) = (Ax) = (x) = (x)
What are eigenvalues?
Given a matrix, A, x is the eigenvector and  is the corresponding eigenvalue if Ax = x
A must be square and the determinant of A -  I must be equal to zero
Ax - x = 0 iff (A - I) x = 0
Trivial solution is if x = 0
The non trivial solution occurs when det(A - I) = 0
Are eigenvectors unique?
If x is an eigenvector, then x is also an eigenvector and  is an eigenvalue
A(x) = (Ax) = (x) = (x)

4. Calculating the Eigenvectors/values
Expand the det(A - I) = 0 for a 2 x 2 matrix
For a 2 x 2 matrix, this is a simple quadratic equation with two solutions (maybe complex)
This “characteristic equation” can be used to solve for x

5. Eigenvalue example Consider,
The corresponding eigenvectors can be computed as
For  = 0, one possible solution is x = (2, -1)
For  = 5, one possible solution is x = (1, 2)
For more information: Demos in Linear algebra by G. Strang,

9. Eigen/diagonal Decomposition
Let be a square matrix that has m linearly independent eigenvectors (a “non-defective” matrix)
Theorem: Exists an eigen decomposition
(cf. matrix diagonalization theorem)
Columns of U are eigenvectors of S
Diagonal elements of are eigenvalues of
Unique for distinct eigen-values
diagonal

10. Diagonal decomposition: why/how
Let U have the eigenvectors as columns:
Then, SU can be written
Thus SU=U, or U–1SU=
And S=UU–1.

11. Diagonal decomposition - example
For
The eigenvectors and form
Recall
UU–1 =1.
Inverting, we have
Then, S=UU–1 =

12. Example continued  Let’s divide U (and multiply U–1) by Then, S= Q
(Q-1= QT )
Why? Stay tuned …

13. Symmetric Eigen Decomposition
If is a symmetric matrix:
Theorem: Exists a (unique) eigen decomposition
where Q is orthogonal:
Q-1= QT
Columns of Q are normalized eigenvectors
Columns are orthogonal.
(everything is real)

14. Physical interpretation
Consider a covariance matrix, A, i.e., A = 1/n S ST for some S
The ellipse with the major axis as the larger eigenvalue and the minor axis as the smaller eigenvalue

15. Physical interpretation
Original Variable A
Original Variable B
PC 1
PC 2
Orthogonal directions of greatest variance in data
Projections along PC1 (Principal Component) discriminate the data most along any one axis

16. Physical interpretation
First principal component is the direction of greatest variability (covariance) in the data
Second is the next orthogonal (uncorrelated) direction of greatest variability
So first remove all the variability along the first component, and then find the next direction of greatest variability
And so on …
Thus each eigenvectors provides the directions of data variances in decreasing order of eigenvalues
For more information: See Gram-Schmidt Orthogonalization in G. Strang’s lectures