1. Principal Components
What matters most?

2. DRAFT: Copyright GA Tagliarini, PhD
Basic Statistics
Assume x1, x2,…, xn represent a distribution x of samples of a random variable.
The expected value or mean of the distribution x, written E{x} or m, is given by E{x} = m = (x1+ x2+,…,+ xn)/n
The population variance E{(x-m)2} = σ2, the mean of the squared deviations from the mean, is given by σ2=[(x1-m)2+(x2-m)2+…+(xn-m)2]/n
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

3. Generalization to Vectors
Assume d-dimensional vectors xi=<x1,…,xd>T whose dimensions sample random distributions
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

4. DRAFT: Copyright GA Tagliarini, PhD
Interpretation
The mean vector is d-dimensional
The covariance matrix:
d x d
Symmetric
Real valued
Main diagonal entries are the variances of the d dimensions
The off-diagonal entry in row i column j is the covariance in dimensions i and j
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

5. Example: Finding a Covariance Matrix
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

6. Example: Finding a Covariance Matrix
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

7. Example: Finding a Covariance Matrix
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

8. Example: Finding a Covariance Matrix
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

9. Example: Finding a Covariance Matrix
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

10. Example: Finding a Covariance Matrix
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

11. Eigenvectors and Eigenvalues
Suppose A is an d x d matrix, λ is a scalar, and e ≠ 0 is an d-dimensional vector.
If Ae = λe, then e is an eigenvector of A corresponding to the eigenvalue λ.
If A is real and symmetric, one can always find d, unit length, orthogonal (orthonormal) eigenvectors for A
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

12. Characteristic Polynomial
Since eigenvalues and the corresponding eigenvectors satisfy Ae = λe, one can find an eigenvector corresponding to λ by solving the homogeneous linear system (λI-A)e = 0
To find λ, construct the characteristic polynomial p(λ) = det(λI-A), where det(.) is the determinant operator
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

13. Example: Finding Eigenvalues
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

14. Example: Finding Eigenvalues
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

15. Example: Finding Eigenvalues
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

16. Finding Eigenvectors For The Eigenvalues: λ1 and e1
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

17. Finding Eigenvectors For The Eigenvalues: λ1 and e1
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

18. Finding Eigenvectors For The Eigenvalues: λ1 and e1
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

19. Finding Eigenvectors For The Eigenvalues: λ2 and e2
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

20. Finding Eigenvectors For The Eigenvalues: λ2 and e2
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

21. Finding Eigenvectors For The Eigenvalues: λ3 and e3
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

22. Finding Eigenvectors For The Eigenvalues: λ3 and e3
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

23. Assumptions and Goals of PCA
Given a set of vectors xi, i=1,…,n
Goal: Find a unit vector u such that the variance after projecting onto the line through u is maximal
Assume that the xi have zero mean,
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

24. Pictorially-Principal Axis
Raw data
Data with principal axis z z y y x x
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

25. Pictorially-Second Axis
Raw data
Data with second principal axis z z y y x x
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

26. Pictorially-Third Axis
Raw data
Data with third principal axis z z y y x x
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

27. DRAFT: Copyright GA Tagliarini, PhD
All Three Axes
Raw data
Data with three new axis
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

28. Recall from Vector Projection
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

29. DRAFT: Copyright GA Tagliarini, PhD
Setting Things Up
There is some abuse of notation in the last two expressions, which represent matrix operations having 1x1 results.
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

30. The Scalars Have Zero Mean
Notice:
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

31. What is the Variance of the Projection Scalars?
Where R is the covariance matrix of the xi.
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

32. Variance as a Function of u
Variance is commonly denoted σ2
Variance of the scalars depends on u and is given by σ2(u) = uT R u
Using Lagrange multipliers one may maximize variance by seeking extrema for J = uT R u + λ (1 –uT u)
Differentiating we obtain deluJ = 2 (R u - λ u)
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

33. Finding Extreme Variance
Setting R u - λ u = 0
Or R u = λ u
So what???
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

34. Finding Extreme Variance
Setting R u - λ u = 0
Or R u = λ u
The variance (potential capacity for discrimination) is maximized when u is an eigenvector of the covariance matrix.
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

35. DRAFT: Copyright GA Tagliarini, PhD
Which Eigenvector??
Since σ2(u) = uT R u
And σ2(u) = uT R u = uT (λ u) = λ (uTu) = λ
Maximum variance (capacity for discrimination)
Equals the largest eigenvalue
Occurs when u is the corresponding eigenvector
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

36. DRAFT: Copyright GA Tagliarini, PhD
Notes
The covariance matrix is symmetric
Its eigenvectors are orthogonal
The unit eigenvectors form an orthonormal basis
One may order the eigenvectors ui according to the magnitude of the corresponding eigenvalues λ1 ≥ λ2≥ … ≥ λi≥ … ≥ λd
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

37. Reducing Dimensionality
Since the uj, j=1,…,d, form a basis, any x can be written as
Or estimated for some m < d, with error e as
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

38. Accounting for Variability
Since the u’s are orthogonal, and e are likewise
Hence, x e
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

39. DRAFT: Copyright GA Tagliarini, PhD
Using PCA
Find the mean of the distribution
Find the covariance matrix
Find the eigenvalues and eigenvectors
Reduce data dimensionality by projecting onto a sub-space of the original determined by using only the eigenvectors with largest eigenvalues
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

40. A Similarity Transform
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD

41. DRAFT: Copyright GA Tagliarini, PhD
Hotelling Transform
Form matrix A using the eigenvectors of Cx
Order the eigenvalues from largest to smallest l1≥l2≥...≥ld and the corresponding eigenvectors e1, e2,…, ed
Enter eigenvectors as the rows in A
y = A(x-mx)
my = E{y} = 0
Cy = A Cx AT
2/28/2019
DRAFT: Copyright GA Tagliarini, PhD