Download presentation
Presentation is loading. Please wait.
1
SVD and PCA COS 323
2
Dimensionality Reduction Map points in high-dimensional space to lower number of dimensionsMap points in high-dimensional space to lower number of dimensions Preserve structure: pairwise distances, etc.Preserve structure: pairwise distances, etc. Useful for further processing:Useful for further processing: – Less computation, fewer parameters – Easier to understand, visualize
3
PCA Principal Components Analysis (PCA): approximating a high-dimensional data set with a lower-dimensional linear subspacePrincipal Components Analysis (PCA): approximating a high-dimensional data set with a lower-dimensional linear subspace Original axes * * * * * * * *** * * *** * * * * * * * * * Data points First principal component Second principal component
4
SVD and PCA Data matrix with points as rows, take SVDData matrix with points as rows, take SVD – Subtract out mean (“whitening”) Columns of V k are principal componentsColumns of V k are principal components Value of w i gives importance of each componentValue of w i gives importance of each component
5
PCA on Faces: “Eigenfaces” Average face First principal component Other components For all except average, “gray” = 0, “white” > 0, “black” < 0
6
Uses of PCA Compression: each new image can be approximated by projection onto first few principal componentsCompression: each new image can be approximated by projection onto first few principal components Recognition: for a new image, project onto first few principal components, match feature vectorsRecognition: for a new image, project onto first few principal components, match feature vectors
7
PCA for Relighting Images under different illuminationImages under different illumination [Matusik & McMillan]
8
PCA for Relighting Images under different illuminationImages under different illumination Most variation captured by first 5 principal components – can re-illuminate by combining only a few imagesMost variation captured by first 5 principal components – can re-illuminate by combining only a few images [Matusik & McMillan]
9
PCA for DNA Microarrays Measure gene activation under different conditionsMeasure gene activation under different conditions [Troyanskaya]
10
PCA for DNA Microarrays Measure gene activation under different conditionsMeasure gene activation under different conditions [Troyanskaya]
11
PCA for DNA Microarrays PCA shows patterns of correlated activationPCA shows patterns of correlated activation – Genes with same pattern might have similar function [Wall et al.]
12
PCA for DNA Microarrays PCA shows patterns of correlated activationPCA shows patterns of correlated activation – Genes with same pattern might have similar function [Wall et al.]
13
Multidimensional Scaling In some experiments, can only measure similarity or dissimilarityIn some experiments, can only measure similarity or dissimilarity – e.g., is response to stimuli similar or different? – Frequent in psychophysical experiments, preference surveys, etc. Want to recover absolute positions in k-dimensional spaceWant to recover absolute positions in k-dimensional space
14
Multidimensional Scaling Example: given pairwise distances between citiesExample: given pairwise distances between cities – Want to recover locations [Pellacini et al.]
15
Euclidean MDS Formally, let’s say we have n n matrix D consisting of squared distances d ij = ( x i – x j ) 2Formally, let’s say we have n n matrix D consisting of squared distances d ij = ( x i – x j ) 2 Want to recover n d matrix X of positions in d -dimensional spaceWant to recover n d matrix X of positions in d -dimensional space
16
Euclidean MDS Observe thatObserve that Strategy: convert matrix D of d ij 2 into matrix B of x i x jStrategy: convert matrix D of d ij 2 into matrix B of x i x j – “Centered” distance matrix – B = XX T
17
Euclidean MDS Centering:Centering: – Sum of row i of D = sum of column i of D = – Sum of all entries in D =
18
Euclidean MDS Choose x i = 0Choose x i = 0 – Solution will have average position at origin – Then, So, to get B :So, to get B : – compute row (or column) sums – compute sum of sums – apply above formula to each entry of D – Divide by –2
19
Euclidean MDS Now have B, want to factor into XX TNow have B, want to factor into XX T If X is n d, B must have rank dIf X is n d, B must have rank d Take SVD, set all but top d singular values to 0Take SVD, set all but top d singular values to 0 – Eliminate corresponding columns of U and V – Have B 3 = U 3 W 3 V 3 T – B is square and symmetric, so U = V – Take X = U 3 times square root of W 3
20
Multidimensional Scaling Result ( d = 2):Result ( d = 2): [Pellacini et al.]
21
Multidimensional Scaling Caveat: actual axes, center not necessarily what you want (can’t recover them!)Caveat: actual axes, center not necessarily what you want (can’t recover them!) This is “classical” or “Euclidean” MDS [Torgerson 52]This is “classical” or “Euclidean” MDS [Torgerson 52] – Distance matrix assumed to be actual Euclidean distance More sophisticated versions availableMore sophisticated versions available – “Non-metric MDS”: not Euclidean distance, sometimes just inequalities – “Weighted MDS”: account for observer bias
22
Computation SVD very closely related to eigenvalue/vector computationSVD very closely related to eigenvalue/vector computation – Eigenvectors/values of A T A – In practice, similar class of methods, but operate on A directly
23
Methods for Eigenvalue Computation Simplest: power methodSimplest: power method – Begin with arbitrary vector x 0 – Compute x i+1 =Ax i – Normalize – Iterate Converges to eigenvector with maximum eigenvalue!Converges to eigenvector with maximum eigenvalue!
24
Power Method As this is repeated, coefficient of e 1 approaches 1As this is repeated, coefficient of e 1 approaches 1
25
Power Method II To find smallest eigenvalue, similar process:To find smallest eigenvalue, similar process: – Begin with arbitrary vector x 0 – Solve Ax i+1 = x i – Normalize – Iterate
26
Deflation Once we have found an eigenvector e 1 with eigenvalue 1, can compute matrix A – 1 e 1 e 1 TOnce we have found an eigenvector e 1 with eigenvalue 1, can compute matrix A – 1 e 1 e 1 T This makes eigenvalue of e 1 equal to 0, but has no effect on other eigenvectors/valuesThis makes eigenvalue of e 1 equal to 0, but has no effect on other eigenvectors/values In principle, could find all eigenvectors this wayIn principle, could find all eigenvectors this way
27
Other Eigenvector Computation Methods Power method OK for a few eigenvalues, but slow and sensitive to roundoff errorPower method OK for a few eigenvalues, but slow and sensitive to roundoff error Modern methods for eigendecomposition/SVD use sequence of similarity transformations to reduce to diagonal, then read off eigenvaluesModern methods for eigendecomposition/SVD use sequence of similarity transformations to reduce to diagonal, then read off eigenvalues
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.