SVD and PCA COS 323, Spring 05

SVD and PCA Principal Components Analysis (PCA): approximating a high-dimensional data set with a lower-dimensional subspacePrincipal Components Analysis (PCA): approximating a high-dimensional data set with a lower-dimensional subspace Original axes * * * * * * * *** * * *** * * * * * * * * * Data points First principal component Second principal component

SVD and PCA Data matrix with points as rows, take SVDData matrix with points as rows, take SVD – Subtract out mean (“whitening”) Columns of V k are principal componentsColumns of V k are principal components Value of w i gives importance of each componentValue of w i gives importance of each component

PCA on Faces: “Eigenfaces” Average face First principal component Other components For all except average, “gray” = 0, “white” > 0, “black” < 0

Uses of PCA Compression: each new image can be approximated by projection onto first few principal componentsCompression: each new image can be approximated by projection onto first few principal components Recognition: for a new image, project onto first few principal components, match feature vectorsRecognition: for a new image, project onto first few principal components, match feature vectors

PCA for Relighting Images under different illuminationImages under different illumination [Matusik & McMillan]

PCA for Relighting Images under different illuminationImages under different illumination Most variation captured by first 5 principal components – can re-illuminate by combining only a few imagesMost variation captured by first 5 principal components – can re-illuminate by combining only a few images [Matusik & McMillan]

PCA for DNA Microarrays Measure gene activation under different conditionsMeasure gene activation under different conditions [Troyanskaya]

PCA for DNA Microarrays Measure gene activation under different conditionsMeasure gene activation under different conditions [Troyanskaya]

PCA for DNA Microarrays PCA shows patterns of correlated activationPCA shows patterns of correlated activation – Genes with same pattern might have similar function [Wall et al.]

PCA for DNA Microarrays PCA shows patterns of correlated activationPCA shows patterns of correlated activation – Genes with same pattern might have similar function [Wall et al.]

Multidimensional Scaling In some experiments, can only measure similarity or dissimilarityIn some experiments, can only measure similarity or dissimilarity – e.g., is response to stimuli similar or different? Want to recover absolute positions in k- dimensional spaceWant to recover absolute positions in k- dimensional space

Multidimensional Scaling Example: given pairwise distances between citiesExample: given pairwise distances between cities – Want to recover locations [Pellacini et al.]

Euclidean MDS Formally, let’s say we have n  n matrix D consisting of squared distances d ij = ( x i – x j ) 2Formally, let’s say we have n  n matrix D consisting of squared distances d ij = ( x i – x j ) 2 Want to recover n  d matrix X of positions in d -dimensional spaceWant to recover n  d matrix X of positions in d -dimensional space

Euclidean MDS Observe thatObserve that Strategy: convert matrix D of d ij 2 into matrix B of x i x jStrategy: convert matrix D of d ij 2 into matrix B of x i x j – “Centered” distance matrix – B = XX T

Euclidean MDS Centering:Centering: – Sum of row i of D = sum of column i of D = – Sum of all entries in D =

Euclidean MDS Choose  x i = 0Choose  x i = 0 – Solution will have average position at origin – Then, So, to get B :So, to get B : – compute row (or column) sums – compute sum of sums – apply above formula to each entry of D – Divide by –2

Euclidean MDS Now have B, want to factor into XX TNow have B, want to factor into XX T If X is n  d, B must have rank dIf X is n  d, B must have rank d Take SVD, set all but top d singular values to 0Take SVD, set all but top d singular values to 0 – Eliminate corresponding columns of U and V – Have B 3 = U 3 W 3 V 3 T – B is square and symmetric, so U = V – Take X = U 3 times square root of W 3

Multidimensional Scaling Result ( d = 2):Result ( d = 2): [Pellacini et al.]

Multidimensional Scaling Caveat: actual axes, center not necessarily what you want (can’t recover them!)Caveat: actual axes, center not necessarily what you want (can’t recover them!) This is “classical” or “Euclidean” MDS [Torgerson 52]This is “classical” or “Euclidean” MDS [Torgerson 52] – Distance matrix assumed to be actual Euclidean distance More sophisticated versions availableMore sophisticated versions available – “Non-metric MDS”: not Euclidean distance, sometimes just relative distances – “Weighted MDS”: account for observer bias