Three Algorithms for Nonlinear Dimensionality Reduction Haixuan Yang Group Meeting Jan. 011, 2005

2 Outline Problem Problem PCA (Principal Component Analysis) PCA (Principal Component Analysis) MDS (Multidimentional Scaling) MDS (Multidimentional Scaling) Isomap (isometric mapping) Isomap (isometric mapping) –A Global Geometric Framework for Nonlinear Dimensionality Reduction. Science, 292(22), , LLE (locally linear embedding) LLE (locally linear embedding) –Nonlinear Dimensionality Reduction by Locally Linear Embedding. Science, 292(22), , Eigenmap Eigenmap –Laplacian Eigenmaps and Spectral Techniques for Embedding and Clustering. NIPS01.

3 Problem Given a set x 1, …, x k of k points in R l, find a set of Given a set x 1, …, x k of k points in R l, find a set of points y 1, …, y k in R m (m << l) such that y i “represents” x i as accurately as possible. points y 1, …, y k in R m (m << l) such that y i “represents” x i as accurately as possible. If the data x i is placed in a super plane in high dimensional space, the traditional algorithms, such as PCA and MDS, work well. If the data x i is placed in a super plane in high dimensional space, the traditional algorithms, such as PCA and MDS, work well. However, when the data x i is placed in a nonlinear manifold in high dimensional space, then the linear algebra technique can not work any more. However, when the data x i is placed in a nonlinear manifold in high dimensional space, then the linear algebra technique can not work any more. –A nonlinear manifold can be roughly understood as a distorted super plane, which may be twisted, folded, or curved.

4 PCA (Principal Component Analysis) Reduce dimensionality of data by transforming correlated variables (bands) into a smaller number of uncorrelated components Reveals meaningful latent information Best preserves the variance as measured in the high-dimensional input space Nonlinear structure is invisible to PCA

5 First, a graphical look at the problem … Band 1 Band2Band2 Two (correlated) Bands of data

6 Band 1 Band2Band2 Regression Line Summarizes the Two Bands

7 Band 1 Band2Band2 Rotate axes to create two orthogonal (uncorrelated) components PC1 PC2 “Reflected” X- and y-axes

8 Band 1 Band2Band2 Partitioning of Variance PC1 PC2 Var(PC1) Var(PC2)

9 PCA: algorithm description Step 1: Calculate the average x of x i. Step 1: Calculate the average x of x i. Step 2: Estimate the Covariance Matrix by Step 2: Estimate the Covariance Matrix by Step 3: Let λ p be the p-th eigenvalue (in decreasing order) of the matrix M, and v p i be the i-th component of the p-th eignvector. Then set the p- th componet of the d-dimentional coordinate vector y i equal to Step 3: Let λ p be the p-th eigenvalue (in decreasing order) of the matrix M, and v p i be the i-th component of the p-th eignvector. Then set the p- th componet of the d-dimentional coordinate vector y i equal to

10 MDS Step 1: Given the distance d(i, j) between i and j. Step 1: Given the distance d(i, j) between i and j. Step 2: From d(i, j), get the covariance matrix M by Step 2: From d(i, j), get the covariance matrix M by Step3: The same as PCA Step3: The same as PCA

11 An example of embedding of a two dimentional manifold into a three dimentional space Not the true distance The true distance

12 Isomap: basic idea Learn the global distance by the local distance. Learn the global distance by the local distance. The local distance calculated by the Euclidean distance is relatively accurate because a patch in the nonlinear manifold looks like a plane when it is small, and therefore the direct Euclidean distance approximates the true distance in this small patch. The local distance calculated by the Euclidean distance is relatively accurate because a patch in the nonlinear manifold looks like a plane when it is small, and therefore the direct Euclidean distance approximates the true distance in this small patch. The global distance calculated by the Euclidean distance is not accurate because the manifold is curved. The global distance calculated by the Euclidean distance is not accurate because the manifold is curved. Best preserve the estimated distance in the embedded space in the same way as MDS. Best preserve the estimated distance in the embedded space in the same way as MDS.

13 Isomap: algorithm description Step 1: Construct neighborhood graph Define the graph over all data points by connecting points i and j if they are closer than ε (ε-Isomap), or if i is one of the n nearest neighbors of j (k-Isomap). Set edge lengths equal to d X (i,j). Define the graph over all data points by connecting points i and j if they are closer than ε (ε-Isomap), or if i is one of the n nearest neighbors of j (k-Isomap). Set edge lengths equal to d X (i,j). Step 2: Compute shortest paths Initialize d G (i,j)= d X (i,j) if i and j are linked by an edge; d G (i,j)= ∞ Initialize d G (i,j)= d X (i,j) if i and j are linked by an edge; d G (i,j)= ∞ otherwise. Then compute the shortest path distances d G (i,j) between all otherwise. Then compute the shortest path distances d G (i,j) between all pairs of points in weighted graph G. Let D G =( d G (i,j) ). pairs of points in weighted graph G. Let D G =( d G (i,j) ). Step 3: Construct d-dimensional embedding Let λ p be the p-th eigenvalue (in decreasing order) of the matrix τ( D G ), and v p i be the i-th component of the p-th eignvector. Then set the p-th componet of the d-dimentional coordinate vector y i equal to. Let λ p be the p-th eigenvalue (in decreasing order) of the matrix τ( D G ), and v p i be the i-th component of the p-th eignvector. Then set the p-th componet of the d-dimentional coordinate vector y i equal to.

14 An example: each picture, a 4096 (64*64)-dimensional point, can be mapped into 2-dinesional plane

15 Another example: the 3-dimentional points are maped into 2-dimentional plane

16 LLE: basic idea Learn the local linear relation by the local data Learn the local linear relation by the local data The local data is relatively linear because a patch in the nonlinear manifold looks like a plane when it is small. The local data is relatively linear because a patch in the nonlinear manifold looks like a plane when it is small. Globally the data is not linear because the manifold is curved. Globally the data is not linear because the manifold is curved. Best preserve the local linear relation in the embedded space in the similar way as PCA. Best preserve the local linear relation in the embedded space in the similar way as PCA.

17 LLE: algorithm description Step 1: Discovering the Adjacency Information For each x i find its n nearest neighbors,. For each x i find its n nearest neighbors,. Step 2: Constrcting the Approximation Matrix Choose W ij by minimizing Choose W ij by minimizing Under the condition that Under the condition that Step 3: Compute the Embedding The embedding vectors y i can be found by minimizing The embedding vectors y i can be found by minimizing

18 An example: 4096-dimentional face pictures are embedded into a 2-dimentional plane

19 Eigenmap: Basic Idea Use the local information to decide the embedded data. Use the local information to decide the embedded data. Motivated by the way that heat transmits from one point to another point. Motivated by the way that heat transmits from one point to another point.

20 Eigenmap Step 1: Construct neighborhood graph The same as Isomap. The same as Isomap. Step 2: Compute the weights of the graph If node i and node j are connected, put If node i and node j are connected, put Step 3: Construct d-dimensional embedding Compute the eigenvalues and eigenvectors for the generalized eigenvector problem:, where D is a diagonal matrix, and Compute the eigenvalues and eigenvectors for the generalized eigenvector problem:, where D is a diagonal matrix, and

21 Cont. Let f 0,…,f k-1 be the solutions of the above equation, ordered increasingly according to their eignvalues, Lf 0 =λ 0 Df 0 Lf 1 =λ 1 Df 1 … Lf k-1 =λ k-1 Df k-1 Then y i is determined by the ith component of the d eigenvectors f 1,…,f d.

22 An example: 256-dimentional speech data is represented in a 2-dimentional plane

23 Conclusion Isomap, LLE and Eigenmap can find the meaningful low-dimensional structure hidden in the high-dimensional observation. Isomap, LLE and Eigenmap can find the meaningful low-dimensional structure hidden in the high-dimensional observation. These three algorithms work well especially in the nonlinear manifold. In such a case, the linear methods such as PCA and MDS can not work. These three algorithms work well especially in the nonlinear manifold. In such a case, the linear methods such as PCA and MDS can not work.