1. Math 285 Project Diffusion Maps Xiaoyan Chong Department of Mathematics and Statistics San Jose State University

2. Outline Motivation Algorithm Implement on toy data and real data Comparison with other dimensional reduction techniques Future work

3. Motivation Data lie on a low-dimensional manifold. The shape of the manifold is not known, discovering the underlying manifold PCA would fail to make compact representation since the manifold is not linear Low-dimensional Manifold X Y Z -- Datum

4. Diffusion Maps: Random Walk The Idea: to estimate the “true” distance between two data points via a diffusion (i.e., Markov random walk) process. Each jump has a probability associated with it Dash line from point 1 to point 6: Probability = p(node1, node2) * p(node2, node6) Jumping to a nearby data-point is more likely than jumping to a far away point This observation provides a relation between distance in the feature space and probability p1 p2

5. Diffusion Maps: Intuition

6. Diffusion Maps: The Math (I) Diffusion kernel: (The kernel indicates a local measure of similarity within a certain neighborhood ) Compute “one-step” probabilities, and normalized it (in row) Diffusion matrix P, with entries P ij = p(X i,X j ) The probability of stepping from i to j in t step is P T – With increased values of t, the probability of following a path along the underlying geometric structure of the data set increases. -- Along the geometric structure, points are dense and therefore highly connected. Pathways form along short, high probability jumps

7. Diffusion Maps: The Math (II) Diffusion distance is defined as: - Calculating diffusion distance is computationally expensive - Consider to map data points into a Euclidean space Diffusion map: -- using it for reducing dimension, and preserving the diffusion distance. -- The diffusion distance can be expressed in terms of the eigenvectors and eigenvalues of diffusion matrix P -- Dimensional reduction is achieved by retaining the m dimensions associated with the dominant eigenvectors The set of orthogonal eigenvectors of P form a basis for the diffusion space, and the associated eigenvalues indicate the importance of each dimension

8. Diffusion Maps Algorithm  INPUT: High dimensional data set Xi 1.Construct similarity graph (kernel) 2.Create diffusion matrix by normalizing the rows of the kernel matrix 3.Calculate the eigenvectors of the diffusion matrix 4.Map points to the d-dimensional diffusion space at time t, using d dominant eigenvectors and eigenvalues  Output: Low dimensional dataset Yi

9. Toy Data: Annulus

10. t = 1 t = 10 t = 50 t = 200t = 500t = 1000 The probability of jumping to another in one time-step is small At this time scale, all points are equally well connected, and the diffusion distances between points are small

11. Principal Component(PCA) – Linear structure Multidimensional Scaling (MDS) – Linear; Euclidean Distance Isomap – Nonlinear; Geodesic Distance, not robust to noise Diffusion Maps – Nonlinear (The technique is robust to noise perturbation and is computationally inexpensive) Methods Comparison

12. Iris Data

13. PCA Diffusion map ISOmap MDS

14. Toy data II t = 1 t = 2 t = 3 t = 10

15. Comparison PCA MDS ISOmap Diffusion Maps

16. Comparison of methods PCAMDSISOMAPDiffusion Map SpeedExtremely fast Very slowExtremely slowFast Infers geometry?NO YESMAYBE Handles non-convex?NO MAYBE Handles non-uniform sampling? YES Handles curvature?NO YES Handles corners?NO YES Clusters?YES Handles noise?YES NOYES Handles sparsity?YES NO Sensitive to parameters? NO YESVERY

17. Future work Task: isolated-word recognition on a small vocabulary The embedding of the lip data into the top 3 diffusion coordinates These coordinates essentially capture two parameters: One controlling the opening of the mouth Measuring the portion of teeth that are visible

18. Thank you