Diffusion Maps and Spectral Clustering 1/14 Machine Learning Seminar Series Diffusion Maps and Spectral Clustering Author : Ronald R. Coifman et al. (Yale University) Presenter : Nilanjan Dasgupta (SIG Inc.)

Motivation Data lie on a low-dimensional manifold. The shape of the 2/14 Motivation X Y Z -- Datum Low-dimensional Manifold Data lie on a low-dimensional manifold. The shape of the manifold is not known a priori. PCA would fail to make compact representation since the manifold is not linear ! Spectral clustering as a non-linear dimensionality reduction scheme.

Outline Non-linear dimensionality reduction and spectral clustering. 3/14 Outline Non-linear dimensionality reduction and spectral clustering. Diffusion based probabilistic interpretation of spectral methods. Eigenvectors of normalized graph Laplacian is a discrete approximation of the continuous Fokker-Plank operator. Justification of the success of spectral clustering. Conclusions.

Spectral clustering Nomalized graph Laplacian : 4/14 Spectral clustering Nomalized graph Laplacian : Given N data points where each , the distance (similarity) between any two points xi and xj is given by with Gaussian kernel of width e and a diagonal normalization matrix Solve the normalized eigenvalue problem Use first few eigenvectors of M for low-dimensional representation of data or good coordinates for clustering.

Spectral Clustering : previous work 5/14 Spectral Clustering : previous work Non-linear dimensionality analysis by S. Roweis and L.Saul (published in Science magazine, 2000). Belkin & Niyogi (NIPS’02) show that if data are sampled uniformly from the low-dimensional manifold, first few eigenvectors of M=D-1L are discrete approximation of the Laplace-Beltrami operator on the manifold. Meila & Shi (AIStat’01) interpret M as a stochastic matrix representing random walk on the graph.

Diffusion distance and Diffusion map 6/14 Diffusion distance and Diffusion map A symmetric matrix Ms can be derived from M as M and Ms has same N eigenvalues, Under random walk representation of the graph M f : left eigenvector of M y : right eigenvector of M e : time step

Diffusion distance and Diffusion map 7/14 Diffusion distance and Diffusion map e has the dual representation (time step and kernel width). If one starts random walk from location xi , the probability of landing in location y after r time steps is given by For large e, all points in the graph are connected (Mi,j >0) and the eigenvalues of M where ei is a row vector with all zeros except that ith position = 1.

Diffusion distance and Diffusion map 8/14 Diffusion distance and Diffusion map One can show that regardless of starting point xi Left eigenvector of M with eigenvalue l0=1 with Eigenvector f0(x) has the dual representation : 1. Stationary probability distribution on the curve, i.e., the probability of landing at location x after taking infinite steps of random walk (independent of the start location). 2. It is the density estimate at location x.

Diffusion distance For any finite time r, 9/14 Diffusion distance For any finite time r, yk and fk are the right and left eigenvectors of graph Laplacian M. is the kth eigenvalue of M r (arranged in descending order). Given the definition of random walk, we denote Diffusion distance as a distance measure at time t between two pmfs as with empirical choice w(y)=1/f0(y).

Diffusion Map k eigenvectors as Relationship : 10/14 Diffusion Map Diffusion distance : Diffusion map : Mapping between original space and first k eigenvectors as Relationship : This relationship justifies using Euclidean distance in diffusion map space for spectral clustering. Since , it is justified to stop at appropriate k with a negligible error of order O(lk+1/lk)t).

Asymptotics of Diffusion Map 11/14 Asymptotics of Diffusion Map Suppose {xi} are sampled i.i.d. from probability density p(x) defined over manifold Z X Y Suppose p(x) = e-U(x) with U(x) is potential (energy) at location x. As , random walk on a discrete graph converges to random walk on the continuous manifold W. The forward and backward operators are given by

Asymptotics of Diffusion Map 12/14 Asymptotics of Diffusion Map Tf[f] : the probability distribution after one time-step e f(x) is probability distribution on the graph at t=0. Tb[y](x) is the mean of function y after one time-step e, for a random walk that started at location x at time t=0. Consider the limit , i.e., when each data point contains infinite nearby neighbors. Hence in that limit, random walk converges to a diffusion process with probability density evolving continuously in time as

Fokker-Plank operator 13/14 Fokker-Plank operator Infinitesimal generators (propagators) : The eigenfunctions of Tf and Tb converge to those of Hf and Hb, respectively. The backward generator is given by the Fokker –Plank operator which corresponds to a diffusion process in a potential field 2U(x).

Spectral clustering and Fokker-Plank operator 14/14 Spectral clustering and Fokker-Plank operator The term is interpreted as the drift term towards low potential (higher data density). The left and right eigenvectors of M can be viewed as discrete approximations of Tf and Tb, respectively. Tf and Tb can be viewed as approximation to Hf and Hb, which in the asymptotic case ( ) can be viewed as diffusion process with potential 2U(x) (p(x)=exp(-U(x)).