Power Iteration Clustering Speaker: Xiaofei Di

Outline Authors Abstract Background Power Iteration Clustering(PIC) Conclusion

Authors Frank Lin PhD Student Language Technologies Institute School of Computer Science Carnegie Mellon University William W. Cohen Associate Research Professor, Machine Learning Department, Carnegie Mellon UniversityMachine Learning DepartmentCarnegie Mellon University

Abstract We present a simple and scalable graph clustering method called power iteration clustering. PIC finds a very low-dimensional embedding of a dataset using truncated power iteration on a normalized pair-wise similarity matrix of the data. This embedding turns out to be an effective cluster indicator, consistently outperforming widely used spectral methods such as Ncut on real datasets. PIC is very fast on large datasets, running over 1000 times faster than an Ncut implementation based on the state-of-the-art IRAM eigenvector computation technique.

摘要 本文提出了一种简单可扩展的图聚类方法： 快速迭代聚类（ PIC ）。 PIC 利用数据归一化的逐对相似度矩阵，采 用截断的快速迭代法，寻找数据集的一个 超低维嵌入。这种嵌入恰好是很有效的聚 类指标，使它在真实的数据集上总是好于 广泛使用的谱聚类方法，比如 NCut 。 在大规模数据集上， PIC 非常快，比基于最 好的特征计算技术实现的 Ncut 快 1000 倍。

Background spectral clustering

Background Power Iteration Method Advantage dose not compute matrix decomposition Disadvantages finds only the largest eigenvalue and converges slowly An eigenvalue algorithm – Input: initial vector b0 and the matrix A – Iteration: Convergence Under the assumptions:  A has an eigenvalue that is strictly greater in magnitude than its other eigenvalues  The starting vector b 0 has a nonzero component in the direction of an eigenvector associated with the dominant eigenvalue. then:  A subsequence of converges to an eigenvector associated with the dominant eigenvalue

Power Iteration Clustering(PIC) Unfortunately, since the sum of each row of NA is 1, the largest eigenvector of NA (the smallest of L) is a constant vector with eigenvalue 1. Fortunately, the intermediate vectors during the convergence process are interesting. Example: Conclusion: PI first converges locally within a cluster.

PI’s Convergence Let: W = NA (Normalized affinity matrix ), Spectral representation of Spectral distance between a and b:

a)The size of the radius is of no importance in clustering, because most clustering methods based on relative distance, not absolute one. b) The importance of the dimension associated with the i-th eigenvector is downweighted by (a power of) its eigenvalue, which often improves performance for spectral methods. c) For many natural problems, W is approximately block-stochastic, and hence the first k eigenvectors are approximately piecewise constant over the k clusters. It is easy to see that when spec(a,b) is small, signal must also small. However, when a and b are in different clusters, since the terms are signed and additively combined, it is possible that they may “cancel out” and make a and b seem to be in the same cluster. Fortunately, this seems to be uncommon in practice when the cluster number k is not too large.

Early stopping for PI While the clusters are ‘’locally converging”, the rate of convergence changes rapidly; whereas during the final global convergence, the converge rate appears more stable.

Experiments (1/3) Purity : cluster purity NMI : normalized mutual information RI : rand index The Rand index or Rand measure is a measure of the similarity between two data clusterings. Given a set of n elements and two partitions of S to compare, and, we define the following: data clusteringssetelementspartitions a = | S * |, where b = | S * |, where c = | S * |, where d = | S * |, where for some Then:

Experiments (2/3) Experimental comparisons on accuracy of PIC Experimental comparisons on eigenvalue weighting

Experiments (3/3) Experimental comparisons on scalability NCutE uses slower, classic eigenvalue decomposition method to find all eigenvectors. NCutI uses fast Implicitly Restarted Arnoldi Method(IRAM) for the top k eigenvectors. Synthetic dataset

Conclusion Novel Simple Efficient

Appendix ----NCut

Appendix ----NJW