Clustering Clustering definition: Partition a given set of objects into M groups (clusters) such that the objects of each group are ‘similar’ and ‘different’ from the objects of the other groups. A distance (or similarity) measure is required. Unsupervised learning: no class labels Clustering is NP-complete ---> equivalence to graph partitioning Clustering Examples: documents, images, time series, image segmentation, video analysis, gene clustering, motif discovery, web applications

Clustering Cluster Assignments: hard vs (fuzzy/probabilistic) Clustering Methods Hierarchical (agglomerative, divisive) Density-based (non-parametric) Parametric (k-means, mixture models etc) Clustering Problems Data Vectors Similarity Matrix

Agglomerative Clustering The simplest approach and a good starting point Starting from singleton clusters at each step we merge the two most similar clusters A similarity (or distance) measure between clusters is needed Output: dendrogram Drawbacks: - merging decisions are permanent (cannot be corrected at a later stage) - cubic complexity

Density-Based Clustering (eg DBSCAN) Identify ‘dense regions’ in the data space Merge neighboring dense regions Require a lot of points Complexity: O(n2) Core Border Outlier Eps = 1cm MinPts = 5 (set empirically, how?)

Parametric methods k-means (data vectors): O(n) (n: the number of objects to be clustered) k-medoids (similarity matrix): O(n2) Mixture models (data vectors): O(n) Spectral clustering (similarity matrix): O(n3) Kernel k-means (similarity matrix): O(n2) Affinity Propagation (similarity matrix): O(n2)

k-means Partition a dataset X of N vectors xi into M subsets (clusters) Ck such that intra-cluster variance is minimized. Intra-cluster variance: distance from the cluster prototype mk k-means: Prototype = cluster center Finds local minima w.r.t. clustering error sum of intra-cluster variances Highly dependent on the initial positions (examples) of the centers mk

Spectral Clustering (Ng & Jordan, NIPS2001) Input: Similarity matrix between pairs of objects, number of clusters M Example: a(x,y)=exp(-||x-y||2/σ2) (RBF kernel) Spectral analysis of the similarity matrix: compute top M eigenvectors and form matrix U The i-th object corresponds to a vector in Rk : i-th row of U. Rows are clustered in M clusters using k-means

Spectral Clustering 2 rings dataset k-means spectral (RBF kernel,σ=1)

Spectral Clustering ↔ Graph cut Data graph Vertices: objects Edge weight: pairwise similarity Clustering = Graph Partitioning

Spectral Clustering ↔ Graph cut Cluster Indicator vector zi=(0,0,…,0,1,0,…0)T for object i Indicator matrix Z=[z1,…,zn] (nxk, for k clusters), ZTZ=I Graph partitioning = trace maximization wrt Z: The relaxed problem: is solved optimally using the spectral algorithm to obtain Y k-means is applied on yij to obtain zij

Kernel-Based Clustering (non-linear cluster separation) Given a set of objects and the kernel matrix K=[Kij] containing the similarities between each pair of objects Goal: Partition the dataset into subsets (clusters) Ck such that intra-cluster similarity is maximized. Kernel trick: Data points are mapped from input space to a higher dimensional feature space through a transformation φ(x). RBF kernel: K(x,y)=exp(-||x-y||2 /σ2)

Kernel k-Means Kernel k-means = k-means in feature space Minimizes the clustering error in feature space Differences from k-means Cluster centers mk in feature space cannot be computed Each cluster Ck is explicitly described by its data objects Computation of distances from centers in feature space: Finds local minima - Strong dependence on the initial partition Spectral clustering and kernel k-means optimize the same objective function, but in a different way  may converge to different local optima of the objective function

Exemplar-Based Methods Cluster data by identifying representative exemplars An exemplar is an actual dataset point, similar to a medoid All data points are considered as possible exemplars The number of clusters is decided during learning (but a depends on a user-defined parameter) Methods Convex Mixture Models Affinity Propagation

Affinity Propagation (AP) (Frey et al., Science 2007) Clusters data by identifying representative exemplars Exemplars are identified by transmitting messages between data points Input to the algorithm A similarity matrix where s(i,k) indicates how well data point xk is suited to be an exemplar for data point xi. Self-similarities s(k,k) that control the number of identified clusters and a higher value means that xk is more likely to become an exemplar Self-similarities are independent of the other similarities Higher values result in more clusters

Affinity Propagation Clustering criterion: s(i,ci) is the similarity between the data point xi and its exemplar Minimized by passing messages between data points, called responsibilities and availabilities Responsibility r(i,k): Sent from xi to candidate exemplar xk reflects the accumulated evidence for how well suited xk is to serve as the exemplar of xi taking into account other potential exemplars for xi

Affinity Propagation Availability a(i,k) Sent from candidate exemplar xk to xi reflects the accumulated evidence for how appropriate it would be for xi to choose xk as its exemplar, taking into account the support from other points that xk should be an exemplar The algorithm alternates between responsibility and availability calculation and The exemplars are the points with r(k,k)+a(k,k)>0 http://www.psi.toronto.edu/index.php?q=affinity%20propagation

Affinity Propagation

Maximum-Margin Clustering (MMC) MMC extends the large margin principle of SVM to clustering (also called unsupervised SVM) SVM training: Given a two-class dataset X={(xi , yi )} (yi =1 or -1) and a kernel matrix K (with corresponding feature transformation φ(x) ): find the maximum margin separating hyperplane (w, b) in φ(x).

Maximum-Margin Clustering (MMC) SVM training: MMC clustering: find the partition y of the dataset into two clusters such that the margin between the clusters is maximum: A cluster balance constraint is imposed to treat outliers Finds local minima – high complexity – two clusters only

Maximum-Margin Clustering (MMC) k-means solutions

Incremental Clustering Bisecting k-means (Steinbach,Karypis & Kumar, SIGKDD 2000) Start with k=1 (m1= data average) Assume a solution with k clusters Find the ‘best’ cluster split in two subclusters Replace the cluster center with the two subcluster centers Run k-means with k+1 centers (optional) k:=k+1 Until M clusters have been added Split a cluster using several random trials Each trial: Randomly initialize two centers from the cluster points Run 2-means using the cluster points only Keep the split of the trial with the lowest clustering error

Global k-means (Likas, Vlassis & Verbeek, PR 2003) Incremental, deterministic clustering algorithm that runs k-Means several times Finds near-optimal solutions wrt clustering error Idea: a near-optimal solution for k clusters can be obtained by running k-means from an initial state the k-1 centers are initialized from a near-optimal solution of the (k-1)-clustering problem the k-th center is initialized at some data point xn (which?) Consider all possible initializations (one for each xn)

Global k-means In order to solve the M-clustering problem: Solve the 1-clustering problem (trivial) Solve the k-clustering problem using the solution of the (k-1)-clustering problem Execute k-Means N times, initialized as at the n-th run (n=1,…,N). Keep the solution corresponding to the run with the lowest clustering error as the solution with k clusters k:=k+1, Repeat step 2 until k=M.

Best Initial m2 Best Initial m3 Best Initial m4 Best Initial m5

Clustering Big datasets First perform summarization of the big dataset using a low cost algorithm, then cluster the summarized dataset using any clustering algorithm: A linear clustering algorithm (e.g. kmeans) or one-shot clustering can be applied to the big dataset to initially partition this dataset into a large number (e.g. thousands) of clusters A representative (e.g. centroid or medoid) is selected for each cluster. The set of representatives is further clustered into a smaller number of clusters using any clustering method to get the final solution.

Clustering Methods: Summary Usually we assume that the number of clusters is given k-means is still the most widely used method Spectral clustering (or kernel k-means) the most popular when similarity matrix is given Beware of the parameter initialization problem! Ground truth absence makes evaluation difficult How could we estimate the number of clusters?