Clustering Ram Akella Lecture 6 February 23, & 280I University of California Berkeley Silicon Valley Center/SC

Class Outline  Clustering Introduction  Mechanics  Recommender Systems

What is Cluster Analysis?  Finding groups of objects such that the objects in a group will be similar (or related) to one another and different from (or unrelated to) the objects in other groups Inter-cluster distances are maximized Intra-cluster distances are minimized

Potential Applications  Numerical Taxonomy Differentiation between different species of animals or plants according to their physical similarities Fifty plants from3 different species were collected and some measurements were taken such as: sepal length, sepal width, petal length and petal width.

Potential Applications  Market Segmentation The market is divided into smaller groups that are more homogeneous needs and wants of the market place

Objectives of cluster analysis  Address the heterogenity in the data Divide the data into more homogeneous groups  Find the natural modality of data Determine whether the data contains naturally occurring, homogeneous subsets

Measures of distance, dissimilarity and density  To measure of proximity of data we can use Direct assessment of proximity Derived measure of proximity or similarity

Distance Measurements Euclidean Distance This is usually applied to standardized data to give the same weight to all the metrics (except when the input is the Principal Components)

Distance Measurements Minkowski Metric  The Euclidean distance is a especial case when p=2  When p=1 it is called city-block metric because it is like walking from point A to point B in a city laid out with a grid of streets

Mahalanobis Distance Where Σ is the population covariance matrix of the data matrix X The Mahalanobis distance captures the fact that a point A and a point B are equally likely to have been drawn from a multivariate normal distribution This distance takes into account the covariance patterns of the data

Agglomerative Clustering  Start with the points as individual clusters  At each step, merge the closest pair of clusters until only one cluster (or k clusters) left  More popular hierarchical clustering technique   Key operation is the computation of the proximity of two clusters Different approaches to defining the distance between clusters distinguish the different algorithms

Agglomerative Clustering Algorithm Basic algorithm is : 1.Compute the proximity matrix 2.Let each data point be a cluster 3.Repeat 4.Merge the two closest clusters 5.Update the proximity matrix 6.Until only a single cluster remains

Starting Situation Start with clusters of individual points and a proximity matrix p1 p3 p5 p4 p2 p1p2p3p4p Proximity Matrix

Intermediate Situation After some merging steps, we have some clusters C1 C4 C2 C5 C3 C2C1 C3 C5 C4 C2 C3C4C5 Proximity Matrix

Intermediate Situation We want to merge the two closest clusters (C2 and C5) and update the proximity matrix. C1 C4 C2 C5 C3 C2C1 C3 C5 C4 C2 C3C4C5 Proximity Matrix

After Merging The question is: “How do we update the proximity matrix?” C1 C4 C2 U C5 C3 ? ? ? ? ? C2 U C5 C1 C3 C4 C2 U C5 C3C4 Proximity Matrix

How to Define Inter-Cluster Similarity p1 p3 p5 p4 p2 p1p2p3p4p  MIN  MAX  Group Average  Distance Between Centroids  Ward’s Method Similarity? Proximity Matrix

How to Define Inter-Cluster Similarity p1 p3 p5 p4 p2 p1p2p3p4p Proximity Matrix  MIN  MAX  Group Average  Distance Between Centroids  Ward’s Method

How to Define Inter-Cluster Similarity p1 p3 p5 p4 p2 p1p2p3p4p Proximity Matrix  MIN  MAX  Group Average  Distance Between Centroids  Ward’s Method

How to Define Inter-Cluster Similarity p1 p3 p5 p4 p2 p1p2p3p4p Proximity Matrix  MIN  MAX  Group Average  Distance Between Centroids  Ward’s Method

How to Define Inter-Cluster Similarity p1 p3 p5 p4 p2 p1p2p3p4p Proximity Matrix  MIN  MAX  Group Average  Distance Between Centroids  Ward’s Method 

Ward’s Method  Similarity of two clusters is based on the increase in squared error when two clusters are merged  Less susceptible to noise and outliers  Biased towards globular clusters  Hierarchical analogue of K-means Can be used to initialize K-means

K-means Clustering  Partitional clustering approach  Each cluster is associated with a centroid (center point)  Each point is assigned to the cluster with the closest centroid  Number of clusters, K, must be specified  The basic algorithm is very simple

K-means Clustering – Details  Initial centroids are often chosen randomly. Clusters produced vary from one run to another.  The centroid is (typically) the mean of the points in the cluster.  ‘Closeness’ is measured by Euclidean distance, cosine similarity, correlation, etc.  K-means will converge for common similarity measures mentioned above.  Most of the convergence happens in the first few iterations. Often the stopping condition is changed to ‘Until relatively few points change clusters’

Two different K-means Clustering Sub-optimal ClusteringOptimal Clustering Original Points

Importance of Choosing Initial Centroids

How many clusters  To decide the appropriate number of clusters Run the analysis for different values of k and compare the solutions with the pseudo-F statistic given by Where B is the inter-cluster sum of squares matrix, W is the intra-cluster sum of squares, K is the number of clusters and n the number of data points The larger the pseudo-F statistic, the more efficient the clustering is

Evaluating K-means Clusters  Most common measure is Sum of Squared Error (SSE) For each point, the error is the distance to the nearest cluster To get SSE, we square these errors and sum them. x is a data point in cluster C i and m i is the representative point for cluster C i  m i corresponds to the center (mean) of the cluster Given two clusters, we can choose the one with the smallest error One easy way to reduce SSE is to increase K, the number of clusters  A good clustering with smaller K can have a lower SSE than a poor clustering with higher K

Problems with Selecting Initial Points  If there are K ‘real’ clusters then the chance of selecting one centroid from each cluster is small. Chance is relatively small when K is large If clusters are the same size, n, then For example, if K = 10, then probability = 10!/10 10 = Sometimes the initial centroids will readjust themselves in ‘right’ way, and sometimes they don’t Consider an example of five pairs of clusters

Solutions to Initial Centroids Problem  Multiple runs Helps, but probability is not on your side  Sample and use hierarchical clustering to determine initial centroids  Select more than k initial centroids and then select among these initial centroids Select most widely separated  Post-processing  Bisecting K-means Not as susceptible to initialization issues

Handling Empty Clusters  Basic K-means algorithm can yield empty clusters  Several strategies Choose the point that contributes most to SSE Choose a point from the cluster with the highest SSE If there are several empty clusters, the above can be repeated several times.

Updating Centers Incrementally  In the basic K-means algorithm, centroids are updated after all points are assigned to a centroid  An alternative is to update the centroids after each assignment (incremental approach) Each assignment updates zero or two centroids More expensive Introduces an order dependency Never get an empty cluster Can use “weights” to change the impact

Pre-processing and Post- processing  Pre-processing Normalize the data Eliminate outliers  Post-processing Eliminate small clusters that may represent outliers Split ‘loose’ clusters, i.e., clusters with relatively high SSE Merge clusters that are ‘close’ and that have relatively low SSE Can use these steps during the clustering process  ISODATA

Bisecting K-means  Bisecting K-means algorithm Variant of K-means that can produce a partitional or a hierarchical clustering