1. Spectral Clustering Eric Xing Lecture 8, August 13, 2010
Machine Learning
Spectral Clustering
Eric Xing
Lecture 8, August 13, 2010
Eric Xing
© Eric CMU,
Reading:

2. Data Clustering
Eric Xing
© Eric CMU,

3. Eric Xing
© Eric CMU,

4. Data Clustering Two different criteria Connectivity Compactness
Compactness, e.g., k-means, mixture models
Connectivity, e.g., spectral clustering
Connectivity
Compactness
Eric Xing
© Eric CMU,

5. Spectral Clustering Data Similarities Eric Xing
© Eric CMU,

6. Weighted Graph Partitioning
Wij i j
Some graph terminology
Objects (e.g., pixels, data points)
iÎ I = vertices of graph G
Edges (ij) = pixel pairs with Wij > 0
Similarity matrix W = [ Wij ]
Degree
di = SjÎG Wij
dA = SiÎA di degree of A G
Assoc(A,B) = SiÎA SjÎB Wij i A B
Eric Xing
© Eric CMU,

7. Cuts in a Graph
(edge) cut = set of edges whose removal makes a graph disconnected
weight of a cut:
cut( A, B ) = SiÎA SjÎB Wij=Assoc(A,B)
Normalized Cut criteria: minimum cut(A,Ā)
More generally:
Eric Xing
© Eric CMU,

8. Graph-based Clustering
Data Grouping
Image sigmentation
Affinity matrix:
Degree matrix:
Laplacian matrix:
(bipartite) partition vector:
Wij i j
G = {V,E}
Eric Xing
© Eric CMU,

9. Affinity Function Affinities grow as s grows 
How the choice of s value affects the results?
What would be the optimal choice for s?
Eric Xing
© Eric CMU,

10. Clustering via Optimizing Normalized Cut
The normalized cut:
Computing an optimal normalized cut over all possible y (i.e., partition) is NP hard
Transform Ncut equation to a matrix form (Shi & Malik 2000):
Still an NP hard problem
Subject to:
Rayleigh quotient
Eric Xing
© Eric CMU,

11. Relaxation
Instead, relax into the continuous domain by solving generalized eigenvalue system:
Which gives:
Note that so, the first eigenvector is y0=1 with eigenvalue 0.
The second smallest eigenvector is the real valued solution to this problem!!
Rayleigh quotient
Subject to:
Rayleigh quotient theorem
Eric Xing
© Eric CMU,

12. Algorithm Define a similarity function between 2 nodes. i.e.:
Compute affinity matrix (W) and degree matrix (D).
Solve
Do singular value decomposition (SVD) of the graph Laplacian
Use the eigenvector with the second smallest eigenvalue, , to bipartition the graph.
For each threshold k, Ak={i | yi among k largest element of y*}
Bk={i | yi among n-k smallest element of y*}
Compute Ncut(Ak,Bk)
Output Þ
Eric Xing
© Eric CMU,

13. Ideally …
y2i i A
y2i i A
Eric Xing
© Eric CMU,

14. Example (Xing et al, 2001)
Eric Xing
© Eric CMU,

15. Poor features can lead to poor outcome (Xing et al 2001)
Eric Xing
© Eric CMU,

16. Cluster vs. Block matrix
Eric Xing
© Eric CMU,

17. Compare to Minimum cut Criterion for partition: A Problem! B Cuts with
Weight of cut is directly proportional to the number of edges in the cut. B
Ideal Cut
Cuts with
lesser weight
than the
ideal cut
First proposed by Wu and Leahy
Eric Xing
© Eric CMU,

18. Superior Performance?
K-means and Gaussian mixture methods are biased toward convex clusters
Eric Xing
© Eric CMU,

19. Ncut is superior in certain cases
Eric Xing
© Eric CMU,

20. Why?
Eric Xing
© Eric CMU,

21. General Spectral Clustering
Data
Similarities
Eric Xing
© Eric CMU,

22. Representation Partition matrix X: Pair-wise similarity matrix W:
Degree matrix D:
Laplacian matrix L:
segments
pixels
Eric Xing
© Eric CMU,

23. A Spectral Clustering Algorithm Ng, Jordan, and Weiss 2003
Given a set of points S={s1,…sn}
Form the affinity matrix
Define diagonal matrix Dii= Sk aik
Form the matrix
Stack the k largest eigenvectors of L to for the columns of the new matrix X:
Renormalize each of X’s rows to have unit length and get new matrix Y. Cluster rows of Y as points in R k
Eric Xing
© Eric CMU,

24. SC vs Kmeans
Eric Xing
© Eric CMU,

25. Why it works? K-means in the spectrum space ! Eric Xing
© Eric CMU,

26. Eigenvectors and blocks
Block matrices have block eigenvectors:
Near-block matrices have near-block eigenvectors:
1= 2
2= 2
3= 0
4= 0 1
.71
.71
eigensolver
1= 2.02
2= 2.02
3= -0.02
4= -0.02 1 .2
-.2
.71
.69
.14
-.14
.69
.71
eigensolver
Eric Xing
© Eric CMU,

27. Spectral Space Can put items into blocks by eigenvectors:
Clusters clear regardless of row ordering: e1 1 .2
-.2
.71
.69
.14
-.14
.69
.71 e2 e1 e2 e1 1 .2
-.2
.71
.14
.69
.69
-.14
.71 e2
Eric Xing
© Eric CMU, e1 e2

28. More formally … Recall generalized Ncut
Minimizing this is equivalent to spectral clustering
segments
pixels
Eric Xing
© Eric CMU,

29. Spectral Clustering
Algorithms that cluster points using eigenvectors of matrices derived from the data
Obtain data representation in the low-dimensional space that can be easily clustered
Variety of methods that use the eigenvectors differently (we have seen an example)
Empirically very successful
Authors disagree:
Which eigenvectors to use
How to derive clusters from these eigenvectors
Two general methods
Eric Xing
© Eric CMU,

30. Method #1 Partition using only one eigenvector at a time
Use procedure recursively
Example: Image Segmentation
Uses 2nd (smallest) eigenvector to define optimal cut
Recursively generates two clusters with each cut
Eric Xing
© Eric CMU,

31. Method #2 Use k eigenvectors (k chosen by user)
Directly compute k-way partitioning
Experimentally has been seen to be “better”
Eric Xing
© Eric CMU,

32. Images from Matthew Brand (TR-2002-42)
Toy examples
Images from Matthew Brand (TR )
Eric Xing
© Eric CMU,

33. User’s Prerogative Choice of k, the number of clusters
Choice of scaling factor
Realistically, search over and pick value that gives the tightest clusters
Choice of clustering method: k-way or recursive bipartite
Kernel affinity matrix
Eric Xing
© Eric CMU,

34. Conclusions Good news: Bad news:
Simple and powerful methods to segment images.
Flexible and easy to apply to other clustering problems.
Bad news:
High memory requirements (use sparse matrices).
Very dependant on the scale factor for a specific problem.
Eric Xing
© Eric CMU,