1. Spectral clustering of graphs
Sushmita Roy
Computational Network Biology Biostatistics & Medical Informatics 826 Computer Sciences 838
Nov 3rd, Nov 10th 2016

2. Goals for this lecture Graph Laplacian A few linear algebra concepts
Connected components from the Laplacian
Spectral clustering
Spectral clustering and graph cut
Spectral clustering demo with Zachary Karate Club
Application of spectral clustering to identify integrated clusters

3. Notation
Graph G={V, E} where V is the vertex set and E is the edge set
D: Degree matrix of a graph
A: Adjacency matrix of a graph
L: Graph Laplacian

4. Unnormalized Graph Laplacian
For a given graph G={V,E}
The unnormalized graph Laplacian is |V|X|V| matrix

5. Unnormalized Graph Laplacian example
Adjacency matrix(A)
Degree matrix (D) 1 2 3 4 2 3 4 1 1 1 1 2 1 3 2 3 4 2 4
Laplacian (L=D-A) 1 2 3 4 2 -1 1
Example graph

6. A few linear algebra concepts
Let A be an nXn matrix
Let A’ denote the transpose of A
Let Rn denote the space of nX1-dimensional real-valued vectors
Eigen vector of A
v, in Rn, is the eigen vector (nX1) of a matrix A with eigen value λ, (a scalar) if
Positive semi-definite
A is said to be positive semi-definite if, for any x in Rn, the following holds

7. Spectral clustering
Based on the graph Laplacian derived from the adjacency matrix
Graph Laplacian L=D-A
D is the diagonal degree of matrix
A is the adjacency matrix
Obtain the k eigen vectors associated with k smallest eigen values of L
Represent each node as the k-dimensional vector
Cluster nodes based on k-means clustering

8. Spectral clustering key steps1 N 1 N 1 k k 1
Adjacency matrix (A) 1
Laplacian (L) 1
First k eigen vectors
k clusters … N N

9. Properties of the Laplacian
L is symmetric and positive semi-definite
For every vector f in Rn,
The smallest eigen value of L is 0 and its corresponding eigen vector is all 1s
Intuitively the L is a second derivative, it is asking how fast is a function changing on the graph

10. Connected components
Two connected components
A subgraph where there is path from one node to another
The number of connected components is inherently tied to the Laplacian matrix B G C F A H E D
Connected component: A subgraph spanning a vertex subset where every vertex can be “reached” from another vertex

11. Number of connected components and the multiplicity of λ=0
Let G be an undirected graph with non-negative weights.
Then the multiplicity, k, of the eigenvalue 0 of L equals the number of connected components in the graph A1 , , Ak

12. Number of Connected Components and L’s smallest eigen value
To see why this is true, we use the property of an eigen vector, consider the case of one connected component
If f is an eigen vector of L, then Lf=λf
For eigen value 0, Lf=0 (vector or all zeros)
In addition we know
If f is an eigen vector corresponding to eigen value =0, this must be 0
The only way this can be 0 is if fi=fj because wij is non-zero
This holds for all vertices connected by a path
If all vertices are connected, then f is a vector of constant of all 1s

13. Now consider a graph with k components
A is block diagonal with k blocks
L is also block diagonal

14. Now consider the case of a graph with k components
Eigen vectors of each block diagonal matrices and the full graph are directly related
Eigen vectors of the full graph is the same as the eigen vectors of individual blocks with the remaining entries set to 0.
Thus L for the entire graph has as many eigen vectors as the number of connected components (blocks)
Each eigen vector is constant non-zeros for the entries corresponding to each connected component

15. Graph cuts
A cut on graph is defined as a partitioning on the nodes of the graph into two sets
The size of a cut is the number of edges spanning the cut
Cut size=2

16. Different types of graph cuts
A cut can be
Min Cut: No other cut is smaller (fewer edges)
Max Cut: No other cut is larger

17. Graph clustering from a graph cut point of view
Clustering on the graph can be re-stated as follows:
Find a partition of the graph such that the edges between different groups have a very low weight (which means that points in different clusters are dissimilar from each other) and the edges within a group have high weight (which means that points within the same cluster are similar to each other).
The most direct way to find such a partition is by solving the Min-Cut problem
This is an NP-hard problem
The spectral clustering algorithm can be thought of as solving a continuous relaxation of this problem
Luxburg Tutorial on Spectral graph clustering

18. Goals for this lecture Graph Laplacian A few linear algebra concepts
Connected components from the Laplacian
Spectral clustering
Spectral clustering and graph cut
Spectral clustering demo with Zachary Karate Club
Application of spectral clustering to identify integrated clusters

19. An example with 2 connected components
For this matrix, we expect to have two eigen vectors associated with eigen value =0 A1 A2

20. The corresponding Laplacian
L=D-A L1 L2

21. First 10 ten eigen values
The first two eigen values are 0

22. First 10 eigen vectors
First two eigen vectors are piece-wise constant

23. Recall the Zachary karate club study

24. Adjacency matrix of Zachary Karate club study

25. First 20 eigen values of the ZKC graph
Only 1 eigen value=0
K=5 clusters
K=9 clusters
Only one connected component

26. First two eigen vectors of the ZKC graph
Cluster 2
Cluster 1

27. Clusters depicted on the graph

28. Spectral vs Girvan-Newman

29. First five eigen vectors of Zachary Karate club data

30. Reordered matrix post clustering

31. ZKC graph clustered into k=5 clusters

32. Goals for this lecture Graph Laplacian A few linear algebra concepts
Connected components from the Laplacian
Spectral clustering
Spectral clustering and graph cut
Spectral clustering demo with Zachary Karate Club
Application of spectral clustering to identify integrated clusters
Similarity network fusion

33. Application of spectral clustering
Similarity network fusion for aggregating data types on a genomic scale
B. Wang, A. M. Mezlini, F. Demir, M. Fiume, Z. Tu, M. Brudno, B. Haibe-Kains, and A. Goldenberg
This paper had two goals:
Integrate different types of data using a network-based approach
Identify groups of samples representing integrated data types

34. Similarity network fusion for aggregating data types on a genomic scale
Recent high throughput technologies have made it possible to collect many different types of genomic data for individual patients
How do we combine patient data to describe a disease?
This is challenging because of the following issues:
Noisy samples
Small number of samples than variables
Complimentary nature of the data

35. Similarity Network Fusion
Given N different types of measurements for different individuals Do
Construct a similarity matrix of individuals for each data type
Integrate the networks using a single similarity matrix using an iterative algorithm
Cluster the network into a groups of individuals

36. Similarity network fusion with two data types
Similarity network fusion (Nodes are patients, edges represent similarities).

37. Defining a similarity graph over patient samples
For each data type, create a weighted graph, with vertices corresponding to patients
Let xi and xj denote the measurements of patients i and j
Edge weights, W(i,j) correspond to how similar patient i is to patient j based on xi and xj
Euclidean distance
Hyper-parameter
Scaling term (average of the distance between each node and its neighborhood)

38. Creating a fused matrix
Define two matrices for each data type
A full matrix: normalized weight matrix
A sparse matrix (based on k nearest neighbors or each node)
This makes the assumption that the local similarities are the most reliable

39. Iterate for fusion Input m data types
Construct W(v) for each data type v
Construct dense matrix P(v) and sparse matrix S(v)
At each iteration, update the dense similarity matrix of one data type using the similarity matrix of the other data type

40. Iteration with m=2 data types
For iteration t+1
Update similarity matrix of data type 1
Update similarity matrix of data type 2
Update similarity matrix of data type 1 using weight matrix from data type 2 and vice-versa

41. What is going on in the iteration step
Neighbors of i
Neighbors of j
We are updating the similarity matrix using the most confident common neighbors of i and j

42. Extending to m>2 data types
Just average over all other data types

43. SNF termination
After repeating the iterative updates for t steps, final similarity matrix is
This is then clustered using spectral clustering

44. Application of SNF to Glioblastoma
Contradicting information about subtypes depending upon the type of data used
Glioblastoma dataset
Three data types among 215 patients
DNA methylation (1491 genes)
mRNA (12,042 genes)
miRNA (534 miRNAs)

45. SNF application to GBM identifies 3 subtypes
DNA methylation
mRNA expression
miRNA expression

46. Validation of SNF identified subtypes
Subtypes are associated with patient populations of different survival.
Blue curve (subtype 3) are patients with more favorable prognosis

47. Key points of graph clustering algorithms
Flat or hierarchical clustering
Algorithms differ in
how they define the similarity/distance measure
Local topology measures
Global measures
Whether the algorithm takes as input the number of clusters or the goodness of clusters (e.g. the approximate cluster algorithm)
Next time: Seed and scoring methods

48. References A Tutorial on Spectral Clustering.
Ulrike von Luxburg, 2007
Similarity network fusion for aggregating data types on a genomic scale
B. Wang, A. M. Mezlini, F. Demir, M. Fiume, Z. Tu, M. Brudno, B. Haibe-Kains, and A. Goldenberg, Nat Meth, vol. 11, no. 3, pp , Mar [Online]. Available: