1. CSE572, CBS572: Data Mining by H. Liu
Clustering
Basic concepts with simple examples
Categories of clustering methods
Challenges
1/18/2019
CSE572, CBS572: Data Mining by H. Liu

2. CSE572, CBS572: Data Mining by H. Liu
What is clustering?
The process of grouping a set of physical or abstract objects into classes of similar objects.
It is also called unsupervised learning.
It is a common and important task that finds many applications
Examples of clusters?
Examples where we need clustering?
1/18/2019
CSE572, CBS572: Data Mining by H. Liu

3. Differences from Classification
How different?
Which one is more difficult as a learning problem?
Do we perform clustering in daily activities?
How do we cluster?
How to measure the results of clustering?
With/without class labels
Between classification and clustering
Semi-supervised clustering
1/18/2019
CSE572, CBS572: Data Mining by H. Liu

4. Major clustering methods
Partitioning methods
k-Means (and EM), k-Medoids
Hierarchical methods
agglomerative, divisive, BIRCH
Similarity and dissimilarity of points in the same cluster and from different clusters
Distance measures between clusters
minimum, maximum
Means of clusters
Average between clusters
1/18/2019
CSE572, CBS572: Data Mining by H. Liu

5. CSE572, CBS572: Data Mining by H. Liu
Clustering -- Example 1
For simplicity, 1-dimension objects and k=2.
Objects: 1, 2, 5, 6,7
K-means:
Randomly select 5 and 6 as centroids;
=> Two clusters {1,2,5} and {6,7}; meanC1=8/3, meanC2=6.5
=> {1,2}, {5,6,7}; meanC1=1.5, meanC2=6
=> no change.
Aggregate dissimilarity = 0.5^ ^2 + 1^2 + 1^2 = 2.5
1/18/2019
CSE572, CBS572: Data Mining by H. Liu

6. CSE572, CBS572: Data Mining by H. Liu
Issues with k-means
A heuristic method
Sensitive to outliers
How to prove it?
Determining k
Trial and error
X-means, PCA-based
Crisp clustering
EM, Fuzzy c-means
Not be confused with k-NN
X-means: Extending K-means with Efficient Estimation of the Number of Clusters (2000)
Dan Pelleg, Andrew Moore  
C-means,
1/18/2019
CSE572, CBS572: Data Mining by H. Liu

7. CSE572, CBS572: Data Mining by H. Liu
Clustering -- Example 2
For simplicity, we still use 1-dimension objects.
Objects: 1, 2, 5, 6,7
agglomerative clustering – a very frequently used algorithm
How to cluster:
find two closest objects and merge;
=> {1,2}, so we have now {1.5,5, 6,7};
=> {1,2}, {5,6}, so {1.5, 5.5,7};
=> {1,2}, {{5,6},7}.
1/18/2019
CSE572, CBS572: Data Mining by H. Liu

8. Issues with dendrograms
How to find proper clusters
An alternative: divisive algorithms
Top down
Comparing with bottom-up, which is more efficient
What’s the time complexity?
How to efficiently divide the data
A heuristic – Minimum Spanning Tree
What’s the time complexity
1/18/2019
CSE572, CBS572: Data Mining by H. Liu

9. CSE572, CBS572: Data Mining by H. Liu
Distance measures
Single link
Measured by the shortest edge between the two clusters
Complete link
Measured by the longest edge
Average link
Measured by the average edge length
An example is shown next.
1/18/2019
CSE572, CBS572: Data Mining by H. Liu

10. An example to show different Links
Single link
Merge the nearest clusters measured by the shortest edge between the two
(((A B) (C D)) E)
Complete link
Merge the nearest clusters measured by the longest edge between the two
(((A B) E) (C D))
Average link
Merge the nearest clusters measured by the average edge length between the two A B C D E 1 2 3 4 5 A B
This example is from M. Dunham’s book (see the bib) E C D
1/18/2019
CSE572, CBS572: Data Mining by H. Liu

11. CSE572, CBS572: Data Mining by H. Liu
Other Methods
Density-based methods
DBSCAN: a cluster is a maximal set of density-connected points
Core points defined using epsilon-neighborhood and minPts
Apply directly density reachable (e.g., P and Q, Q and M) and density-reachable (P and M, assuming so are P and N), and density-connected (any density reachable points, P, Q, M, N) form clusters
Grid-based methods
STING: the lowest level is the original data
statistical parameters of higher-level cells are computed from the parameters of the lower-level cells (count, mean, standard deviation, min, max, distribution
Model-based methods
Conceptual clustering: COBWEB
Category utility
Intraclass similarity
Interclass dissimilarity
1/18/2019
CSE572, CBS572: Data Mining by H. Liu

12. CSE572, CBS572: Data Mining by H. Liu
Density-based
DBSCAN – Density-Based Clustering of Applications with Noise
It grows regions with sufficiently high density into clusters and can discover clusters of arbitrary shape in spatial databases with noise.
Many existing clustering algorithms find spherical shapes of clusters
DEBSCAN defines a cluster as a maximal set of density-connected points.
Density is defined by an area and # of points
Fig 8.9 J. Han and M. Kamber
1/18/2019
CSE572, CBS572: Data Mining by H. Liu

13. CSE572, CBS572: Data Mining by H. Liu
Defining density and connection
-neighborhood of an object x (core object) (M, P, O)
MinPts of objects within -neighborhood (say, 3)
directly density-reachable (Q from M, M from P)
Only core objects are mutually density reachable
density-reachable (Q from P, P not from Q) [asymmetric]
density-connected (O, R, S) [symmetric] for border points
What is the relationship between DR and DC?
Han & Kamber2001 Q M P S R O
1/18/2019
CSE572, CBS572: Data Mining by H. Liu

14. CSE572, CBS572: Data Mining by H. Liu
Clustering with DBSCAN
Search for clusters by checking the -neighborhood of each instance x
If the -neighborhood of x contains more than MinPts, create a new cluster with x as a core object
Iteratively collect directly density-reachable objects from these core object and merge density-reachable clusters
Terminate when no new point can be added to any cluster
DBSCAN is sensitive to the thresholds of density, but it is fast
Time complexity O(N log N) if a spatial index is used, O(N2) otherwise
1/18/2019
CSE572, CBS572: Data Mining by H. Liu

15. CSE572, CBS572: Data Mining by H. Liu
Neural networks
Self-organizing feature maps (SOMs)
Subspace clustering
Clique: if a k-dimensional unit space is dense, then so are its (k-1)-d subspaces
More will be discussed later
Semi-supervised clustering
1/18/2019
CSE572, CBS572: Data Mining by H. Liu

16. CSE572, CBS572: Data Mining by H. Liu
Challenges
Scalability
Dealing with different types of attributes
Clusters with arbitrary shapes
Automatically determining input parameters
Dealing with noise (outliers)
Order insensitivity of instances presented to learning
High dimensionality
Interpretability and usability
1/18/2019
CSE572, CBS572: Data Mining by H. Liu