1. Unsupervised Classification
CHAPTER 14
CLASSIFICATION
Clustering and
Unsupervised Classification
A. Dermanis

2. Clustering = dividing of N pixels into K classes ω1, ω2, …, ωK
scatter matrix of class ωi :
mean of class ωi : 
xi
Si = (x – mi)(x – mi)T
mi = x 
xi 1 ni
covariance matrix of class ωi :
Ci = Si 1 ni
total scatter matrix:
global mean
 
i xi
ST = (x – mi)(x – mi)T
m = x 1 N
 
i xi
total covariance matrix:
CT = ST 1 N
A. Dermanis

3.      Sin = Si = (x – mi)(x – mi)T Sex = ni (mi – m)(mi – m)T
Clustering criteria
overall compactness of the clusters  internal scatter matrix
 
i xi
Sin = Si = (x – mi)(x – mi)T  i
degree of distinction between the clusters  external scatter matrix
Sex = ni (mi – m)(mi – m)T  i
ST = Sin + Sex = constant
Optimal algorithm: Sin = min and Sex = max (simultaneously)
Problem: How many clusters ? (K = ?)
Extreme choice: K = N (each pixel a different class) k = {xk}
mk = xk, Sk = 0, Sin = Sk = 0 = min, Sex = ST =max  k
Extreme choice: K = 1 (all pixels in a single class) Sin = ST, Sex = 0
A. Dermanis

4. Hierarchical Clustering1 2 3 4 5 6 A
Agglomerative clustering:
Unifying at each step
the two closest clusters B
AGGLOMERATIVE
DIVISIVE C
Divisive clustering :
Dividing at each step
the most disperse cluster
into two new clusters D E F
Needed:
Unification criteria.
Division criteria and procedures.
A. Dermanis

5. Hierarchical Clustering1 2 3 4 5 6 1 2 3 4 5 6 A B
AGGLOMERATIVE
DIVISIVE C D E F A B C D E F
A. Dermanis

6. Distance between two clusters (alternatives):
mean distance:
minimum distance:
maximum distance:
Used in agglomerative and divisive clustering
A. Dermanis

7. The K-means or migrating means algorithm1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
A. Dermanis

8. The K-means or migrating means algorithm1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Step 0:
Selection of K = 3 pixels
as initial positions of means
A. Dermanis

9. The K-means or migrating means algorithm1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Step 1:
Assignment each pixels to the cluster
of its closest mean
Calculation of the new means
for each cluster
A. Dermanis

10. The K-means or migrating means algorithm1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Step 2:
Assignment each pixels to the cluster
of its closest mean
Calculation of the new means
for each cluster
A. Dermanis

11. The K-means or migrating means algorithm1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Step 3:
Assignment each pixels to the cluster
of its closest mean
Calculation of the new means
for each cluster
A. Dermanis

12. The K-means or migrating means algorithm1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Step 4:
Assignment each pixels to the cluster
of its closest mean
All pixels remain in the same cluster.
Means remain the same.
Termination of the algorithm !
A. Dermanis

13. A variant of the K means algorithm.
The Isodata Algorithm
A variant of the K means algorithm.
In each step one of 3 additional procedures can be used:
1. Cluster ELIMINATION
Eliminate clusters
with very few pixels
2. Cluster UNIFICATION
Unify pairs of clusters
Very close to each other
3. Cluster DIVISION
Divide large clusters
which are elongated
Into two clusters
A. Dermanis

14. The Isodata Algorithm 1. Cluster ELIMINATION Eliminate clusters
with very few pixels
A. Dermanis

15. The Isodata Algorithm 2. Cluster UNIFICATION Unify pairs of clusters
Very close to each other
A. Dermanis

16. The Isodata Algorithm 3. Cluster DIVISION Divide large clusters
which are elongated
Into two clusters
A. Dermanis

17. m2+kσ2 m2 m2–kσ2 m1 The Isodata Algorithm The unification process
The division process
m2+kσ2
m2–kσ2 m2 m1
A. Dermanis

18. Examples of classifiction using the K-mean algorithm
K-means: 3 classes
K-means: 5 classes
K-means: 7 classes
K-means: 9 classes
A. Dermanis

19. Examples of classifiction using the ISODATA algorithm
ISODATA : 3 classes
ISODATA : 5 classes
ISODATA : 7 classes
ISODATA : 9 classes
A. Dermanis