1. Agglomerative clustering (AC)
Clustering algorithms: Part 2c
Agglomerative clustering (AC)
Pasi Fränti
Speech & Image Processing Unit
School of Computing
University of Eastern Finland
Joensuu, FINLAND

2. Agglomerative clustering Categorization by cost function
Single link
Minimize distance of nearest vectors
Complete link
Minimize distance of two furthest vectors
Ward’s method
Minimize mean square error
In Vector Quantization, known as Pairwise Nearest Neighbor (PNN) method
We focus on this

3. Pseudo code

4. Pseudo code PNN(X, M) → C, P FOR i←1 TO N DO p[i]←i; c[i]←x[i]; O(N)
REPEAT
a,b ← FindSmallestMergeCost();
MergeClusters(a,b);
m←m-1;
UNTIL m=M;
O(N)
O(N2)
N times
T(N) = O(N3)

5. Ward’s method [Ward 1963: Journal of American Statistical Association]
Merge cost:
Local optimization strategy:
Nearest neighbor search:
Find the cluster pair to be merged
Update of NN pointers

6. Example of distance calculations

7. Example of the overall process.
M=50
M=16
M=15
M=16
M=15

8. Detailed example of the process

9. Example - 25 Clusters
MSE ≈ 1.01*109

10. Example - 24 Clusters
MSE ≈ 1.03*109

11. Example - 23 Clusters
MSE ≈ 1.06*109

12. Example - 22 Clusters
MSE ≈ 1.09*109

13. Example - 21 Clusters
MSE ≈ 1.12*109

14. Example - 20 Clusters
MSE ≈ 1.16*109

15. Example - 19 Clusters
MSE ≈ 1.19*109

16. Example - 18 Clusters
MSE ≈ 1.23*109

17. Example - 17 Clusters
MSE ≈ 1.26*109

18. Example - 16 Clusters
MSE ≈ 1.30*109

19. Example - 15 Clusters
MSE ≈ 1.34*109

20. Storing distance matrix
Maintain the distance matrix and update rows for the changed cluster only!
Number of distance calculations reduces from O(N2) to O(N) for each step.
Search of the minimum pair still requires O(N2) time  still O(N3) in total.
It also requires O(N2) memory.

21. Heap structure for fast search [Kurita 1991: Pattern Recognition]
Search reduces O(N2)  O(logN).
In total: O(N2 logN)

22. Store nearest neighbor (NN) pointers [Fränti et al. , 2000: IEEE Trans
Store nearest neighbor (NN) pointers [Fränti et al., 2000: IEEE Trans. Image Processing]
Time complexity reduces to O(N 3)  Ω (N 2)

23. Pseudo code PNN(X, M) → C, P FOR i←1 TO N DO p[i]←i; c[i]←x[i]; O(N)
NN[i]← FindNearestCluster(i);
REPEAT
a ← SmallestMergeCost(NN);
b ← NN[i];
MergeClusters(C,P,NN,a,b,);
UpdatePointers(C,NN);
UNTIL m=M;
O(N)
O(N2)
O(N)
O(N)

24. Example with NN pointers [Virmajoki 2004: Pairwise Nearest Neighbor Method Revisited ]

25. Example Step 1

26. Example Step 2

27. Example Step 3

28. Example Step 4

29. Example Final

30. Time complexities of the variants

31. Number of neighbors (τ)

32. Processing time comparison
With NN pointers

33. Algorithm: Lazy-PNN
T. Kaukoranta, P. Fränti and O. Nevalainen, "Vector quantization by lazy pairwise nearest neighbor method", Optical Engineering, 38 (11), , November 1999

34. Monotony property of merge cost [Kaukoranta et al
Monotony property of merge cost [Kaukoranta et al., Optical Engineering, 1999]
Merge costs values are monotonically increasing:
d(Sa, Sb)  d(Sa, Sc)  d(Sb, Sc) 
d(Sa, Sc)  d(Sa+b, Sc)

35. Additional data structure
Lazy variant of the PNN
Store merge costs in heap.
Update merge cost value only when it appears at top of the heap.
Processing time reduces about 35%.
Method
Ref.
Time complexity
Additional data structure
Space compl.
Trivial PNN
[10]
O(d∙N3) -
O(N)
Distance matrix
[6]
O(d∙N2+ N3)
O(N2)
Kurita’s method
[5]
O(d∙N2+ N2∙logN)
Dist. matrix + heap
-PNN
[1]
O(d∙N2)
NN-table
Lazy-PNN
[4]

36. Combining PNN and K-means

37. Algorithm: Iterative shrinking
P. Fränti and O. Virmajoki “Iterative shrinking method for clustering problems“ Pattern Recognition, 39 (5), , May 2006.

38. Agglomerative clustering based on merging

39. Agglomeration based on cluster removal [Fränti and Virmajoki, Pattern Recognition, 2006]

40. Merge versus removal

41. Pseudo code of iterative shrinking (IS)

42. Cluster removal in practice
Find secondary cluster:
Calculate removal cost for every vector:

43. Partition updates

44. Complexity analysis Number of vectors per cluster:
If we iterate until M=1:
Adding the processing time per vector:

45. Algorithm: PNN with kNN-graph
P. Fränti, O. Virmajoki and V. Hautamäki, "Fast agglomerative clustering using a k-nearest neighbor graph". IEEE Trans. on Pattern Analysis and Machine Intelligence, 28 (11), , November 2006

46. Agglomerative clustering with kNN graph

47. Example of 2NN graph

48. Example of 4NN graph

49. Graph using double linked lists

50. Merging a and b

51. Effect on calculations number of steps
STAGE
Theoretical
Observed
-PNN
Single link
Double link
Single link
Double link
Find pair N 1
8 357 3
Merge
k2 + logN
k2 + k + logN
8 367
200
305
Remove last
k + logN
LogN
8 349
102 45
Find neighbors kN k
41 769
204
Update costs
N (1+)
 + /klogN
48 538
198
187
TOTAL
O(N2)
O(kN2)
O(NlogN)
81 970
42 274
746

52. Processing time as function of k (number of neighbors in graph)

53. Time distortion comparison
-PNN (229 s)
Trivial-PNN (>9999 s)
Graph-PNN (1)
MSE = 5.36
Graph-PNN (2)
Graph created by MSP
Graph created by D-n-C

54. Conclusions Simple to implement, good clustering quality
Straightforward algorithm slow O(N3)
Fast exact (yet simple) algorithm O(τN2)
Beyond this possible:
O(τ∙N∙logN) complexity
Complicated graph data structure
Compromizes the exactness of the merge

55. Literature
P. Fränti, T. Kaukoranta, D.-F. Shen and K.-S. Chang, "Fast and memory efficient implementation of the exact PNN", IEEE Trans. on Image Processing, 9 (5), , May 2000.
P. Fränti, O. Virmajoki and V. Hautamäki, "Fast agglomerative clustering using a k-nearest neighbor graph". IEEE Trans. on Pattern Analysis and Machine Intelligence, 28 (11), , November 2006.
P. Fränti and O. Virmajoki, "Iterative shrinking method for clustering problems", Pattern Recognition, 39 (5), , May 2006.
T. Kaukoranta, P. Fränti and O. Nevalainen, "Vector quantization by lazy pairwise nearest neighbor method", Optical Engineering, 38 (11), , November 1999.
T. Kurita, "An efficient agglomerative clustering algorithm using a heap", Pattern Recognition 24 (3) (1991)

56. Literature
J. Shanbehzadeh and P.O. Ogunbona, "On the computational complexity of the LBG and PNN algorithms". IEEE Transactions on Image Processing 6 (4), 614‑616, April 1997.
O. Virmajoki, P. Fränti and T. Kaukoranta, "Practical methods for speeding-up the pairwise nearest neighbor method ", Optical Engineering, 40 (11), , November 2001.
O. Virmajoki and P. Fränti, "Fast pairwise nearest neighbor based algorithm for multilevel thresholding", Journal of Electronic Imaging, 12 (4), , October 2003.
O. Virmajoki, Pairwise Nearest Neighbor Method Revisited, PhD thesis, Computer Science, University of Joensuu, 2004.
J.H. Ward, Hierarchical grouping to optimize an objective function, J. Amer. Statist.Assoc. 58 (1963)