1. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 1 CLUSTERS Prof. George Papadourakis, Ph.D.

2. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 2 Creating Clusters (1/4)  Methods for Creating Data Clusters: Methods for finding the categories and subcategories that form the data of every problem.  Clustering: Tool for data analysis, organizes the patterns into clusters or categories, patterns that belong to a cluster (resemblance between them)  Clustering results: Classification of new data, control for data homogeneity, data compression  Scheme 5.1: Data Sets in two-dimensional space  Two Clusters – There isn’t just one unique clustering technique for all cases  Human perception: The best clustering technique (2-d, 3-d space)  Higher Dimensions not well  Plenty of methods – concerning specific applications

3. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 3 Creating Clusters (2/4) A) Compact and well separable B) Uneven clusters C) Non compact D) Contiguous ClustersL) AlongateZ) Non linearly separable H) Clusters within clusters I) Inexistence of clusters

4. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 4  Clustering Algorithm functions even with random data Creating Clusters (3/4) Scheme 5.2: Two or Four clusters?

5. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 5  What do points represents, what’s the meaning of cluster in the application  Instead of finding a natural data structure, impose arbitrary structure  Skepticism for results – Increase of problem comprehension?  Yes! Clusters correctly chosen  If the result is in contrast with intuition, the problem has not been fully understood Creating Clusters (4/4)

6. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 6  Medicine: Disease and Symptoms Clustering  Psychiatry: Diagnosis of Symptom Sets - Paranoia, Schizophrenia  Archeology: Classifying stone tools, grave goods  Image Analysis: Pixels Clusters with similar features (color, texture) Applications

7. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 7  Consists of similar patterns  Gap between patterns of the same cluster smaller to that between patterns from different clusters  Clusters form connected areas with rather high density, separable from other clusters with areas of low density  Initial design defines what a cluster means for the application  Clustering Methodologies are based on ideal clustering structures  Most of the clustering algorithms always place the two nearest patterns at the same cluster Ideal Clusters

8. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 8  Hierarchical  Partitional Clustering Methodologies

9. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 9  Hierarchical data structure  Trees  Agglomerate  Initially: N clusters – one for each pattern  Intermediately: clusters merged  Finally: one cluster containing all patterns  Divisive  Initially: one cluster which containing all the patterns  Intermediately: clusters split  Finally: N clusters – one for each pattern  Applications: Classifying plants and animals Hierarchical

10. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 10  Goal: creating a set of clusters which separate the data in similar modules  Close Patterns are considered to be similar  Pre-defined number of categories  Criterion functions are used, such as minimize squares functions, density valuators and nearest neighbors.  Applications: Problem categories don’t form distinct clusters (they overlap)  Distinction between clustering methods and clustering algorithms  The same clustering method can be implemented differently, having as a result the creation of many different clustering algorithms  Forgy’s, Isodata are based on methods which minimize the square error Partitional

11. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 11  Pet shop veterinarian, two big clusters: cats and dogs  If two patterns belong to the same cluster in one level, then they will belong to the same cluster in a higher level as well  Patterns 1, 2 Hierarchical Clustering (1/2) {1,2,3,4,5} {1,2,3} {1,2} {4,5 } {1}{2}{5}{4}{3} Level 4 Level 3 Level 2 Level 1

12. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 12 1. Register each one of the N patterns to a single cluster 2. Find clusters with greater resemblance and merge them to one cluster 3. Repeat step 2 until all patterns belong to the same cluster  Using different methods to define the resemblance among the clusters, different algorithms arise.  Popular distance metrics: Euclidean and Hippodamian Hierarchical Clustering (2/2)

13. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 13  Nearest neighbor method or minimum method  Distance between clusters: minimum distance between two patterns of different clusters Single Linkage

14. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 14  Distance between clusters C i and C j is defined by the equation: Where d(x,y) a distance function between the patterns x,y  5 clusters and each one corresponds to one pattern Single Linkage algorithm (1/2) 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

15. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 15  Initial Distance table Single Linkage Algorithm (2/2)  {1,2}, {3}, {4}, {5}

16. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 16 Single Linkage Scheme 5.5: Hierarchical clustering using the simple linkage algorithm

17. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 17  Also known as maximum method or farthest neighbor.  Distance between clusters: maximum between two patterns which belong to different clusters  So the distance between the clusters C i and C j is defined by the equation:  Where d(x,y) is a distance function between the patterns x,y  5 clusters and each one corresponds to one pattern  Nearest Clusters {1} and {2} are merged Complete Linkage Algorithms (1/4)

18. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 18 Complete Linkage Algorithms (2/4)  Maximum distance between clusters {1,2} {3}: 11.7  The rest of the columns are just sustained and copied  Minimum distance between clusters 8: {4}, {5} Second distance table of Complete Linkage {1,2}, {3}, {4,5}

19. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 19  Maximum distance between clusters {1,2} {3,4}: 21.5  The rest of the columns are just sustained and copied  Minimum distance between clusters 9.8: {4,5}, {3} {1,2} {3,4,5} Complete Linkage Algorithms (3/4) Scheme 5.10: Third distance table Complete Linkage

20. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 20 ¤ Cutting tree at 10 2 clusters. ¤ Cutting tree at 5 4 clusters ¤ Where do we need to cut the tree? ¤ Heuristic Method: μεγάλο κάθετο κενό ¤ Big life time: the difference between the distance during which the cluster merges from the distance it was created. ¤ Cutting tree at 10 2 clusters Complete Linkage Algorithms (4/4)

21. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 21  Different way of defining the pattern resemblance  Different clustering for the same data  Single linkage algorithm comes short of the chain phenomenon:  Where distance patterns are placed at the same cluster because they have a common neighbor cluster. Comparing Algorithms

22. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 22  The graph of the Single Linkage Algorithm is a minimum spanning tree which is created by adding the nearest value between the two clusters. Duda and Hart Examples

23. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 23 Comparing Algorithms (1/4) Scheme 5.7: Examples of Hierarchical clustering. a) Three data sets b) Results of Single Linkage Algorithm c) Results of Complete Linkage Algorithm

24. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 24  α) Clusters are compact and well separable  Single Linkage Algorithm find easy the separable clusters.  β) Patterns create a bridge among clusters  Result: Creation of unexpected clusters, a big elongate and a small compact.  Chain phenomenon is obvious.  Since results of a clustering are particularly sensitive to noise and small cluster deviations in space then the chain phenomenon is a disadvantage.  If the clusters are elongate then chain creation might be an advantage  γ) example Comparing Algorithms (2/4)

25. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 25  Graphical representation of complete linkage algorithm is the creation of a graph whose edges connect all the patterns or nodes in a cluster.  Each cluster consists a complete sub-graph.  Distance within the clusters is defined by the distance of the most distant patterns of the two clusters.  When neighbor patterns connect the graph changes by added edges between all the patterns of each cluster.  Cluster Diameter: maximum distance between patterns in the cluster  Cluster Distance: diameter of connecting two clusters. Comparing Algorithms (3/4)

26. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 26  Each repeat of the Complete Linkage Algorithm steps increases the less possible the cluster diameter.  Advantage when the real clusters are compact and similar in size,  Examples a) and b)  On the contrary, when the real clusters don’t have these features, the results are arbitrary  Examble c)  Other functions as Average, and Centroid.  Average Linkage Algorithm  Ccentroid Linkage Algorithm  Ward Method: Hierarchical Algorithm which uses analysis of variance techniques Comparing Algorithms (4/4)

27. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 27  The single linkage algorithm creates elongate clusters while the complete linkage algorithm creates more compact clusters.  The Average Linkage Algorithm is an effort of compromising the two algorithms, single and complete linkage.  Distance between clusters: mean distance of patterns of each different cluster.  5 Clusters each one corresponds to one pattern  If C i is a cluster with n i members and C j is a cluster with n j members then the distance between the two clusters is defined as”  Nearest clusters {1} and {2} are merged Average Linkage Algorithm (1/3)

28. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 28  Initial distance matrix: d(1,3) = 11.7 d(2,3)=8.1 d({1,2},3)=9.9  The rest of the columns are just sustained and copied  Minimum distance between clusters 8: {4}, {5} Average Linkage Algorithm (2/3) {1,2}, {3}, {4,5}

29. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 29  Second distance matrix: d({1,2},4) = 18.0 d({1,2},5)=19.7 d({1,2},{4,5})= 18.9  Minimum cluster distance 9.8: {4,5}, {3} {1,2} {3,4,5}  The result is exactly the same as with the complete linkage algorithm.  Average Linkage Algorithm is effective with compact patterns.  Moreover, it can be used with elongate clusters. Average Linkage Algorithm (3/3)

30. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 30  Also named as the Mimimum Variance Method.  Initially all the patterns are individual clusters.  At every repeat the pair which generates the minimum square error is merged.  m patterns where  Pattern square error x i  Cluster Square Error  The total variation is defined as: Ward Method (1/4)

31. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 31  5 Clusters, each consists of one patter  Square Error 0  10 different ways of merging 5 clusters  {1} [4,4] {2} [8,4] μ = [6,4]  E{3}=0 E{4}=0 E{5}=0  E = 8 + 0 + 0 + 0 =8  Clusters {1} and {2} merged Ward Method (2/4)

32. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 32  Clusters {1,2} {3} {4} {5}  Smaller E = 40  {4,5} merged  {1,2} {3} {4,5} Ward Method (3/4)

33. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 33 Ward Method (4/4)  Smaller E = 94  Clusters {3} and {4,5} merged  Generally, the Ward Method is considered very effective, while it tends to create small size clusters.  Hierarchical Algorithms are effective for a small number of patterns (<20)

34. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 34 SEPARATING CLUSTERING

35. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 35  Separating Clustering Techniques: Ν patterns  k clusters Ν>>k Usually k is defined by the user  Separating techniques: they allow patterns to move from cluster to cluster (not the hierarchical techniques) An initially bad separation can be corrected.  Separation is implemented with maximization or minimization of the criteria function.  Square error very popular  Separation Clustering Techniques ομαδοποίησης:  Forgy Algorithm  K-means  Isodata Introduction

36. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 36  Simple  Input: Ν patterns, k clusters, k representative patterns (random or predefined) Pseudocode 1.Initial setting of k clusters and k representative patterns 2.For each pattern find the nearest representative pattern and register it to the corresponding cluster 3.Calculate the new representative pattern for each cluster which is the average of the cluster patterns 4.If there are changes in patterns registration to clusters, that is new representative patterns are not the same as before, return to step 2. Forgy Algorithm (1/5)

37. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 37 Forgy Algorithm (2/5)  Example 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

38. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 38  Step 3: Calculation of representative patterns  Cluster A : [4,4]  Cluster B: [17.75,7]  There is a change in the pattern registration, return to step 2. Second repeat of the algorithm  Step 2: Calculation of the nearest cluster for each pattern. Forgy Algorithm (3/5)

39. Technological Educational Institute Of Crete Department Of Applied Informatics and Multimedia Intelligent Systems Laboratory 39  Step 3: Calculation of new representative patterns  Cluster A: [6,4]  Cluster B: [21,8]  There is a change in pattern registration, return to Step 2. Third repeat of the algorithm  Step 2: Calculation of the nearest cluster for each pattern.  There is no change to the pattern registration, algorithm terminated. Forgy Algorithm (4/5)