1 Pattern Recognition: Statistical and Neural Lonnie C. Ludeman Lecture 28 Nov 9, 2005 Nanjing University of Science & Technology

2 Lecture 28 Topics 1.Review Clustering Methods 2. Agglomerative Hierarchical Clustering Example 3. Introduction to Fuzzy Sets 4. Fuzzy Partitions 5. Define Hard and soft clusters

3 K-Means Clustering Algorithm: Basic Procedure Randomly Select K cluster centers from Pattern Space Distribute set of patterns to the cluster center using minimum distance Compute new Cluster centers for each cluster Continue this process until the cluster centers do not change or a maximum number of iterations is reached. Review

4 Flow Diagram for K-Means Algorithm Review

5 Iterative Self Organizing Data Analysis Technique A ISODATA Algorithm Performs Clustering of unclassified quantitative data with an unknown number of clusters Similar to K-Means but with ability to merge and split clusters thus giving flexibility in number of clusters Review

6 Hierarchical Clustering Dendrogram Review

7 Example - Hierarchical Clustering Given the following data (a) Perform a Hierarchical Clustering of the data (b) Give the results for 3 clusters.

8 Solution: Plot of data vectors

9 Calculate distances between each pair of original data vectors Data sample x 5 and x 7 are the closest together thus we combine them to give the following for 9 clusters S 5 (10) S 7 (10) = S 5 (9) U

10 S 5 (9) S 5 (10) U S 7 (10) = S 5 (9) Combine Closest Clusters

11 Compute distances between new clusters Clusters S 8 (9) and S 9 (9) are the closest together thus we combine them to give the following for 8 clusters

12 S 5 (8) S 8 (8) S 8 (9) U S 9 (9) = S 8 (8) Combine Closest Clusters

13 Compute distances between new clusters Clusters S 5 (8) and S 8 (8) are the closest together thus we combine them to give the following for 7 clusters

14 Combine Closest Clusters S 5 (8) U S 8 (8) = S 5 (7) S 5 (8) S 8 (8) S 5 (7)

15 Compute distances between new clusters Clusters S 1 (7) and S 4 (7) are the closest together thus we combine them to give the following for 6 clusters

16 S 1 (7) U S 4 (7) = S 1 (6) Combine Closest Clusters S 1 (6) S 5 (7)

17 Continuing this process we see the following combinations of clusters at the given levels

18 Level 5

19 Level 4

20 Level 3

21 Level 2

22 Level 1

23 Dendogram for Given Example

24 (b) Using the dendrogram determine the results for just three clusters From the dendrogram at level 3 we see the following clusters S 5 (3) = { 5, 7, 8, 9, 10 ) S 2 (3) = { 2, 6 } S 1 (3) = { 1,4,3 } Cl 1 = { x 5, x 7, x 8, x 9, x 10 ) Cl 2 = { x 2, x 6 } Cl 3 = { x 1, x 4, x 3 } Answer

25 Introduction to Fuzzy Clustering K-means, Hierarchical, and ISODATA clustering algorithms are what we call “Hard Clustering”. The assignment of clusters is a partitioning of the data into mutually disjoint and exhaustive non empty sets. Fuzzy clustering is a relaxation of this property and provides another way of solving the clustering problem. Before we present the Fuzzy Clustering algorithm we first lay a background by defining Fuzzy sets.

26 Given a set S composed of pattern vectors as follows A proper subset of S is any nonempty collection of pattern vectors. Examples follow B = { x 2, x 4, x N } C = { x 4 } D = { x 1, x 3, x 5, x N-1 } S = { x 1, x 2,..., x N } A = { x 1, x 2 }

27 We can also specify subsets by using the characteristic function which is defined on the set S as follows for the subset A µ A ( x k ) = 1 if x k is in the subset A = 0 if x k is not in the subset A S = { x 1, x 2,..., x k,..., x N } Given the following Set µ A (.): [ µ A (x 1 ), µ A (x 2 ),..., µ A (x k ),..., (x N ) ] Characteristic Function for subset A

28 A = { x 1, x 2 } µ A (x k ): [ 1, 1, 0, 0, 0,..., 0 ] Examples

29 A = { x 1, x 2 } µ A (x k ): [ 1, 1, 0, 0, 0,..., 0 ] B = { x 2, x 4, x N } µ B (x k ): [ 0, 1, 0, 1, 0,..., 1 ] Examples

30 A = { x 1, x 2 } µ A (x k ): [ 1, 1, 0, 0, 0,..., 0 ] B = { x 2, x 4, x N } C = { x 4 } µ B (x k ): [ 0, 1, 0, 1, 0,..., 1 ] µ C (x k ): [ 0, 0, 0, 1, 0,..., 0 ] Examples

31 A = { x 1, x 2 } µ A (x k ): [ 1, 1, 0, 0, 0,..., 0 ] B = { x 2, x 4, x N } C = { x 4 } D = { x 1, x 3, x 5, x N-1 } µ B (x k ): [ 0, 1, 0, 1, 0,..., 1 ] µ C (x k ): [ 0, 0, 0, 1, 0,..., 0 ] µ D (x k ): [ 1, 0, 1, 0, 1,...,1, 0 ] Examples

32 Partitions of a set S Given a set S of N S n-dimensional pattern vectors: S = { x j ; j =1, 2,..., N S } A partition of S is a set of M subsets of S, S k, k=1, 2,..., M, that satisfy the following conditions. Note: Thus Clusters can be specified as a partition of the pattern space S.

33 1. S k ≠ Φ Not empty 2. S k ∩ S j ≠ Φ Pairwise disjoint where Φ is the Null Set SkSk ∩ k = 1 K 3. = S Exhaustive Properties of subsets of a Partition

34 Partition in terms of Characteristic Functions [ µ S (x 1 ), µ S (x 2 ),..., µ S (x k ),..., µ S (x N ) ] M M MM... Sum = 1 for each column µ S (x k ) = 0 or 1 j for all k and j µS :µS : 1 2 M µ S :

35 Cl 1 : [ ] Cl 2 : [ ] Cl 3 : [ ] Cl 1 Cl 2 Cl 3 Hard Partition x1 x2 x3 x4 x5 x6 x7x1 x2 x3 x4 x5 x6 x7

36 A Fuzzy Set can be defined by extending the concept of the characteristic function to allow positive values between and including 0 and 1 as follows A Fuzzy Subset F, of a set S, is defined by its membership function F: [ µ F (x 1 ), µ F (x 2 ),..., µ F (x k ),..., µ F (x N ) ] 0 < µ F (x k ) < 1 = = where x k is from S and Given a set S = { x 1, x 2,..., x N }

37 S Function defined on S

38 Example: Define a Fuzzy set A by the following membership function Or equivalently

39 A Fuzzy Partition F, of a set S, is defined by its membership functions for the fuzzy sets F k : k =1, 2,..., K ) ] Fuzzy Partition

40 where Each value bounded by 0 and 1 Sum of each columns values =1 Sum of each row less than n

41 x1x1 x3x3 x5x5 x4x4 x2x2 x6x6 P 1 = {x 1, x 2, x 3 } P1P1 P2P2 P 2 = {x 4, x 5, x 6 } “Hard” or Crisp Clusters Set Descriptions

42 x1x1 x3x3 x5x5 x4x4 x2x2 x6x6 “Soft” or Fuzzy Clusters Membership Functions Pattern Vectors

43 Hard or Crisp Partition Soft or Fuzzy Partition

44 Membership Functions for Fuzzy Clusters Domain Pattern Vectors Membership Function for F 1 Membership Function for F 2

45 Note: Sum of columns = 1

46 “Fuzzy” Kitten

47 Lecture 28 Topics 1.Reviewed Clustering Methods 2. Gave an Example using Agglomerative Hierarchical Clustering 3. Introduced Fuzzy Sets 4. Described Crisp and Fuzzy Partitions 5. Defined Hard and soft clusters

48 End of Lecture 28