Iterative Optimization and Simplification of Hierarchical Clusterings Doug Fisher Department of Computer Science, Vanderbilt University Journal of Artificial Intelligence Research, 4 (1996) Presented by: Biyu Liang

2 Outline Introduction Generating Initial Hierarchical Clustering Iterative Optimization Methods and Comparison Simplification of Hierarchical Clustering Conclusion

3 Introduction Clustering is a process of unsupervised learning, which groups objects into clusters. Major Clustering Methods  Partitioning  Hierarchical  Density-based  Grid-based  Model-based

4 Introduction (Continued) Clustering systems differ in objective function control strategy Usually a search strategy cannot be both computationally inexpensive and give any guarantee about the quality.

5 Introduction (Continued)  This paper discusses the use of iterative optimization and simplification to construct clusters that satisfy both conditions: High quality Computationally inexpensive  The suggested method involves 3 steps: Constructing a initial clustering inexpensively Iterative optimization to improve the clustering Retrospective simplification of the clustering

6 Outline Introduction Generating Initial Hierarchical Clustering Iterative Optimization Methods and Experiments Simplification of Hierarchical Clustering Conclusion

7 Category Utility CU(C K ) = P(C k )  i  j [P(A i = V ij |C K ) 2 -P(A i = V ij ) 2 ] PU({C 1, C 2, … C N }) =  k CU(C K )/N Where an observation is a vector of V ij along attributes(or variables) A i This measure rewards clusters C k, that increases the predictability of V ij within C k (i.e. P(A i =V ij |C k )) relative to their predictability in the population as a whole (i.e. P(A i = V ij ))

8 P(Color=gre|C1)

9 Hierarchical Sorting Given an observation and current partition, evaluate the quality of the clusterings that result from  Placing the observation in each of the existing clusters  Creating a new cluster that only covers the new observation Select the option that yields the highest quality score (PU)

10

11 Outline Introduction Generating Initial Hierarchical Clustering Iterative Optimization Methods and Comparison Simplification of Hierarchical Clustering Conclusion

12 Iterative Optimization Methods Reorder-resort (Cluster/2): seed selection, reordering, and re-clustering. Iterative redistribution of single observation: moving single observation one by one. Iterative hierarchical redistribution: moving clusters together with its sub-tree.

13 Reorder-resort (k-mean) k random seeds are selected, and k clusters are growing around these attractors the centroids of the clusters are picked as new seeds, new clusters are growing The process iterates until there is no further improvement in the quality of generated clustering

14 Reorder-resort (k-mean) con’t Ordering data to make consecutive observations dissimilar leads to good clusterings Extracting biased “dissimilarity” ordering from the hierarchical clustering Initial sorting, extraction dissimilarity ordering, re-clustering

15 Iterative Redistribution of Single Observations Moves single observations from cluster to cluster A cluster contains only one observation is removed and its single observation is resorted Iterate until two consecutive iterations yield the same clustering

16 The ISODATA algorithm determines a target cluster for each observation but does not move the cluster until targets for all observations have been determined A sequential version that moves each observation as its target is identified through sorting Single Observation Redistribution Variations

17 Iterative Hierarchical Redistribution Takes large steps in the search for a better clustering Remove and resorts sub-tree instead of single observation Requires update variable value counts of ancestor clusters and host cluster

18 Scheme Given an existing hierarchical clustering, a recursive loop examines sibling clusters in the hierarchy in a depth first fashion. An inner, iterative loop reclassifies each sibling based on the objective function. And repeats until two consecutive iterations lead to the same set of siblings.

19 (Continued) The recursive loop then turns its attention to the children of each of these remaining siblings. Finally the leaves will be reached and resorted. The recursive loop will be applied several times until there are no changes that occur from one pass to the next.

20

21 Main findings from the experiments Hierarchical redistribution achieves the highest mean PU scores in most cases Reordering and re-clustering comes closest to hierarchical redistribution’s performance in all cases Single-observation redistribution modestly improves an initial sort, and is substantially worse than the other two optimization methods

22 Outline Introduction Generating Initial Hierarchical Clustering Iterative Optimization Methods and Comparison Simplification of Hierarchical Clustering Conclusion

23 Simplifying Hierarchical Clustering Simplify hierarchical clustering and minimize classification cost Minimize Error Rate Validation set to identify the frontier of clusters for prediction of each variable Node lies below the frontier of every variable would be pruned

24 Validation For each variable, A i, the objects from the validation set are each classified through the hierarchical clustering with the value of variable A i “masked” for purposes of classification. At each cluster encountered during classification, prediction correct if the observation’s value for A i is equal to the most frequent value for A i at the cluster. A Count of all correct predictions for each variable at a cluster is maintained. A preferred frontier for each variable is identified that maximizes the number of correct counts for the variable.

25

26 Concluding Remarks There are three phases in searching the space of hierarchical clusterings:  Inexpensive generation of an initial clustering  Iterative optimization for clusterings  Retrospective simplification of generated clusterings Experiments found that the new method, hierarchical redistribution optimization works well

Thanks! Question?

28 Final Exam Questions The main idea in this paper is to construct clusterings which satisfy two conditions.  Name the conditions (p.5)  name the two steps to satisfy the conditions Discribe the three iterative methods for clustering optimization (p.12-20) The cluster is better when the relative CU score is a) big, b) small, c) equal to 0 (p.7) Which sorting method is better? a) random sorting, b) similarity sorting (p.14)