Characterization of Linkage-Based Algorithms Margareta Ackerman Joint work with Shai Ben-David and David Loker University of Waterloo To appear in COLT 2010

There are a wide variety of clustering algorithms, which often produce very different clusterings. How can we distinguish between clustering algorithms? How should a user decide which algorithm to use for a given application? Motivation

We propose a framework that lets a user utilize prior knowledge to select an algorithm Identify properties that distinguish between different clustering paradigms The properties should be: 1) Intuitive and “user-friendly” 2) Useful for classifying clustering algorithms Our approach for clustering algorithm selection

Kleinberg proposes abstract properties (“Axioms”) of clustering functions (NIPS, 2002) Bosagh Zadeh and Ben-David provide a set of properties that characterize single linkage clustering (UAI, 2009) Previous work

Propose a set of intuitive properties that uniquely indentify linkage-based clustering algorithms Construct a taxonomy of clustering algorithms based on the properties Our contributions

Define linkage-based clustering Our new clustering properties Main result Sketch of proof A taxonomy of common clustering algorithms using clustering properties Conclusions Outline

For a finite domain set X, a dissimilarity function d over the members of X. A Clustering Function F maps Input: (X,d) and k>0to Output: a k -partition (clustering) of X Formal setup

Start with the clustering of singletons Merge the closest pair of clusters Repeat until only k clusters remain. Ex. Single linkage, average linkage, complete linkage Informally, a linkage function is an extension of the between-point distance that applies to subsets of the domain. The choice of the linkage function distinguishes between different linkage-based algorithms. ? Linkage-based algorithm: An informal definition

Define linkage-based clustering Our new clustering properties Main result Sketch of proof A taxonomy of common clustering algorithms using our properties Conclusions Outline

A clustering C is a refinement of clustering C’ if every cluster in C’ is a union of some clusters in C. A clustering function is hierarchical if for and every F(X,d,k’) is a refinement of F(X,d,k). Hierarchical clustering

F is local if for any C Locality

Many clustering algorithms are local:  K-means  K-median  Single-linkage  Average-linkage  Complete-linkage Notably, some clustering algorithms fail locality:  Ratio cut  Normalized cut Which paradigms satisfy locality ?

If d’ equals d, except for increasing between-cluster distances, then F(X,d,k)=F(X,d’,k) for all d, X, and k. dd’ F(X,d,3)F(X,d’,3) Outer Consistency Based on Kleinberg, 2002.

K-means K-median Single-linkage Average-linkage Complete-linkage Not all clustering algorithms are outer-consistent  Ratio cut  Normalized cut Which paradigms satisfy outer-consistency?

Extended Richness

F satisfies extended richness if for any set of domains there is a d over that extends each of the so that Extended Richness

K-means K-median Single-linkage Average-linkage Complete-linkage Ratio cut Normalized cut Many clustering algorithms satisfy extended richness

Define linkage-based clustering Our new clustering properties Main result Sketch of proof A taxonomy of common clustering algorithms using our properties Conclusions Outline

Theorem: A clustering function is Linkage Based if and only if it is Hierarchical and it satisfies Outer Consistency, Locality and Extended Richness. Our main result

Every Linkage Based clustering function is Hierarchical, Local, Outer-Consistent, and satisfies Extended Richness. The proof is quite straight-forward. Easy direction of proof

If F is Hierarchical and it satisfies Outer Consistency, Locality and Extended-Richness then F is Linkage-Based. To prove this direction we first need to formalize linkage-based clustering, by formally defining what is a linkage function. Interesting direction of proof

A linkage function is a function l :{ : d is a distance function over } that satisfies the following: What do we expect from linkage function? 1) Representation independent: Doesn’t change if we re-label the data 2) Monotonic: if we increase edges that go between and, then l doesn’t decrease. 3) Any pair of clusters can be made arbitrarily distant: By increasing edges that go between and, we can make l reach any value in the range of l.

Recall direction: If F is a hierarchical function that satisfies outer- consistency, locality, and extended richness then F is linkage-based. Goal: Define a linkage function l so that the linkage based clustering based on l outputs F(X,d,k) (for every X, d and k ). Sketch of proof

Define an operator < F : (A,B,d 1 ) < F (C,D,d 2 ) if when we run F on, where d extends d 1 and d 2, A and B are merged before C and D. Sketch of proof (continued…)

Define an operator < F : (A,B,d 1 ) < F (C,D,d 2 ) if when we run F on, where d extends d 1 and d 2, A and B are merged before C and D.

Sketch of proof (continued…) Prove that < F can be extended to a partial ordering Use the ordering to define l Define an operator < F : (A,B,d 1 ) < F (C,D,d 2 ) if when we run F on, where d extends d 1 and d 2, A and B are merged before C and D.

Sketch of proof continue: Show that < F is a partial ordering We show that < F is cycle-free. Lemma: Given a function F that is hierarchical, local, outer-consistent and satisfies extended richness, there are no so that and

By the above Lemma, the transitive closure of < F is a partial ordering. R This implies that there exists an order preserving function l that maps pairs of data sets to R. It can be shown that l satisfies the properties of a linkage function. Sketch of proof (continued…)

Define linkage-based clustering Our new clustering properties Main result Sketch of proof A taxonomy of common clustering algorithms using our properties Conclusions Outline

LocalOuter Con. Inner Con. Heirar- chical Path Dist. Order Inv. Extent. Rich. Scale Inv. Iso. Inv. Single linkage Average linkage   Complete linkage   K-means  K-median  Min-Sum  Ratio-cut  Normalized- cut  Taxonomy of clustering algorithms

LocalOuter Con. Inner Con. Heirar- chical Path Dist. Order Inv. Extent. Rich. Scale Inv. Iso. Inv. Single linkage Average linkage   Complete linkage   K-means  K-median  Min-Sum  Ratio-cut  Normalized- cut  Characterization of Linkage-Based Algorithms

LocalOuter Con. Inner Con. Heirar- chical Path Dist. Order Inv. Extent. Rich. Scale Inv. Iso. Inv. Single linkage Average linkage   Complete linkage   K-means  K-median  Min-Sum  Ratio-cut  Normalized- cut  Characterization of Single-Linkage By Bosagh Zadeh and Ben-David (UAI, 09)

LocalOuter Con. Inner Con. Heirar- chical Path Dist. Order Inv. Extent. Rich. Scale Inv. Iso. Inv. Single linkage Average linkage   Complete linkage   K-means  K-median  Min-Sum  Ratio-cut  Normalized- cut  Distinguishing among Linkage-Based Algorithms

A function F is order invariant if for all d and d’ where for all points p,q,r,s d(p,q)< d(r,s) iff d’(p,q)< d’(r,s), we have that F(X,d) = F(X,d’). LocalOuter Con. Inner Con. Heirar- chical Path Dist. Order Inv. Extent. Rich. Scale Inv. Iso. Inv. Single linkage Average linkage   Complete linkage   Distinguishing among Linkage-Based Algorithms

LocalOuter Con. Inner Con. Heirar- chical Path Dist. Order Inv. Extent. Rich. Scale Inv. Iso. Inv. Single linkage Average linkage   Complete linkage   K-means  K-median  Min-Sum  Ratio-cut  Normalized- cut  When “Natural” properties are not satisfied

LocalOuter Con. Inner Con. Heirar- chical Path Dist. Order Inv. Extent. Rich. Scale Inv. Iso. Inv. Single linkage Average linkage   Complete linkage   K-means  K-median  Min-Sum  Ratio-cut  Normalized- cut  PropertiesAxioms

Using this framework, clustering users can utilize prior knowledge to determine which properties make sense for their application The goal is to construct a property-based taxonomy for many useful clustering algorithms Using this approach, a user will be able to find a suitable algorithm without the overhead of executing many clustering algorithms Advantages of the Framework

We introduced new properties of clustering algorithms. We use these properties to provide a characterization of linkage-based algorithms. We classified common clustering algorithms using these properties. Conclusions

Kleinberg (NIPS, 02) proposed 3 “axioms” of clustering functions, which he showed to be inconsistent. Ackerman and Ben-David (NIPS, 08) showed that the these properties are consistent in the setting of clustering quality measures. Goal: find a consistent set of axioms of clustering functions. Axioms of clustering

An axiom is a property that is satisfied by all members of a class A complete set of axioms of clustering functions would be satisfied by all clustering functions, and only by clustering functions Our goal is to find a complete set of axioms of clustering We use Kleinberg’s axioms as a starting point If we fix k, Kleinberg’s axioms are consistent. Axioms VS properties

Scale Invariance: The output of the function doesn’t change if the data is scaled uniformly.  Satisfied by common clustering algorithms. Richness: For all k -clustering C of X, there exists a distance function d over X so that F(X,d,k) = C.  Richness is implied by extended richness.  Satisfied by common clustering algorithms. Kleinberg’s axioms for fixed K Consistency: If d’ equals d, except for increasing between-cluster distances, then F(X,d,k)=F(X,d’,k).  Not satisfied by some common clustering algorithms.  Relaxations of this property, inner and outer consistency are also not satisfied by some common algorithms.

We propose using the following as axioms of clustering.  Scale invariance  Isomorphism invariance  Extended richness Are there natural clustering functions that fail any of these properties? Are these axioms sufficient? Towards axioms of clustering functions

LocalOuter Con. Inner Con. Heirar- chical Path Dist. Order Inv. Extent. Rich. Scale Inv. Iso. Inv. Single linkage Average linkage   Complete linkage   K-means  K-median  Min-Sum  Ratio-cut  Normalized- cut  PropertiesAxioms