Genetic Algorithms for clustering problem Pasi Fränti

General structure Genetic Algorithm: Generate S initial solutions REPEAT Z iterations Select best solutions Create new solutions by crossover Mutate solutions END-REPEAT

Main principle

Components of GA Representation of solution Selection method Crossover method Mutation Most critical !

Representation

Representation of solution Partition (P): – Optimal centroid can be calculated from P – Only local changes can be made Codebook (C): – Optimal partition can be calculated from C – Calculation of P takes O(NM)  slow Combined (C, P): – Both data structures are needed anyway – Computationally more efficient

Selection method To select which solutions will be used in crossover for generating new solutions Main principle: good solutions should be used rather than weak solutions Two main strategies: – Roulette wheel selection – Elitist selection Exact implementation not so important

Roulette wheel selection Select two candidate solutions for the crossover randomly. Probability for a solution to be selected is weighted according to its distortion:

Elitist selection Elitist approach using zigzag scanning among the best solutions Main principle: select all possible pairs among the best candidates.

Crossover

Crossover methods Different variants for crossover: Random crossover Centroid distance Pairwise crossover Largest partitions PNN Local fine-tuning: All methods give new allocation of the centroids. Local fine-tuning must be made by K-means. Two iterations of K-means is enough.

Random crossover Solution 1Solution 2 + Select M/2 centroids randomly from the two parent.

New Solution: How to create a new solution? Picking M/2 randomly chosen cluster centroids from each of the two parents in turn. How many solutions are there? 36 possibilities how to create a new solution. Probability to select a good one? Not high, some are good but K-Means is needed, most are bad. See statistics. Parent solution A Parent solution B Data point Centroid Explanation M – number of clusters Parent AParent BRating c 2, c 4 c 1, c 4 Optimal c 1, c 2 c 3, c 4 Good (K-Means) c 2, c 3 Bad Some possibilities: M = 4 c1c1 c4c4 c3c3 c2c c1c1 c4c4 c3c3 c2c2 Rough statistics: Optimal: 1 Good: 7 Bad:28

c1c1 c4c4 c3c3 c2c c1c1 c4c4 c3c3 c2c2 c1c1 c3c3 c2c2 c4c4 Child solution (optimal) Child solution (good) Child solution (bad) c1c1 c3c3 c2c2 c4c4 c1c1 c2c2 c4c4 c3c3 Parent solution A Parent solution B

Centroid distance crossover [Pan, McInnes, Jack, 1995: Electronics Letters ] [Scheunders, 1997: Pattern Recognition Letters ] For each centroid, calculate its distance to the center point of the entire data set. Sort the centroids according to the distance. Divide into two sets: central vectors (M/2 closest) and distant vectors (M/2 furthest). Take central vectors from one codebook and distant vectors from the other.

Parent solution A Parent solution B New solution: Variant (a) Take cental vectors from parent solution A and distant vectors from parent solution B OR Variant (b) Take distant vectors from parent solution A and central vectors from parent solution B Data point Centroid Explanation M – number of clusters Centroid of entire dataset A: d(c 4, C ed ) < d(c 2, C ed ) < d(c 1, C ed ) < d(c 3, C ed ) B: d(c 1, C ed ) < d(c 3, C ed ) < d(c 2, C ed ) < d(c 4, C ed ) 1) Distances d(c i, C ed ): 2) Sort centroids according to the distance: A: c 4, c 2, c 1, c 3, B: c 1, c 3, c 2, c 4 3) Divide into two sets (M = 4): A: central vectors: c 4, c 2, distant vectors: c 1, c 3 B: central vectors: c 1, c 3, distant vectors: c 2, c c1c1 c2c2 c3c3 c4c4 C ed c1c1 c2c2 c3c3 c4c4 c2c2 c4c4 c2c2 c4c4 c1c1 c3c3 c1c1 c3c

Child - variant (a) Child – variant (b) New solution: Variant (a) Take cental vectors from parent solution A and distant vectors from parent solution B OR Variant (b) Take distant vectors from parent solution A and central vectors from parent solution B Data point Centroid Explanation M – number of clusters Centroid of entire dataset c2c2 c4c4 c2c2 c4c4 c1c1 c3c3 c1c1 c3c c1c1 c2c2 c3c3 c4c4 C ed c1c1 c2c2 c3c3 c4c4

Pairwise crossover [Fränti et al, 1997: Computer Journal] Greedy approach: For each centroid, find its nearest centroid in the other parent solution that is not yet used. Among all pairs, select one of the two randomly. Small improvement: No reason to consider the parents as separate solutions. Take union of all centroids. Make the pairing independent of parent.

Initial parent solutions Pairwise crossover example MSE=8.79  10 9 MSE=11.92  10 9

Pairwise crossover example Pairing between parent solutions MSE=7.34  10 9

Pairing without restrictions MSE=4.76  10 9 Pairwise crossover example

Largest partitions [Fränti et al, 1997: Computer Journal] Crossover algorithm: Each cluster in the solutions A and B is assigned with a number, cluster size S, indicating how many data objects belong to it. In each phase we pick the centroid of the largest cluster. Assume that cluster i was chosen from A. The cluster centroid C i is removed from A to avoid its reselection. For the same reason we update the cluster sizes of B by removing the effect of those data objects in B that were assigned to the chosen cluster i in A.

Largest partitions [Fränti et al, 1997: Computer Journal] Data point Centroid Explanation Parent solution B S=100 S=50 S=30 S=20 Parent solution A S=50 S=30 S=20 c1c1 S=100

PNN crossover for GA [Fränti et al, 1997: The Computer Journal] Initial 2 After PNN Union Combined Initial 1 PNN

The PNN crossover method (1) [Fränti, 2000: Pattern Recognition Letters]

The PNN crossover method (2)

Importance of K-means (Random crossover) Best Worst Bridge

Effect of crossover method (with k-means iterations) Bridge

Effect of crossover method (with k-means iterations) Binary data Bridge2

Mutations

Purpose is to implement small random changes to the solutions. Happens with a small probability. Sensible approach: change the location of one centroid by the random swap! Role of mutations is to simulate local search. If mutations are needed  crossover method is not very good.

Effect of k-means and mutations Mutations alone better than random crossover! K-means improves but less vital

GAIS – Going extreme

Agglomerative clustering PNN: Pairwise Nearest Neigbor method – Merges two clusters – Preserves hierarchy of clusters IS: Iterative shrinking method Removes one cluster Repartition data vectors in removed cluster

Iterative shrinking

Pseudo code

Local optimization of IS Removal cost of single vector: Finding secondary cluster:

Example (1)

Example (2)

Pseudo code of GAIS [Virmajoki & Fränti, 2006: Pattern Recognition]

PNN vs. IS crossovers Further improvement of about 1%

Optimized GAIS variants GAIS short (optimized for speed): -Create new generations only as long as the best solution keeps improving (T=*). -Use a small population size (Z=10) -Apply two iterations of k ‑ means (G=2). GAIS long (optimized for quality): -Create a large number of generations (T=100) -Large population size (Z=100) -Iterate k ‑ means relatively long (G=10).

Comparison with image data Popular Previous GA BEST! Simplest of the good ones

What does it cost? Bridge Random:~0 s K-means:8 s SOM: 6 minutes GA-PNN:13 minutes GAIS – short:~1 hour GAIS – long:~3 days

Comparison of algorithms

Variation of the result

Time vs. quality comparison Bridge

Conclusions Best clustering obtained by GA Crossover method most important Mutations not needed

References 1.P. Fränti and O. Virmajoki, "Iterative shrinking method for clustering problems", Pattern Recognition, 39 (5), , May P. Fränti, "Genetic algorithm with deterministic crossover for vector quantization", Pattern Recognition Letters, 21 (1), 61-68, January P. Fränti, J. Kivijärvi, T. Kaukoranta and O. Nevalainen, "Genetic algorithms for large scale clustering problems", The Computer Journal, 40 (9), , J. Kivijärvi, P. Fränti and O. Nevalainen, "Self-adaptive genetic algorithm for clustering", Journal of Heuristics, 9 (2), , J.S. Pan, F.R. McInnes and M.A. Jack, VQ codebook design using genetic algorithms. Electronics Letters, 31, , August P. Scheunders, A genetic Lloyd-Max quantization algorithm. Pattern Recognition Letters, 17, , 1996.

Working space Text box