Clustering methods Course code: Pasi Fränti Speech & Image Processing Unit School of Computing University of Eastern Finland Joensuu, FINLAND Part 1: Introduction

Sample data Sources of RGB vectors Red-Green plot of the vectors

Sample data Employment statistics:

Application example 1 Color reconstruction Image with compression artifacts Image with original colors

Application example 2 speaker modeling for voice biometrics Training data Feature extraction and clustering Matti Mikko Tomi Speaker models Tomi Matti Feature extraction Best match: Matti ! Mikko ?

Speaker modeling Speech dataResult of clustering

Application example 3 Image segmentation Normalized color plots according to red and green components. Image with 4 color clusters red green

Application example 4 Quantization Quantized signal Original signal Approximation of continuous range values (or a very large set of possible discrete values) by a small set of discrete symbols or integer values

Color quantization of images Color imageRGB samples Clustering

Application example 5 Clustering of spatial data

Clustered locations of users

Clustering of photos Timeline clustering

Clustering GPS trajectories Mobile users, taxi routes, fleet management

Conclusions from clusters Cluster 1: Office Cluster 2: Home

Part I: Clustering problem

Subproblems of clustering Where are the clusters? (Algorithmic problem) How many clusters? (Methodological problem: which criterion?) Selection of attributes (Application related problem) Preprocessing the data (Practical problems: normalization, outliers)

Clustering result as partition Illustrated by Voronoi diagram Illustrated by Convex hulls Cluster prototypes Partition of data

Cluster prototypes Partition of data Centroids as prototypes Partition by nearest prototype mapping Duality of partition and centroids

Cluster missingClusters missing Too many clusters Incorrect cluster allocation Incorrect number of clusters Challenges in clustering

How to solve? Solve the clustering:   Given input data (X) of N data vectors, and number of clusters (M), find the clusters.   Result given as a set of prototypes, or partition. Solve the number of clusters:   Define appropriate cluster validity function f.   Repeat the clustering algorithm for several M.   Select the best result according to f. Solve the problem efficiently. Algorithmic problem Mathematical problem Computer science problem

Taxonomy of clustering [Jain, Murty, Flynn, Data clustering: A review, ACM Computing Surveys, 1999.] One possible classification based on cost function. MSE is well defined and most popular.

Definitions and data Set of N data points: X={x 1, x 2, …, x N } Set of M cluster prototypes (centroids): C={c 1, c 2, …, c M }, P={p 1, p 2, …, p M }, Partition of the data:

Distance and cost function Euclidean distance of data vectors: Mean square error:

  Centroid condition: for a given partition (P), optimal cluster centroids (C) for minimizing MSE are the average vectors of the clusters: Dependency of data structures  Optimal partition: for a given centroids (C), optimal partition is the one with nearest centroid :

Complexity of clustering Clustering problem is NP complete [Garey et al., 1982] Optimal solution by branch-and-bound in exponential time. Practical solutions by heuristic algorithms. Number of possible clusterings:

Cluster software Main area Input area Output area Main area: working space for data Input area: inputs to be processed Output area: obtained results Menu Process: selection of operation

Clustering image Data set Codebook Partition Procedure to simulate k-means Open data set (file *.ts), move it into Input area Process – Random codebook, select number of clusters REPEAT Move obtained codebook from Output area into Input area Process – Optimal partition, select Error function Move codebook into Main area, partition into Input area Process – Optimal codebook UNTIL DESIRED CLUSTERING

XLMiner software

Example of data in XLMiner

Distance matrix & dendrogram

Conclusions   Clustering is a fundamental tools needed in Speech and Image processing.   Failing to do clustering properly may defect the application analysis.   Good clustering tool needed so that researchers can focus on application requirements.

1. 1. S. Theodoridis and K. Koutroumbas, Pattern Recognition, Academic Press, 3rd edition, C. Bishop, Pattern Recognition and Machine Learning, Springer, A.K. Jain, M.N. Murty and P.J. Flynn, Data clustering: A review, ACM Computing Surveys, 31(3): , September M.R. Garey, D.S. Johnson and H.S. Witsenhausen, The complexity of the generalized Lloyd-Max problem, IEEE Transactions on Information Theory, 28(2): , March F. Aurenhammer: Voronoi diagrams-a survey of a fundamental geometric data structure, ACM Computing Surveys, 23 (3), , September Literature