Data Clustering 1 – An introduction Slide 1

The Data Explosion “If you feel like you are drowning in information, it’s because you are.” Advance of IT and the Internet Massive increase in ability to: Record: Electronic records and forms Store: Data Warehouses (as we have seen) Analyse: Data Mining and Visualisation (more later) Risk of Information Overload Data Clustering – An Introduction Slide 2

The Aims of Data Mining Classification Association Detection Categorising Risk-Return of Stocks Association Identify products that tend to sell together Detection Identify profiles of customers Prediction Forecasting Market Performance Data Clustering – An Introduction Slide 3

Database Technology Timeline Data collection, database creation, IMS and network DBMS 1970s: Relational data model, relational DBMS implementation 1980s: RDBMS, advanced data models (extended-relational, OO, deductive, etc.) and application-oriented DBMS (spatial, scientific, engineering, etc.) 1990s—2000s: Data mining and data warehousing, multimedia databases, and Web databases Data Clustering – An Introduction Slide 4

From Data to Knowledge Common to break down the process of learning from data into the following: Data, Information and Knowledge Data Clustering – An Introduction Slide 5

From Data to Knowledge Data: Raw numbers Information: Data with context or meaning Knowledge: Data Structures / Patterns (Knowledge must be useful) Data Clustering – An Introduction Slide 6

Data Mining / Intelligent Data Analysis “Data mining is applying Machine Learning techniques to historical data to improve future decisions” Tom Mitchell 1997 Data Clustering – An Introduction Slide 7

Knowledge Discovery Knowledge Discovery in Databases (KDD) The Process (from Advances in KDD and Data mining): Data Knowledge Patterns Target Data Pre-processed Data Transformed Data Data Clustering – An Introduction Slide 8

Data Mining - Tools Typical tools Statistical Analysis Summarisation Outlier Detection Correlation Regression Clustering Association Rules Time Series Models Decision Trees (classification) Data Clustering – An Introduction Slide 9

Data Mining - Applications Some successful examples of its use: Pharmaceutical companies – Drug Discovery Credit card companies – Fraud Detection Transportation companies - Routing Large consumer package goods companies (to improve the sales process to retailers) Hospital Organisation – Decision Analysis Data Clustering – An Introduction Slide 10

Examples of Data Mining Tools We will now look at some core techniques commonly used for analysing and mining business warehouses Correlation Visualisation Clustering Regression Data Clustering – An Introduction Slide 11

Clustering An example in biology… plants animals clustering is a basic learning algorithm we do clustering everywhere of our lives an example in biology introduce the 3 concepts in pattern recognition – patterns, features, classes. Things that are brown and run away Things that are green and don’t run away Data Clustering – An Introduction Slide 12

Clustering An example in biology… Kingdom Phylum Class Order Family Genus Species Hierarchical clustering (more later) clustering is a basic learning algorithm we do clustering everywhere of our lives an example in biology introduce the 3 concepts in pattern recognition – patterns, features, classes. Data Clustering – An Introduction Slide 13

Clustering The process Extract features (colour, movement, sensory organs etc): more later Cluster into categories Consolidation clustering is a basic learning algorithm we do clustering everywhere of our lives an example in biology introduce the 3 concepts in pattern recognition – patterns, features, classes. Data Clustering – An Introduction Slide 14

Clustering Clustering: to partition a data set into subsets (clusters), so that the data in each subset share some common trait - often similarity or proximity for some defined distance measure. The process of organizing objects into groups whose members are similar in some way. A cluster is therefore a collection of objects which are “similar” between them and are “dissimilar” to the objects belonging to other clusters. Unsupervised: No need for the ‘teacher’ signals, i.e. the desired output. x2 Cluster 1 clustering: extend the concept of living beings to a general one key issues of clustering analysis Cluster 2 x1

Supervised and Unsupervised Learning Unsupervised learning: learning without the desired output (‘teacher’ signals). Supervised learning: learning with the desired output. Clustering is one of the widely-used unsupervised learning methods. Other unsupervised learning: Dimensionality reduction (factor analysis, principal component analysis, independent component analysis …) Time serious modelling Source separation Supervised learning: Classification Regression briefly introduce the concepts of supervised and unsupervised learning misunderstanding: clustering = unsupervised learning Data Clustering – An Introduction Slide 16

Patterns, Clusters and Features (1) Patterns: physical objects Clusters: categories of objects Features: attributes of objects animals plants before introducing how to perform clustering, the basic concepts patterns clusters features Colour: brown, green, …

Patterns, Clusters and Features (2) Features’ space Creating vehicles’ clusters 3500 3000 Lorries 2500 cluster 2000 Sports cars Weight [kg] 1500 Medium market cars Another example for vehicle clustering individual cars – patterns(objects) features – weight/speed clusters – lorries/sport cars/cars Feature-1 values 1000 500 100 150 200 250 300 Top speed [ml/h] Feature-2 values

Social networks Marketing Terror networks Allocation of resources in a company / university Data Clustering – An Introduction

Gene networks Understanding gene interactions Identifying important genes linked to disease Data Clustering – An Introduction

How to do clustering? What we know: patterns represented by their feature vectors, e.g. General case: is in the d -dimensional domain of the feature vectors x2 Cluster 1 Cluster 2 what we know patterns feature vectors examples like animals and cars 2. what we need to find out the number of clusters the clusters, in a form easy for computing What we need to find out: the clusters x1

Pattern Similarity A key concept in clustering: similarity. Clusters are formed by similar patterns. In computer science, we need to define some metric to measure similarity. One of the commonly adopted similarity metrics is distance. A general definition of distance (between pattern A and B): b=2: Euclidean distance b=1: Manhattan distance a key concept of clustering, and many other pattern recognition techniques, is similarity. distance similarity is inversely proportional to the distance – this sometimes presents problems. The shorter the distance, the more similar the two patterns.

Pattern Similarity & Distance Metrics Many methods are designed to work on Distance Metrics, e.g. K-Means They assume that the Triangle Inequality holds: “the sum of the lengths of any two sides must be greater than the length of the remaining side” Data Clustering – An Introduction

Pattern Similarity & Distance Metrics Euclidean Correlation Minkowski Manhattan Mahalanobis Relationship Metrics How Long is a Piece of String? Often Application Dependant Data Clustering – An Introduction

K-Means Clustering 25

Algorithm 1: K-Means Clustering Place K points into the feature space. These points represent initial cluster centroids. Assign each pattern to the closest cluster centroid. When all objects have been assigned, recalculate the positions of the K centroids. Repeat Steps 2 and 3 until the assignments do not change. this description is rather generic. many issues are unspecified. initialisation assignment updating For example, if using (1) Euclidean distance, (2) average of patterns, what the algorithm becomes? Data Clustering – An Introduction Slide 26

K-Means Clustering Interactive Demo: http://home.dei.polimi.it/matteucc/Clustering/tutorial_html/AppletKM.html Data Clustering – An Introduction Slide 27

Discussions (1) 1. How to determine k, the number of clusters? Data Clustering – An Introduction Slide 28

Discussions (2) 2. Any alternative ways of choosing the initial cluster centroids? Data Clustering – An Introduction Slide 29

Discussions (3) 3. Does the algorithm converge to the same results with different selections of initial cluster centroids? If not, what should we do in practice? Data Clustering – An Introduction Slide 30

Reading Chapter 9, Section 9.3: David Hand “Principles of Data Mining”, MIT Press Chapter 8: Pang-Ning Tan “Introduction to Data Mining” Anil Jain: “Data Clustering: 50 Years Beyond K-Means”, Pattern Recognition Letters Data Clustering – An Introduction Slide 31

Lab In the lab: Examine a piece of JAVA code for K-Means clustering Explore the use of K-Means on some Toy datasets Visualise the clusterings using an EXCEL macro Data Clustering – An Introduction Slide 32