Dept. of Computer Science University of Liverpool COMP527: Data Mining COMP527: Data Mining M. Sulaiman Khan (mskhan@liv.ac.uk)‏ Dept. of Computer Science University of Liverpool 2009 This is the full course notes, but not quite complete. You should come to the lectures anyway. Really. Introduction to the Course February 05, 2009 Slide 1

COMP527: Data Mining COMP527: Data Mining Introduction to the Course Introduction to Data Mining Introduction to Text Mining General Data Mining Issues Data Warehousing Classification: Challenges, Basics Classification: Rules Classification: Trees Classification: Trees 2 Classification: Bayes Classification: Neural Networks Classification: SVM Classification: Evaluation Classification: Evaluation 2 Regression, Prediction Input Preprocessing Attribute Selection Association Rule Mining ARM: A Priori and Data Structures ARM: Improvements ARM: Advanced Techniques Clustering: Challenges, Basics Clustering: Improvements Clustering: Advanced Algorithms Hybrid Approaches Graph Mining, Web Mining Text Mining: Challenges, Basics Text Mining: Text-as-Data Text Mining: Text-as-Language Revision for Exam Classification: Challenges, Basics February 05, 2009 Slide 2

Today's Topics Classification Basic Algorithms: KNN Perceptron Winnow COMP527: Data Mining Classification Basic Algorithms: KNN Perceptron Winnow Classification: Challenges, Basics February 05, 2009 Slide 3

Training: Learn the classification model from labeled data COMP527: Data Mining Main Idea: Learn the concept of what it means to be part of a named class of instances. Called Supervised Learning, as it learns by example from data which is already classified correctly. Often called the Class Label attribute, hence learns from Labeled data. Two main phases: Training: Learn the classification model from labeled data Prediction: Use the pre-built model to classify new instances Classification: Challenges, Basics February 05, 2009 Slide 4

Classification Accuracy COMP527: Data Mining We need to use previously unseen instances to test a classifier. Over-fitting is the main problem. Classifiers will often learn too specific a model, and testing on data that was used in training would reinforce this problem. Need to split the data set into training and testing. Revised Phases for accuracy estimation: Split data set into distinct Training and Testing sets Build classifier with Training set Assess accuracy with Testing set – normally expressed as % Classification: Challenges, Basics February 05, 2009 Slide 5

Percent of instances classified correctly. Speed Comparing Methods COMP527: Data Mining Accuracy Percent of instances classified correctly. Speed Computational cost of both learning model and predicting classes Robustness Ability to cope with noisy or missing data Scalability Ability to cope with very large amounts of data Interpretability Is the model understandable to a human, or otherwise useful? Classification: Challenges, Basics February 05, 2009 Slide 6

Classification vs Prediction COMP527: Data Mining Classification predicts a class label from a given finite set. The label is a nominal attribute, so unordered and enumerable Some algorithms predict probability for more than one label Sometimes called a categorical attribute Prediction predicts a number instead of a label. Ordered and infinite set of possible outcomes Also often called Regression or Numeric Prediction Often viewed as a function Classification: Challenges, Basics February 05, 2009 Slide 7

Builds a model likely to be very different in structure to the data. Eager vs Lazy Learners COMP527: Data Mining Eager Learner: Constructs the model when it receives the training data. Builds a model likely to be very different in structure to the data. Lazy Learner: Doesn't construct a model when training, only when classifying new instances. Does only enough work to ensure that data can be compared later Sometimes called instance-based learners Most classifiers are Eager, but there's an important Lazy classifier called 'KNN' – K Nearest Neighbour Classification: Challenges, Basics February 05, 2009 Slide 8

Which group, left or right, for these two flowers? But First... COMP527: Data Mining Which group, left or right, for these two flowers? (Experiment reported on in Cognitive Science, 2002)‏ Classification: Challenges, Basics February 05, 2009 Slide 9

A combination of rote memorisation and the notion of 'resembles'. Resemblance COMP527: Data Mining People classify things by finding other items that are similar which have already been classified. For example: Is a new species a bird? Does it have the same attributes as lots of other birds? If so, then it's probably a bird too. A combination of rote memorisation and the notion of 'resembles'. Although kiwis can't fly like most other birds, they resemble birds more than they resemble other types of animals. So the problem is to find which instances most closely resemble the instance to be classified. Classification: Challenges, Basics February 05, 2009 Slide 10

KNN: Distance Measures COMP527: Data Mining Distance (or similarity) between instances is easy if the data is numeric. Typically use Euclidian distance: d = √(x1i-x1j)2 + (x2i-x2j)2 + ... Also Manhattan / City Block distance: d = (x1i-x1j) + (x2i-x2j) + ... However we should normalise all of the values to the same scale first. Otherwise income will overpower age, for example. Classification: Challenges, Basics February 05, 2009 Slide 11

KNN: Non Numeric Distance COMP527: Data Mining For nominal attributes, we can only compare whether the value is the same or not. Equally, this can be done by dividing enumerations into many boolean attributes. Might be able to convert to attributes between which distance can be determined by some function. Eg colour, temperature. Text can be treated as one attribute per word, with the frequency as the value, normalised to 0..1, and preferably with very high frequency words ignored (eg the, a, as, is...)‏ Classification: Challenges, Basics February 05, 2009 Slide 12

Classification process is then straight forward: KNN: Classification COMP527: Data Mining Classification process is then straight forward: Find the k closest instances to the test instance Predict the most common class among those instances Or predict the mean, for numeric prediction What value to use for k? Depends on dataset size. Large dbs need a higher k, whereas a high k for a small dataset might cross out of the class boundaries Calculate accuracy on test set for increasing value of k, and use a hill climbing algorithm to find the best. Typically use an odd number to help avoid ties Classification: Challenges, Basics February 05, 2009 Slide 13

KNN: Classification COMP527: Data Mining 5-NN – find 5 closest to black point, 3 blue and 2 red, so predict blue Classification: Challenges, Basics February 05, 2009 Slide 14

Classification can be very slow to find the k nearest instances. KNN: Classification COMP527: Data Mining Classification can be very slow to find the k nearest instances. In a trivial implementation, it could take |D| comparisons. Using indexing it can easily be improved. Also easy to parallelise as one comparison is completely distinct to other comparisons. Can remove instances from the data set that do not help, for example a tight cluster of 1000 instances of the same class is unnecessary for k<50 Can also use advanced data structures to improve the speed of classification, by storing the instance information appropriately. Classification: Challenges, Basics February 05, 2009 Slide 15

KNN: kD-Trees COMP527: Data Mining KD-Tree is a binary tree that divides the input space with a plane, then splits each such partition recursively. Each split is made parallel to an axis and through an instance. Typical strategy is to find the point closest to the mean in the current partition and split through it, along a different axis to the previous split. (Actually on the axis with the greatest variance)‏ Then to search, descend the tree to the leaf partition that contains the test instance. Search only that partition, then if an edge is closer than any of the k closest instances, search the parent partitions as well. Classification: Challenges, Basics February 05, 2009 Slide 16

KNN: kD-Trees First split at instance (7,4) then again at COMP527: Data Mining First split at instance (7,4) then again at (6,7) which divides the search space into more easily searchable sections. Classification: Challenges, Basics February 05, 2009 Slide 17

KNN: kD-Trees Then to classify the star, descend COMP527: Data Mining Then to classify the star, descend into the section with both star and the black instance. But note that the instance in the other section is closer, so we still must check the adjacent area. Note that the shaded area is the black node's sibling and hence cannot contain closer points. Classification: Challenges, Basics February 05, 2009 Slide 18

Two very simple eager methods: Perceptron and Winnow. COMP527: Data Mining Two very simple eager methods: Perceptron and Winnow. The both use the idea of a single neuron that fires when given the right stimuli. (We'll look at this idea again later under Neural Networks)‏ First thing to keep in mind is that the input to the perceptron must be a vector of numbers. Secondly, that it can only answer a 2 class problem – either the neuron fires (class 1) or it doesn't (class 2). Classification: Challenges, Basics February 05, 2009 Slide 19

Perceptron and Winnow COMP527: Data Mining The square boxes are inputs, the w lines are weights and the circle is the perceptron. The learning problem is to find the correct weights to apply to the attributes. The bias is a fixed value (1)‏ that is then learnt in the same way as the other attributes, in order to ensure that the result perceptron can check if the result is > 0 or not to see if it should fire. Classification: Challenges, Basics February 05, 2009 Slide 20

We can then multiply weight by value, and add them all up... Perceptron COMP527: Data Mining For each attribute, we have an input node. Then there is one output node to which all of them connect, with a weight on each connection. We can then multiply weight by value, and add them all up... w0a0 + w1a1 + ... + wnan Make it an equation equal to 0 and it's the equation for a hyperplane. So essentially we are learning the hyperplane that separates the two classes. Then classification is just checking which side of the plane the instance falls on. But how do we learn the weights? Classification: Challenges, Basics February 05, 2009 Slide 21

No complicated higher math here! Perceptron COMP527: Data Mining Remember that instances are a set of numeric attributes (a vector). We can also treat the weights on the connections as a vector. We only want to classify between two classes. So: weightVector = [0,...0] while classificationFailed, for each training instance I, if not classify(I) == I.class, if I.class == class1: weightVector += I else: weightVector -= I No complicated higher math here! Classification: Challenges, Basics February 05, 2009 Slide 22

Winnow COMP527: Data Mining Winnow only updates when it finds a misclassified instance, and uses multiplication to do the update rather than addition. It only works when the attribute values are also binary. (1 or 0)‏ delta = (user defined)‏ while classificationFailed, for each instance I, if classify(I) != I.class, if I.class == class1, for each attribute ai in I, if ai == 1, wi *= delta else, if ai == 1, wi /= delta Classification: Challenges, Basics February 05, 2009 Slide 23

Further Reading Witten, Section 3.8, and pp 124-136 COMP527: Data Mining Witten, Section 3.8, and pp 124-136 Han, Sections 6.1, 6.9 Dunham Sections 4.1-4.3 Berry and Linoff, Chapter 8 Berry and Browne, Chapter 6 Devijver and Kittler, Pattern Recognition: A Statistical Approach, Chapter 3 For KNN and Perceptron, Wikipedia, as always :)‏ Classification: Challenges, Basics February 05, 2009 Slide 24