1. Data Mining: Concepts and Techniques
Chapter 4 (Introduction to Data Mining) Chapter 6 (Data Mining Concept and Analysis) Chapter 9 (Data Mining Analysis Fundamental Concept )

2. Classification and Prediction
What is classification? What is prediction?
Issues regarding classification and prediction
Classification by decision tree induction
Bayesian classification
Rule-based classification
Classification by back propagation
Support Vector Machines (SVM)
Associative classification
Lazy learners (or learning from your neighbors)
Other classification methods
Prediction
Accuracy and error measures
Ensemble methods
Model selection
Summary

3. Classification vs. Prediction
Given a collection of records (training set )
Each record is by characterized by a tuple (x,y), where x is the attribute set and y is the class label
x: attribute, predictor, independent variable, input
y: class, response, dependent variable, output
Prediction
models continuous-valued functions, i.e., predicts unknown or missing values

4. Examples of Classification Task
Attribute set, x
Class label, y
Categorizing messages
Features extracted from message header and content
spam or non-spam
Identifying tumor cells
Features extracted from MRI scans
malignant or benign cells
Cataloging galaxies
Features extracted from telescope images
Elliptical, spiral, or irregular-shaped galaxies

5. Classification—A Two-Step Process
Model construction: describing a set of predetermined classes
Each tuple/sample is assumed to belong to a predefined class, as determined by the class label attribute
The set of tuples used for model construction is training set
The model is represented as classification rules, decision trees, or mathematical formulae
Model usage: for classifying future or unknown objects
Estimate accuracy of the model
The known label of test sample is compared with the classified result from the model
Accuracy rate is the percentage of test set samples that are correctly classified by the model
Test set is independent of training set, otherwise over-fitting will occur
If the accuracy is acceptable, use the model to classify data tuples whose class labels are not known

6. Process (1): Model Construction
Classification
Algorithms
Training
Data
Classifier
(Model)
IF rank = ‘professor’
OR years > 6
THEN tenured = ‘yes’

7. Process (2): Using the Model in Prediction
Classifier
Testing
Data
Unseen Data
(Jeff, Professor, 4)
Tenured?

8. Classification Techniques
Base Classifiers
Decision Tree based Methods
Rule-based Methods
Nearest-neighbor
Neural Networks
Naïve Bayes and Bayesian Belief Networks
Support Vector Machines
Ensemble Classifiers
Boosting, Bagging, Random Forests

13. Example of a Decision Tree
categorical
categorical
continuous
class
Splitting Attributes
Home Owner
Yes No NO
MarSt
Single, Divorced
Married
Income NO
< 80K
> 80K NO
YES
Training Data
Model: Decision Tree

14. Another Example of Decision Tree
categorical
categorical
continuous
class
MarSt
Single, Divorced
Married NO
Home Owner No
Yes NO
Income
< 80K
> 80K NO
YES
There could be more than one tree that fits the same data!

15. Decision Tree Classification Task

16. Apply Model to Test Data
Start from the root of tree.
Home Owner
Yes No NO
MarSt
Single, Divorced
Married
Income NO
< 80K
> 80K NO
YES

17. Apply Model to Test Data
Home Owner
Yes No NO
MarSt
Single, Divorced
Married
Income NO
< 80K
> 80K NO
YES

18. Apply Model to Test Data
Home Owner
Yes No NO
MarSt
Single, Divorced
Married
Income NO
< 80K
> 80K NO
YES

19. Apply Model to Test Data
Home Owner
Yes No NO
MarSt
Single, Divorced
Married
Income NO
< 80K
> 80K NO
YES

20. Apply Model to Test Data
Home Owner
Yes No NO
MarSt
Single, Divorced
Married
Income NO
< 80K
> 80K NO
YES

21. Apply Model to Test Data
Home Owner
Yes No NO
MarSt
Single, Divorced
Married
Assign Defaulted to “No”
Income NO
< 80K
> 80K NO
YES

22. Supervised vs. Unsupervised Learning
Supervised learning (classification)
Supervision: The training data (observations, measurements, etc.) are accompanied by labels indicating the class of the observations
New data is classified based on the training set
Unsupervised learning (clustering)
The class labels of training data is unknown
Given a set of measurements, observations, etc. with the aim of establishing the existence of classes or clusters in the data

23. Chapter 6. Classification and Prediction
What is classification? What is prediction?
Issues regarding classification and prediction
Classification by decision tree induction
Bayesian classification
Rule-based classification
Classification by back propagation
Support Vector Machines (SVM)
Associative classification
Lazy learners (or learning from your neighbors)
Other classification methods
Prediction
Accuracy and error measures
Ensemble methods
Model selection
Summary

24. Issues: Data Preparation
Data cleaning
Preprocess data in order to reduce noise and handle missing values
Relevance analysis (feature selection)
Remove the irrelevant or redundant attributes
Data transformation
Generalize and/or normalize data

25. Issues: Evaluating Classification Methods
Accuracy
classifier accuracy: predicting class label
predictor accuracy: guessing value of predicted attributes
Speed
time to construct the model (training time)
time to use the model (classification/prediction time)
Robustness: handling noise and missing values
Scalability: efficiency in disk-resident databases
Interpretability
understanding and insight provided by the model
Other measures, e.g., goodness of rules, such as decision tree size or compactness of classification rules

26. Chapter 6. Classification and Prediction
What is classification? What is prediction?
Issues regarding classification and prediction
Classification by decision tree induction
Bayesian classification
Rule-based classification
Classification by back propagation
Support Vector Machines (SVM)
Associative classification
Lazy learners (or learning from your neighbors)
Other classification methods
Prediction
Accuracy and error measures
Ensemble methods
Model selection
Summary

27. Decision Tree Induction
Many Algorithms:
Hunt’s Algorithm (one of the earliest)
CART
ID3, C4.5
SLIQ,SPRINT

28. Decision Tree Induction: Training Dataset
This follows an example of Quinlan’s ID3 (Playing Tennis)

29. Output: A Decision Tree for “buys_computer”
age?
overcast
student?
credit rating?
<=30
>40 no
yes
31..40
fair
excellent

30. Algorithm for Decision Tree Induction
Basic algorithm (a greedy algorithm)
Tree is constructed in a top-down recursive divide-and-conquer manner
At start, all the training examples are at the root
Attributes are categorical (if continuous-valued, they are discretized in advance)
Examples are partitioned recursively based on selected attributes
Test attributes are selected on the basis of a heuristic or statistical measure (e.g., information gain)
Conditions for stopping partitioning
All samples for a given node belong to the same class
There are no remaining attributes for further partitioning – majority voting is employed for classifying the leaf
There are no samples left

31. Design Issues of Decision Tree Induction
How should training records be split?
Method for specifying test condition
depending on attribute types
Measure for evaluating the goodness of a test condition
How should the splitting procedure stop?
Stop splitting if all the records belong to the same class or have identical attribute values
Early termination

32. Methods for Expressing Test Conditions
Depends on attribute types
Binary
Nominal
Ordinal
Continuous
Depends on number of ways to split
2-way split
Multi-way split

33. Test Condition for Nominal Attributes
Multi-way split:
Use as many partitions as distinct values.
Binary split:
Divides values into two subsets
Need to find optimal partitioning.

34. Test Condition for Ordinal Attributes
Multi-way split:
Use as many partitions as distinct values
Binary split:
Divides values into two subsets
Need to find optimal partitioning
Preserve order property among attribute values
This grouping violates order property

35. Test Condition for Continuous Attributes

36. Splitting Based on Continuous Attributes
Different ways of handling
Discretization to form an ordinal categorical attribute
Static – discretize once at the beginning
Dynamic – ranges can be found by equal interval bucketing, equal frequency bucketing (percentiles), or clustering.
Binary Decision: (A < v) or (A  v)
consider all possible splits and finds the best cut
can be more compute intensive

37. How to determine the Best Split
Before Splitting: 10 records of class 0, 10 records of class 1
Which test condition is the best?

38. How to determine the Best Split
Greedy approach:
Nodes with purer class distribution are preferred
Need a measure of node impurity:
High degree of impurity
Low degree of impurity

39. Measures of Node Impurity
Gini Index
Entropy
Misclassification error

40. Finding the Best Split Compute impurity measure (P) before splitting
Compute impurity measure (M) after splitting
Compute impurity measure of each child node
Compute the average impurity of the children (M)
Choose the attribute test condition that produces the highest gain or equivalently, lowest impurity measure after splitting (M)
Gain = P – M

41. Finding the Best Split P A? B? M11 M12 M21 M22 M1 M2
Before Splitting: P A? B?
Yes No
Yes No
Node N1
Node N2
Node N3
Node N4
M11
M12
M21
M22 M1 M2
Gain = P – M1 vs P – M2

42. Measure of Impurity: GINI
Gini Index for a given node t :
(NOTE: p( j | t) is the relative frequency of class j at node t).
Maximum (1 - 1/nc) when records are equally distributed among all classes, implying least interesting information
Minimum (0.0) when all records belong to one class, implying most interesting information

43. Computing Gini Index of a Single Node
P(C1) = 0/6 = P(C2) = 6/6 = 1
Gini = 1 – P(C1)2 – P(C2)2 = 1 – 0 – 1 = 0
P(C1) = 1/ P(C2) = 5/6
Gini = 1 – (1/6)2 – (5/6)2 = 0.278
P(C1) = 2/ P(C2) = 4/6
Gini = 1 – (2/6)2 – (4/6)2 = 0.444

44. Computing Gini Index for a Collection of Nodes
When a node p is split into k partitions (children)
where, ni = number of records at child i,
n = number of records at parent node p.
Choose the attribute that minimizes weighted average Gini index of the children
Gini index is used in decision tree algorithms such as CART, SLIQ, SPRINT

45. Binary Attributes: Computing GINI Index
Splits into two partitions
Effect of Weighing partitions:
Larger and Purer Partitions are sought for. B?
Yes No
Node N1
Node N2
Gini(N1) = 1 – (5/6)2 – (1/6)2 = 0.278
Gini(N2) = 1 – (2/6)2 – (4/6)2 = 0.444
Gini(Children) = 6/12 * /12 * = 0.361

46. Categorical Attributes: Computing Gini Index
For each distinct value, gather counts for each class in the dataset
Use the count matrix to make decisions
Multi-way split
Two-way split
(find best partition of values)

47. Continuous Attributes: Computing Gini Index
Use Binary Decisions based on one value
Several Choices for the splitting value
Number of possible splitting values = Number of distinct values
Each splitting value has a count matrix associated with it
Class counts in each of the partitions, A < v and A  v
Simple method to choose best v
For each v, scan the database to gather count matrix and compute its Gini index
Computationally Inefficient! Repetition of work.

48. Continuous Attributes: Computing Gini Index...
For efficient computation: for each attribute,
Sort the attribute on values
Linearly scan these values, each time updating the count matrix and computing gini index
Choose the split position that has the least gini index
Sorted Values
Split Positions

49. GINI Index: CART CART: The CART measure is
This measure thus prefers a split point that maximizes the difference between the class probability mass function for the two partitions; the higher the CART measure, the better the split point.

50. Measure of Impurity: Entropy
Entropy at a given node t:
(NOTE: p( j | t) is the relative frequency of class j at node t).
Maximum (log nc) when records are equally distributed among all classes implying least information
Minimum (0.0) when all records belong to one class, implying most information
Entropy based computations are quite similar to the GINI index computations

51. Computing Entropy of a Single Node
P(C1) = 0/6 = P(C2) = 6/6 = 1
Entropy = – 0 log 0 – 1 log 1 = – 0 – 0 = 0
P(C1) = 1/ P(C2) = 5/6
Entropy = – (1/6) log2 (1/6) – (5/6) log2 (1/6) = 0.65
P(C1) = 2/ P(C2) = 4/6
Entropy = – (2/6) log2 (2/6) – (4/6) log2 (4/6) = 0.92

52. Computing Information Gain After Splitting
Parent Node, p is split into k partitions;
ni is number of records in partition i
Choose the split that achieves most reduction (maximizes GAIN)
Used in the ID3 and C4.5 decision tree algorithms

53. Problems with Information Gain
Info Gain tends to prefer splits that result in large number of partitions, each being small but pure
Customer ID has highest information gain because entropy for all the children is zero

54. Gain Ratio
Gain Ratio:
Parent Node, p is split into k partitions
ni is the number of records in partition i
Adjusts Information Gain by the entropy of the partitioning (SplitINFO).
Higher entropy partitioning (large number of small partitions) is penalized!
Used in C4.5 algorithm
Designed to overcome the disadvantage of Information Gain

55. Measure of Impurity: Classification Error
Classification error at a node t :
Maximum (1 - 1/nc) when records are equally distributed among all classes, implying least interesting information
Minimum (0) when all records belong to one class, implying most interesting information

56. Computing Error of a Single Node
P(C1) = 0/6 = P(C2) = 6/6 = 1
Error = 1 – max (0, 1) = 1 – 1 = 0
P(C1) = 1/ P(C2) = 5/6
Error = 1 – max (1/6, 5/6) = 1 – 5/6 = 1/6
P(C1) = 2/ P(C2) = 4/6
Error = 1 – max (2/6, 4/6) = 1 – 4/6 = 1/3

57. Comparing Attribute Selection Measures
The three measures, in general, return good results but
Information gain:
biased towards multivalued attributes
Gain ratio:
tends to prefer unbalanced splits in which one partition is much smaller than the others
Gini index:
biased to multivalued attributes
has difficulty when # of classes is large
tends to favor tests that result in equal-sized partitions and purity in both partitions

58. Comparison among Impurity Measures
For a 2-class problem:

59. Misclassification Error vs Gini Index
Yes No
Node N1
Node N2
Gini(N1) = 1 – (3/3)2 – (0/3)2 = 0
Gini(N2) = 1 – (4/7)2 – (3/7)2 = 0.489
Gini(Children) = 3/10 * /10 * = 0.342
Gini improves but error remains the same!!

60. Decision Tree Based Classification
Advantages:
Inexpensive to construct
Extremely fast at classifying unknown records
Easy to interpret for small-sized trees
Accuracy is comparable to other classification techniques for many simple data sets

61. Other Attribute Selection Measures
CHAID: a popular decision tree algorithm, measure based on χ2 test for independence
C-SEP: performs better than info. gain and gini index in certain cases
G-statistics: has a close approximation to χ2 distribution
MDL (Minimal Description Length) principle (i.e., the simplest solution is preferred):
The best tree as the one that requires the fewest # of bits to both (1) encode the tree, and (2) encode the exceptions to the tree
Multivariate splits (partition based on multiple variable combinations)
CART: finds multivariate splits based on a linear comb. of attrs.
Which attribute selection measure is the best?
Most give good results, none is significantly superior than others

62. Enhancements to Basic Decision Tree Induction
Allow for continuous-valued attributes
Dynamically define new discrete-valued attributes that partition the continuous attribute value into a discrete set of intervals
Handle missing attribute values
Assign the most common value of the attribute
Assign probability to each of the possible values
Attribute construction
Create new attributes based on existing ones that are sparsely represented
This reduces fragmentation, repetition, and replication

63. Classification in Large Databases
Classification—a classical problem extensively studied by statisticians and machine learning researchers
Scalability: Classifying data sets with millions of examples and hundreds of attributes with reasonable speed
Why decision tree induction in data mining?
relatively faster learning speed (than other classification methods)
convertible to simple and easy to understand classification rules
can use SQL queries for accessing databases
comparable classification accuracy with other methods

64. Scalable Decision Tree Induction Methods
SLIQ (EDBT’96 — Mehta et al.)
Builds an index for each attribute and only class list and the current attribute list reside in memory
SPRINT (VLDB’96 — J. Shafer et al.)
Constructs an attribute list data structure
PUBLIC (VLDB’98 — Rastogi & Shim)
Integrates tree splitting and tree pruning: stop growing the tree earlier
RainForest (VLDB’98 — Gehrke, Ramakrishnan & Ganti)
Builds an AVC-list (attribute, value, class label)
BOAT (PODS’99 — Gehrke, Ganti, Ramakrishnan & Loh)
Uses bootstrapping to create several small samples