1. A task of induction to find patterns
Classification
A task of induction to find patterns
5/25/2019
CSE 572, CBS598: Data Mining by H. Liu

2. CSE 572, CBS598: Data Mining by H. Liu
Outline
Data and its format
Problem of Classification
Learning a classifier
Different approaches
Key issues
5/25/2019
CSE 572, CBS598: Data Mining by H. Liu

3. CSE 572, CBS598: Data Mining by H. Liu
Data and its format
Data
attribute-value pairs
with/without class
Data type
continuous/discrete
nominal
Data format
flat
5/25/2019
CSE 572, CBS598: Data Mining by H. Liu

4. CSE 572, CBS598: Data Mining by H. Liu
Sample data
5/25/2019
CSE 572, CBS598: Data Mining by H. Liu

5. Induction from databases
Inferring knowledge from data
The task of deduction
infer information that is a logical consequence of querying a database
Who conducted this class before?
Which courses are attended by Mary?
Deductive databases: extending the RDBMS
RDBMS - relational database management systems
RDBMS offer simple operators for the deduction of information, such as join
5/25/2019
CSE 572, CBS598: Data Mining by H. Liu

6. CSE 572, CBS598: Data Mining by H. Liu
Classification
It is one type of induction
data with class labels
Examples -
If weather is rainy then no golf If
Induction is different from deduction and DBMS does not not support induction;
The result of induction is higher-level information or knowledge: general statements about data
There are many approaches. Refer to the lecture notes for CS3244 available at the Co-Op.
We focus on three approaches here, other examples:
Other approaches
Instance-based learning
other neural networks
Concept learning (Version space, Focus, Aq11, …)
Genetic algorithms
Reinforcement learning
5/25/2019
CSE 572, CBS598: Data Mining by H. Liu

7. CSE 572, CBS598: Data Mining by H. Liu
Different approaches
There exist many techniques
Decision trees
Neural networks
K-nearest neighbors
Naïve Bayesian classifiers
Support Vector Machines
Ensemble methods
Co-training
and many more ...
5/25/2019
CSE 572, CBS598: Data Mining by H. Liu

8. CSE 572, CBS598: Data Mining by H. Liu
A decision tree
Outlook
Humidity
Wind
sunny
overcast
rain
YES
high
normal
strong
weak NO
Issues
How to build such a tree from the data?
What are the criteria for performance measurement
correctness
conciseness
What are the key components?
test
stopping criterion
5/25/2019
CSE 572, CBS598: Data Mining by H. Liu

9. Inducing a decision tree
There are many possible trees
let’s try it on the golfing data
How to find the most compact one
that is consistent with the data?
Why the most compact?
Occam’s razor principle
Issue of efficiency w.r.t. optimality
5/25/2019
CSE 572, CBS598: Data Mining by H. Liu

10. CSE 572, CBS598: Data Mining by H. Liu
Information gain
and
Entropy -
Information gain - the difference between the node before and after splitting
5/25/2019
CSE 572, CBS598: Data Mining by H. Liu

11. Building a compact tree
The key to building a decision tree - which attribute to choose in order to branch.
The heuristic is to choose the attribute with the maximum IG.
Another explanation is to reduce uncertainty as much as possible.
5/25/2019
CSE 572, CBS598: Data Mining by H. Liu

12. CSE 572, CBS598: Data Mining by H. Liu
Learn a decision tree
Outlook
sunny
overcast
rain
Humidity
Wind
YES
high
normal
strong
weak NO
YES NO
YES
5/25/2019
CSE 572, CBS598: Data Mining by H. Liu

13. Issues of Decision Trees
Number of values of an attribute
Your solution?
When to stop
Data fragmentation problem
Any solution?
Mixed data types
Scalability
5/25/2019
CSE 572, CBS598: Data Mining by H. Liu

14. CSE 572, CBS598: Data Mining by H. Liu
Rules and Tree stumps
Generating rules from decision trees
One path is a rule
We can do better. Why?
Tree stumps and 1R
For each attribute value, determine a default class (#of values = # of rules)
Calculate the # of errors for each rule
Find # of errors for that attribute’s rule set
Choose one rule set that has the least # of errors
5/25/2019
CSE 572, CBS598: Data Mining by H. Liu

15. CSE 572, CBS598: Data Mining by H. Liu
K-Nearest Neighbor
One of the most intuitive classification algorithm
An unseen instance’s class is determined by its nearest neighbor
The problem is it is sensitive to noise
Instead of using one neighbor, we can use k neighbors
5/25/2019
CSE 572, CBS598: Data Mining by H. Liu

16. CSE 572, CBS598: Data Mining by H. Liu
K-NN
New problems
How large should k be
lazy learning – does it learn?
large storage
A toy example (noise, majority)
How good is k-NN?
How to compare
Speed
Accuracy
5/25/2019
CSE 572, CBS598: Data Mining by H. Liu

17. Naïve Bayes Classifier
This is a direct application of Bayes’ rule
P(C|X) = P(X|C)P(C)/P(X)
X - a vector of x1,x2,…,xn
That’s the best classifier we can build
But, there are problems
There are only a limited number of instances
How to estimate P(x|C)
5/25/2019
CSE 572, CBS598: Data Mining by H. Liu

18. CSE 572, CBS598: Data Mining by H. Liu
NBC (2)
Assume conditional independence between xi’s
We have
P(C|x) ≈ P(x1|C) P(xi|C) (xn|C)P(C)
An example
How good is it in reality?
5/25/2019
CSE 572, CBS598: Data Mining by H. Liu

19. CSE 572, CBS598: Data Mining by H. Liu
No Free Lunch
If the goal is to obtain good generalization performance, there are no context-independent or usage-independent reasons to favor one learning or classification method over another
What does it indicate?
Source: Pattern Classification, 2nd Edition
5/25/2019
CSE 572, CBS598: Data Mining by H. Liu

20. CSE 572, CBS598: Data Mining by H. Liu
Ensemble Methods
Model generation
Bagging (Bootstrap Aggregating)
Boosting
Model combination
Majority voting
Meta learning
Stacking (using different types of classifiers)
5/25/2019
CSE 572, CBS598: Data Mining by H. Liu

21. Classification via Neural Networks
Squash 
A perceptron
5/25/2019
CSE 572, CBS598: Data Mining by H. Liu

22. What can a perceptron do?
Neuron as a computing device
To separate a linearly separable points
Nice things about a perceptron
distributed representation
local learning
weight adjusting
5/25/2019
CSE 572, CBS598: Data Mining by H. Liu

23. CSE 572, CBS598: Data Mining by H. Liu
Linear threshold unit
Basic concepts: projection, thresholding
W vectors evoke 1
W = [.11 .6]
L= [.7 .7] .5
5/25/2019
CSE 572, CBS598: Data Mining by H. Liu

24. E.g. 1: solution region for AND problem
Find a weight vector that satisfies all the constraints
AND problem
0 0 0
0 1 0
1 0 0
1 1 1
5/25/2019
CSE 572, CBS598: Data Mining by H. Liu

25. E.g. 2: Solution region for XOR problem?
0 0 0
0 1 1
1 0 1
1 1 0
5/25/2019
CSE 572, CBS598: Data Mining by H. Liu

26. Learning by error reduction
Perceptron learning algorithm
If the activation level of the output unit is 1 when it should be 0, reduce the weight on the link to the ith input unit by r*Li, where Li is the ith input value and r a learning rate
If the activation level of the output unit is 0 when it should be 1, increase the weight on the link to the ith input unit by r*Li
Otherwise, do nothing
5/25/2019
CSE 572, CBS598: Data Mining by H. Liu

27. Multi-layer perceptrons
Using the chain rule, we can back-propagate the errors for a multi-layer perceptrons.
Output layer
Hidden layer
Differences between DT and NN
Speed
Accuracy
Comprehensibility
Which one to use
Many successful applications of both approaches
Input layer
5/25/2019
CSE 572, CBS598: Data Mining by H. Liu