1. Machine Learning
Lecture 2:
Decision Tree Learning

2. Classifiaction A1 A2
target x1
TRUE x2
FALSE

3. Decision Trees
Decision trees are powerful and popular tools for classification and prediction. The attractiveness of decision trees is due to the fact that, in contrast to neural networks, decision trees represent rules. Rules can readily be expressed so that humans can understand them or even directly used in a database access language like SQL so that records falling into a particular category may be retrieved.

4. Decision Tree Representation
A decision tree is an arrangement of tests that provides an appropriate classification at every step in an analysis.
"In general, decision trees represent a disjunction of conjunctions of constraints on the attribute-values of instances. Each path from the tree root to a leaf corresponds to a conjunction of attribute tests, and the tree itself to a disjunction of these conjunctions" (Mitchell, 1997, p.53).
More specifically, decision trees classify instances by sorting them down the tree from the root node to some leaf node, which provides the classification of the instance. Each node in the tree specifies a test of some attribute of the instance, and each branch descending from that node corresponds to one of the possible values for this attribute.

5. Decision Tree Representation
An instance is classified by starting at the root node of the decision tree, testing the attribute specified by this node, then moving down the tree branch corresponding to the value of the attribute. This process is then repeated at the node on this branch and so on until a leaf node is reached.
Diagram
Each non-leaf node is connected to a test that splits its set of possible answers into subsets corresponding to different test results.
Each branch carries a particular test result's subset to another node.
Each node is connected to a set of possible answers.

6. Decision Tree Representation

7. Play Tennis Dataset Day Outlook Temp. Humidity Wind Play Tennis D1
Sunny
Hot
High
Weak No D2
Sunny
Hot
High
Strong No D3
Overcast
Hot
High
Weak
Yes D4
Rain
Mild
High
Weak
Yes D5
Rain
Cool
Normal
Weak
Yes D6
Rain
Cool
Normal
Strong No D7
Overcast
Cool
Normal
Weak
Yes D8
Sunny
Mild
High
Weak No D9
Sunny
Cold
Normal
Weak
Yes
D10
Rain
Mild
Normal
Strong
Yes
D11
Sunny
Mild
Normal
Strong
Yes
D12
Overcast
Mild
High
Strong
Yes
D13
Overcast
Hot
Normal
Weak
Yes
D14
Rain
Mild
High
Strong No

8. Play Tennis Dataset This data set consists of
4 input discrete features:
Outlook  {Sunny, Overcast, Rain}
Temperature  {Hot, Mild, Cool, Cold}
Humidity  {High, Normal}
Wind  {Strong, Weak}
2 classes: playTennis  {Yes, No}

9. Decision Tree for PlayTennis
Outlook
Sunny
Overcast
Rain
Humidity
Yes
Wind
High
Normal
Strong
Weak No
Yes No
Yes

10. Decision Tree for PlayTennis
Outlook
Sunny
Overcast Rain
Humidity
Each internal node tests an attribute
High Normal
Each branch corresponds to an attribute value node
No Yes
Each leaf node assigns a classification

11. Decision Tree for PlayTennis
Outlook Temperature Humidity Sunny Hot High
Wind PlayTennis Weak ?
Outlook
Sunny
Overcast
Rain
Humidity
Yes
Wind
High
Normal
Strong
Weak No
Yes No
Yes

12. Decision Tree for Conjunction
Outlook=Sunny  Wind=Weak
Outlook
Sunny
Overcast
Rain
Wind No No
Strong
Weak No
Yes

13. Decision Tree for Disjunction
Outlook=Sunny  Wind=Weak
Outlook
Sunny Overcast
Rain
Yes
Wind
Wind
Strong Weak
Strong Weak No
Yes No
Yes

14. Decision Tree for XOR Outlook=Sunny XOR Wind=Weak Outlook Sunny
Overcast
Rain
Wind
Wind
Wind
Strong
Weak
Strong
Weak
Strong
Weak
Yes No No
Yes No
Yes

15. Decision Tree
• decision trees represent disjunctions of conjunctions Outlook
Sunny
Overcast
Rain
Humidity
Yes
Wind
High
Normal
Strong
Weak No
Yes No
Yes
(Outlook=Sunny  Humidity=Normal) 
(Outlook=Overcast) 
(Outlook=Rain  Wind=Weak)

16. When to consider Decision Trees
Decision tree learning is generally best suited to problems with the following characteristics
n Instances are represented by attribute-value pairs
n Target function is discrete valued
n Disjunctive hypothesis may be required
n Possibly noisy training data
n Missing attribute values
n Examples:
n Medical diagnosis
n Credit risk analysis
n Object classification for robot manipulator

17. Top-Down Induction of Decision Trees (ID3)

18. Which Attribute is ”best”?
True False
True False
[21+, 5-] [8+, 30-]
[18+, 33-] [11+, 2-]

19. Entropy n S is a sample of training examples
n p+ is the proportion of positive examples
n p- is the proportion of negative examples
n Entropy measures the impurity of S

20. Conditional Entropy
the conditional entropy (or equivocation) quantifies the amount of information needed to describe the outcome of a random variable X given that the value of another random variable Y is known

21. Mutual Information
the mutual information of two random variables is a measure of the mutual dependence or the amount of shared information between the two random variables.

22. Information Gain I(S; Ai) = H(S) - H(S| Ai)
Gain(S, Ai): expected reduction in entropy (impurity) due to splitting region S on a discrete attribute Ai
I(S; Ai) = H(S) - H(S| Ai)
Gain(S, Ai)=Entropy(S) - vvalues(Ai) |Siv|/|S| Entropy(S | Ai=v)
|S|= number of training examples in region S
|Siv|= number of training examples in region S with Ai =v
[29+,35-] A 1=?
A2=? [29+,35-]
True False [21+, 5-] [8+, 30-]
True False [18+, 33-] [11+, 2-]
Entropy(S) = Entropy([29+,35-])
= -29/64 log2 29/ /64 log2 35/64 = 0.99

23. Information Gain
Entropy([18+,33-]) = 0.94 Entropy([8+,30-]) = 0.62 Gain(S, A2)=Entropy(S)
Entropy([21+,5-]) = 0.71 Entropy([8+,30-]) = 0.74 Gain(S, A1)=Entropy(S)
-51/64*Entropy(S1=[18+,33-])
-26/64*Entropy(S1=[21+,5-])
-13/64*Entropy(S2=[11+,2-]) =0.12
-38/64*Entropy(S2=[8+,30-]) =0.27
[29+,35-] A 1=?
A2=? [29+,35-]
True False [21+, 5-] [8+, 30-]
True False [18+, 33-] [11+, 2-]
The best attribute is the attribute with maximum information gain
A* = arg max Gain(S, Ai)

24. Training Examples Day Outlook Temp. Humidity Wind Play Tennis D1 Sunny
Hot
High
Weak No D2
Sunny
Hot
High
Strong No D3
Overcast
Hot
High
Weak
Yes D4
Rain
Mild
High
Weak
Yes D5
Rain
Cool
Normal
Weak
Yes D6
Rain
Cool
Normal
Strong No D7
Overcast
Cool
Normal
Weak
Yes D8
Sunny
Mild
High
Weak No D9
Sunny
Cold
Normal
Weak
Yes
D10
Rain
Mild
Normal
Strong
Yes
D11
Sunny
Mild
Normal
Strong
Yes
D12
Overcast
Mild
High
Strong
Yes
D13
Overcast
Hot
Normal
Weak
Yes
D14
Rain
Mild
High
Strong No

25. Selecting the Next Attribute
S=[9+,5-] E=0.940 Humidity
S=[9+,5-] E=0.940 Wind
High Normal
Weak Strong
[3+, 4-] [6+, 1-]
[6+, 2-] [3+, 3-]
E=0.985 E=0.592
E=0.811 E=1.0
Gain(S,Wind)
Gain(S,Humidity) =0.940-(7/14)*0.985
=0.940-(8/14)* (6/14)*1.0
- (7/14)*0.592 =0.151
=0.048

26. Selecting the Next Attribute
Outlook
Over cast
Sunny [2+, 3-] E=0.971
Rain
[3+, 2-] E=0.971
[4+, 0] E=0.0
Gain(S,Outlook)
=0.940-(5/14)*0.971
-(4/14)*0.0 - (5/14)* =0.247

27. ID3 Algorithm [D1,D2,…,D14] [9+,5-] Outlook Sunny Overcast Rain
Ssunny=[D1,D2,D8,D9,D11]
[D3,D7,D12,D13][D4,D5,D6,D10,D14]
[2+,3-]
[4+,0-] [3+,2-]
Yes ? ?
Gain(Ssunny , Humidity)=0.970-(3/5) /5(0.0) = 0.970
Gain(Ssunny , Temp.)=0.970-(2/5)0.0 -2/5(1.0)-(1/5)0.0 = Gain(Ssunny , Wind)=0.970= -(2/5) /5(0.918) = 0.019

28. ID3 Algorithm Outlook Sunny Overcast Rain Humidity Yes Wind
[D3,D7,D12,D13]
High Normal Strong
Weak
Yes [D4,D5,D10]
No Yes No
[D6,D14] [D1,D2] [D8,D9,D11]

29. Hypothesis Space Search ID3
+ - + A2 A1
- - A2 A2 - - A3 A4
+ - - +

30. Issues in Decision Tree Learning
Avoiding Overfitting the Data
Incorporating Continuous-Valued Attributes
Alternative Measures for Selecting Attributes
Handling Training Examples with Missing Attribute Values

31. Overfitting Consider error of hypothesis h over
n Training data: errortrain(h)
n Entire distribution D of data: errorD(h)
Hypothesis hH overfits training data if there is an alternative hypothesis h’H such that
errortrain(h) < errortrain(h’)
and
errorD(h) > errorD(h’)

32. Overfitting in Decision Trees
error
validation set
training set
optimal #nodes
#nodes

33. Overfitting in Function Approximation

34. Generalization
n Generalization means the ability of a learning algorithm to generalize from training examples to previously unseen examples and to predict their target value correctly.
n The training error is no indicator of how well the learner is able to predict future examples.
n Therefore, partition the data set into training and validation set.
n Train the learning algorithm on the training data only and use the validation set to predict the accuracy/error of the learner on future unseen instances.

35. Cross-Validation induced by a supervised learning algorithm
n Estimate the accuracy of a hypothesis
induced by a supervised learning algorithm
n Predict the accuracy of a hypothesis over future unseen instances
n Select the optimal hypothesis from a given set of alternative hypotheses
n Pruning decision trees
n Model selection
n Feature selection
n Combining multiple classifiers (boosting)

36. Cross-Validation each time it is trained on D\Di and tested on Di
n k-fold cross-validation splits the data set D into k mutually exclusive subsets D1,D2,…,Dk D1 D2 D3 D4
n Train and test the learning algorithm k times,
each time it is trained on D\Di and tested on Di D1 D2 D3 D4 D1 D2 D3 D4 D1 D2 D3 D4 D1 D2 D3 D4
acccv = 1/n  (vi,yi)D d(I(D\Di,vi),yi)

37. Cross-Validation
n Uses all the data for training and testing
n Complete k-fold cross-validation splits the
dataset of size m in all (m over m/k) possible ways (choosing m/k instances out of m)
n Leave n-out cross-validation sets n instances aside for testing and uses the remaining ones for training (leave one-out is equivalent to n- fold cross-validation)
n In stratified cross-validation, the folds are
stratified so that they contain approximately the same proportion of labels as the original data set

38. Avoid Overfitting How can we avoid overfitting?
n Stop growing when data split not statistically significant
n Grow full tree then post-prune
n Minimum description length (MDL): Minimize:
size(tree) + size(misclassifications(tree))

39. 1. Reduced-Error Pruning
Split data into training and validation set Do until further pruning is harmful:
1. Evaluate impact on validation set of pruning each possible node (plus those below it)
2. Greedily remove the one that most improves the validation set accuracy
Produces smallest version of most accurate subtree

40. 2. Rule-Post Pruning
Method used in C4.5

41. Converting a Tree to Rules
Outlook
Sunny
Overcast
Rain
Humidity
Yes
Wind
High
Normal
Strong
Weak No
Yes No
Yes
R1: If (Outlook=Sunny)  (Humidity=High) Then PlayTennis=No
R2: If (Outlook=Sunny)  (Humidity=Normal) Then PlayTennis=Yes
R3: If (Outlook=Overcast) Then PlayTennis=Yes
R4: If (Outlook=Rain) 
(Wind=Strong) Then PlayTennis=No
R5: If (Outlook=Rain) 
(Wind=Weak) Then PlayTennis=Yes

42. Incorporating Continuous Valued Attributes
Create a discrete attribute to test continuous
Sort the examples by the continuous attribute
Identify adjacent examples that differ in their target
Attribute, evaluate each candidate threshold
Temperatur 40 48 60
0 0 C C
0 0 0 C 72 C 80 C
900C
PlayTennis No No
Yes
Yes
Yes No
(Temperature > threshold) = {true, false} Where to set the threshold?
Temperature >54 and Temperature >85

43. Iris Flower Classification

44. Iris Flower Classification

45. Attributes with many Values
n Problem: if an attribute has many values, maximizing InformationGain will select it.
n E.g.: Imagine using Date= as attribute perfectly splits the data into subsets of size 1 Use GainRatio instead of information gain as criteria: GainRatio(S,A) = Gain(S,A) / SplitInformation(S,A)
SplitInformation(S,A) = -Si=1..c |Si|/|S| log2 |Si|/|S|
Where Si is the subset for which attribute A has the value vi

46. Handling Attributes with Different Costs
Consider:
n Medical diagnosis : blood test costs 1000 SEK
n Robotics: width_from_one_feet has cost 23 secs. How to learn a consistent tree with low expected cost?
Replace Gain by :
Gain2(S,A)/Cost(A)
2Gain(S,A)-1/(Cost(A)+1)w w [0,1]

47. Handling Examples with Missing Attribute Values
What if some examples missing values of some attribute A?
Estimate the missing value based on other examples with known values
n If node n tests A, assign most common value of A among other training examples at node n.
n Assign most common value of A among other examples at node n with the same target attribute value
n Assign probability pi to each possible value vi of A
n Assign fraction pi of example to each descendant in tree
Classify new examples in the same fashion

48. References
T. Mitchell, "Decision Tree Learning", in T. Mitchell, Machine Learning,
The McGraw-Hill Companies, Inc., 1997, pp