1. Decision Trees and Decision Tree Learning Philipp Kärger

3. Outline: Decision Trees Decision Tree Learning Overfitting
ID3 Algorithm
Which attribute to split on?
Some examples
Overfitting
Where to use Decision Trees?

5. Decision tree representation for PlayTennis
Outlook
Sunny
Overcast
Rain
Humidity
Yes
Wind
Normal
Strong
Weak
High No
Yes No
Yes

6. Decision tree representation for PlayTennis
Outlook
Attribute
Sunny
Overcast
Rain
Humidity
Yes
Wind
Normal
Strong
Weak
High No
Yes No
Yes

7. Decision tree representation for PlayTennis
Outlook
Value
Sunny
Overcast
Rain
Humidity
Yes
Wind
Normal
Strong
Weak
High No
Yes No
Yes

8. Decision tree representation for PlayTennis
Outlook
Classification
Sunny
Overcast
Rain
Humidity
Yes
Wind
Normal
Strong
Weak
High No
Yes No
Yes

9. Decision tree representation for PlayTennis
Outlook
Sunny
Overcast
Rain
Humidity
Yes
Wind
Normal
Strong
Weak
High No
Yes No
Yes

10. PlayTennis: Other representations
Logical expression for PlayTennis=Yes:
(Outlook=Sunny  Humidity=Normal)  (Outlook=Overcast)  (Outlook=Rain  Wind=Weak)
If-then rules
IF Outlook=Sunny  Humidity=Normal THEN PlayTennis=Yes
IF Outlook=Overcast THEN PlayTennis=Yes
IF Outlook=Rain  Wind=Weak THEN PlayTennis=Yes
IF Outlook=Sunny  Humidity=High THEN PlayTennis=No
IF Outlook=Rain  Wind=Strong THEN PlayTennis=Yes

11. Decision Trees - Summary
a model of a part of the world
allows us to classify instances (by performing a sequence of tests)
allows us to predict classes of (unseen) instances
understandable by humans (unlike many other representations)

12. Decision Tree Learning

13. Goal: Learn from known instances how to classify unseen instances
by means of building and exploiting a Decision Tree
supervised or unsupervised learning?

14. Classification Task
Application: classification of medical patients by their disease
seen patients
Decision Tree
unseen patients
rules telling which attributes of the patient indicates a disease
check attributes of an unseen patient

15. Exercise: create two decision trees
sunny
odd day
play tennis
yes no

16. Basic algorithm: ID3 (simplified)
ID3 = Iterative Dichotomiser 3
- given a goal class to build the tree for
- create a root node for the tree
- if all examples from the test set belong to the same goal class C then label the root with C
- else
select the ‘most informative’ attribute A
split the training set according to the values V1..Vn of A
recursively build the resulting subtrees T1 … Tn
generate decision tree T:
A1=weather
A2=day
happy
sun
odd
yes
rain no
even A
Humidity vn v1
...
Low
High T1
... Tn No
Yes

17. finding the right attribute A to split on is tricky
lessons learned:
there is always more than one decision tree
finding the “best” one is NP complete
all the known algorithms use heuristics
finding the right attribute A to split on is tricky

18. Decision trees -Binary decision trees
Since each inequality that is used to split the input space is only based on one input variable.
Each node draws a boundary that can be geometrically interpreted as a hyperplane perpendicular to the axis. B C

19. Search heuristics in ID3
Which attribute should we split on?
Need a heuristic
Some function gives big numbers for “good” splits
Want to get to “pure” sets
How can we measure “pure”?
odd
even
sunny
rain

20. E(S) = - p+ log2p+ - p- log2p-
Entropy
S - example set, C1,...,CN - classes
Entropy E(S) – measure of the impurity of training set S
pc = probability of class Cc
Entropy in binary classification problems
E(S) = - p+ log2p+ - p- log2p-

21. Entropy E(S) = - p+ log2p+ - p- log2p-
The entropy function relative to a Boolean classification, as the proportion p+ of positive examples varies between 0 and 1

22. p+ ( - log2p+ ) + p- ( - log2p- ) = - p+ log2p+ - p- log2p-
What is entropy?
Entropy E(S) = expected amount of information (in bits) needed to assign a class to a randomly drawn object in S under the optimal, shortest-length code
Information theory: optimal length code assigns -log2p bits to a message having probability p
So, in binary classification problems, the expected number of bits to encode + or – of a random member of S is:
p+ ( - log2p+ ) + p- ( - log2p- ) = - p+ log2p+ - p- log2p-

23. PlayTennis: Training examples

24. PlayTennis: Entropy Training set S: 14 examples (9 pos., 5 neg.)
Notation: S = [9+, 5-]
E(S) = - p+ log2p+ - p- log2p-
Computing entropy, if probability is estimated by relative frequency
E([9+,5-]) = - (9/14) log2(9/14) - (5/14) log2(5/14) = 0.940

25. PlayTennis: Entropy E(S) = - p+ log2p+ - p- log2p-
E(9+,5-) = -(9/14) log2(9/14) - (5/14) log2(5/14) = 0.940
{D1,D2,D8,D9,D11} [2+, 3-] E=0.970
Sunny
Outlook?
Overcast
{D3,D7,D12,D13} [4+, 0-] E=0
Rain
{D4,D5,D6,D10,D14} [3+, 2-] E=0.970
[3+, 4-] E=0.985
High
Humidity?
Normal
[6+, 1-] E=0.592
[6+, 2-] E=0.811
Weak
Wind?
Strong
[3+, 3-] E=1.00

26. Maximizing Purity and Minimizing Disorder
Select attribute which partitions the learning set into subsets as “pure” as possible
Knowing the ``when’’ attribute values provides larger information gain than ``where’’.
Therefore the ``when’’ attribute should be chosen for testing prior to the ``where’’ attribute.

27. Splitting Based on INFO...
Information Gain:
Parent Node, p is split into k partitions;
ni is number of records in partition i
Measures Reduction in Entropy achieved because of the split. Choose the split that achieves most reduction (maximizes GAIN)
Used in ID3 and C4.5
Disadvantage: Tends to prefer splits that result in large number of partitions, each being small but pure.

28. Splitting Based on INFO...
Gain Ratio:
Parent Node, p is split into k partitions
ni is the number of records in partition i
Adjusts Information Gain by the entropy of the partitioning (SplitINFO). Higher entropy partitioning (large number of small partitions) is penalized!
Used in C4.5
Designed to overcome the disadvantage of Information Gain

29. Measuring Information: Entropy
The average amount of information I needed to classify an object is given by the entropy measure
For a two-class problem:
p(c) = probability of class Cc (sum over all classes)
Tell us how to measure the information needed for classification.
draw the formula to the board. Tell the intuition: measures the disorder in the dataset = measure of the impurity of training set S
entropy
p(c)

30. Information-Theoretic Approach
To classify an object, a certain information is needed
I, information
After we have learned the value of A, we only need some remaining amount of information to classify an object
Ires, residual information
The most ‘informative’ attribute is the one that minimizes Ires, i.e., maximizes the Gain:
Gain
Gain(A) = I – Ires(A)
What does it mean to have Ires = 0?
odd
even
sunny
rain

31. What is the entropy of the set of happy/unhappy days?
A1=weather
A2=day
happy
sun
odd
yes
rain no
even
odd
even
sunny
rain

32. Residual Information
After applying attribute A, S is partitioned into subsets according to values v of A
Ires represents the amount of information still needed to classify an instance
Ires is equal to weighted sum of the amounts of information for the subsets
p(c|v) = probability that an instance belongs to class C given that it belongs to v
=I(v)

33. What is Ires(A) if I split for “weather” and if I split for “day”?
A1=weather
A2=day
happy
sun
odd
yes
rain no
even
odd
even
sunny
rain
Ires(weather) = 0
Ires(day) = 1

34. Information Gain:
= the amount of information I rule out by splitting on attribute A:
Gain(A) = I – Ires(A)
= information in the current set minus the residual information after splitting
The most ‘informative’ attribute is the one that minimizes Ires, i.e., maximizes the Gain

35. Triangles and Squares

36. A set of classified objects
Triangles and Squares
Data Set:
A set of classified objects . . . . . .

37. Entropy 5 triangles 9 squares class probabilities
entropy of the data set . . . . . .

38. Entropy reduction by data set partitioning. .
red
yellow
green
Color?
Entropy reduction by data set partitioning . .

39. . . . . . . .
red
Color?
green
residual information .
yellow .

40. . . . . . . .
red
Information Gain
Color?
green .
yellow .

41. Information Gain of The Attribute
Attributes
Gain(Color) = 0.246
Gain(Outline) = 0.151
Gain(Dot) = 0.048
Heuristics: attribute with the highest gain is chosen
This heuristics is local (local minimization of impurity)

42. Gain(Outline) = 0.971 – 0 = 0.971 bits
red
Color?
green .
yellow .
Gain(Outline) = – 0 = bits
Gain(Dot) = – = bits

43. Gain(Outline) = 0.971 – 0.951 = 0.020 bits
red
Gain(Outline) = – = bits
Gain(Dot) = – 0 = bits
Color?
green .
yellow .
solid .
Outline?
dashed .

44. . . . Dot? . Color? . . . Outline? . red yes no green yellow solid
dashed .

45. Decision Tree . . . . . . Color Dot square Outline triangle square
red
green
yellow
Dot
square
Outline
yes no
dashed
solid
triangle
square
triangle
square

46. A Defect of Ires Ires favors attributes with many values
Such attribute splits S to many subsets, and if these are small, they will tend to be pure anyway
One way to rectify this is through a corrected measure of information gain ratio.
A1=weather
A2=day
happy
sun
yes
rain no

47. Information Gain Ratio
I(A) is amount of information needed to determine the value of an attribute A
Information gain ratio

48. Information Gain Ratio. . . . . . .
red
Color?
green
Information Gain Ratio .
yellow .

49. Information Gain and Information Gain Ratio

50. Overfitting (Example)

51. Overfitting Overfitting
Underfitting: when model is too simple, both training and test errors are large

52. Overfitting due to Noise
Decision boundary is distorted by noise point

53. Overfitting due to Insufficient Examples
Lack of data points in the lower half of the diagram makes it difficult to predict correctly the class labels of that region
- Insufficient number of training records in the region causes the decision tree to predict the test examples using other training records that are irrelevant to the classification task

54. Notes on Overfitting
Overfitting results in decision trees that are more complex than necessary
Training error no longer provides a good estimate of how well the tree will perform on previously unseen records

55. How to Address Overfitting
Idea: prune the tree so that it is not too specific
Two possibilities:
Pre-Pruning
- prune while building the tree
Post-Pruning
- prune after building the tree

56. How to Address Overfitting
Pre-Pruning (Early Stopping Rule)
Stop the algorithm before it becomes a fully-grown tree
More restrictive stopping conditions:
Stop if number of instances is less than some user-specified threshold
Stop if expanding the current node does not improve impurity measures (e.g., information gain).
Not successful in practice

57. How to Address Overfitting…
Post-pruning
Grow decision tree to its entirety
Trim the nodes of the decision tree in a bottom-up fashion
If generalization error improves after trimming, replace sub-tree by a leaf node.
Class label of leaf node is determined from majority class of instances in the sub-tree

58. Example of Post-Pruning
Training Error (Before splitting) = 10/30
Pessimistic error = ( )/30 = 10.5/30
Training Error (After splitting) = 9/30
Pessimistic error (After splitting)
= (9 + 4  0.5)/30 = 11/30
PRUNE!
Class = Yes 20
Class = No 10
Error = 10/30
Class = Yes 8
Class = No 4
Class = Yes 3
Class = No 4
Class = Yes 4
Class = No 1
Class = Yes 5
Class = No 1

59. Examples of Post-pruning
Case 1:
C0: 11
C1: 3
C0: 2
C1: 4
Optimistic error?
Pessimistic error?
Reduced error pruning?
Don’t prune for both cases
Don’t prune case 1, prune case 2
Case 2:
C0: 14
C1: 3
C0: 2
C1: 2
Depends on validation set

60. Oblique Decision Trees
x + y < 1
Class = +
Class =
Test condition may involve multiple attributes
More expressive representation
Finding optimal test condition is computationally expensive

61. Occam’s Razor
Given two models of similar generalization errors, one should prefer the simpler model over the more complex model
For complex models, there is a greater chance that it was fitted accidentally by errors in data
Therefore, one should prefer less complex models in general

62. When to use Decision Tree Learning?

63. Appropriate problems for decision tree learning
Classification problems
Characteristics:
instances described by attribute-value pairs
target function has discrete output values
training data may be noisy
training data may contain missing attribute values

64. Strengths can generate understandable rules
perform classification without much computation
can handle continuous and categorical variables
provide a clear indication of which fields are most important for prediction or classification

65. Weakness Not suitable for prediction of continuous attribute.
Perform poorly with many class and small data.
Computationally expensive to train.
At each node, each candidate splitting field must be sorted before its best split can be found.
In some algorithms, combinations of fields are used and a search must be made for optimal combining weights.
Pruning algorithms can also be expensive since many potential sub-trees must be formed and compared