Decision Trees Advanced Statistical Methods in NLP Ling572 January 10, 2012
Information Gain InfoGain(S,A): expected reduction in entropy due to A 2
Information Gain InfoGain(S,A): expected reduction in entropy due to A 3
Information Gain InfoGain(S,A): expected reduction in entropy due to A 4
Information Gain InfoGain(S,A): expected reduction in entropy due to A Select A with max InfoGain Resulting in lowest average entropy 5
Computing Average Entropy Disorder of class distribution on branch i Fraction of samples down branch i |S| instances Branch1 Branch 2 S a1 a S a1 b S a2 a S a2 b 6
Sunburn Example 7
Picking a Test Hair Color Blonde Red Brown Alex: N Pete: N John: N Emily: B Sarah: B Dana: N Annie: B Katie: N HeightWeightLotion Short Average Tall Alex:N Annie:B Katie:N Sarah:B Emily:B John:N Dana:N Pete:N Sarah:B Katie:N Light Average Heavy Dana:N Alex:N Annie:B Emily:B Pete:N John:N No Yes Sarah:B Annie:B Emily:B Pete:N John:N Dana:N Alex:N Katie:N 8
Entropy in Sunburn Example 9
S = [3B,5N] 10
Entropy in Sunburn Example S = [3B,5N] 11
Entropy in Sunburn Example Hair color = (4/8(-2/4 log 2/4 - 2/4log2/4) + 1/8*0 + 3/8 *0) = = Height = = Weight = = Lotion = = S = [3B,5N] 12
Picking a Test HeightWeightLotion Short Average Tall Annie:B Katie:N Sarah:B Dana:N Sarah:B Katie:N Light Average Heavy Dana:N Annie:B No Yes Sarah:B Annie:B Dana:N Katie:N 13
Entropy in Sunburn Example Height = 1-2/4(-1/2log1/2-1/2log1/2) + 1/4*0+1/4*0 = =0.5 Weight = 1-2/4(-1/2log1/2-1/2log1/2) +2/4(-1/2log1/2-1/2log1/2) = 1 Lotion = 1- 0 = 1 S=[2B,2N] 14
Building Decision Trees with Information Gain Until there are no inhomogeneous leaves 15
Building Decision Trees with Information Gain Until there are no inhomogeneous leaves Select an inhomogeneous leaf node 16
Building Decision Trees with Information Gain Until there are no inhomogeneous leaves Select an inhomogeneous leaf node Replace that leaf node by a test node creating subsets that yield highest information gain 17
Building Decision Trees with Information Gain Until there are no inhomogeneous leaves Select an inhomogeneous leaf node Replace that leaf node by a test node creating subsets that yield highest information gain Effectively creates set of rectangular regions Repeatedly draws lines in different axes 18
Alternate Measures Issue with Information Gain: 19
Alternate Measures Issue with Information Gain: Favors features with more values Option: 20
Alternate Measures Issue with Information Gain: Favors features with more values Option: Gain Ratio 21
Alternate Measures Issue with Information Gain: Favors features with more values Option: Gain Ratio S a : elements of S with value A=a 22
Overfitting Overfitting: Model fits the training data TOO well Fits noise, irrelevant details 23
Overfitting Overfitting: Model fits the training data TOO well Fits noise, irrelevant details Why is this bad? 24
Overfitting Overfitting: Model fits the training data TOO well Fits noise, irrelevant details Why is this bad? Harms generalization Fits training data too well, fits new data badly 25
Overfitting Overfitting: Model fits the training data TOO well Fits noise, irrelevant details Why is this bad? Harms generalization Fits training data too well, fits new data badly For model m, training_error(m), D_error(m) – D=all data 26
Overfitting Overfitting: Model fits the training data TOO well Fits noise, irrelevant details Why is this bad? Harms generalization Fits training data too well, fits new data badly For model m, training_error(m), D_error(m) – D=all data If overfit, for another model m’, training_error(m) < training_error(m’), but D_error(m) > D_error(m’) 27
Avoiding Overfitting Strategies to avoid overfitting: 28
Avoiding Overfitting Strategies to avoid overfitting: Early stopping: 29
Avoiding Overfitting Strategies to avoid overfitting: Early stopping: Stop when InfoGain < threshold Stop when number of instances < threshold Stop when tree depth > threshold Post-pruning 30
Avoiding Overfitting Strategies to avoid overfitting: Early stopping: Stop when InfoGain < threshold Stop when number of instances < threshold Stop when tree depth > threshold Post-pruning Grow full tree and remove branches Which is better? 31
Avoiding Overfitting Strategies to avoid overfitting: Early stopping: Stop when InfoGain < threshold Stop when number of instances < threshold Stop when tree depth > threshold Post-pruning Grow full tree and remove branches Which is better? Unclear, both used. For some applications, post-pruning better 32
Post-Pruning Divide data into Training set: used to build the original tree Validation set: used to perform pruning 33
Post-Pruning Divide data into Training set: used to build the original tree Validation set: used to perform pruning Build decision tree based on training data 34
Post-Pruning Divide data into Training set: used to build the original tree Validation set: used to perform pruning Build decision tree based on training data Until pruning does not reduce validation set performance Compute perf. for pruning each nodes (& its children) Greedily remove nodes that do not reduce VS performance 35
Post-Pruning Divide data into Training set: used to build the original tree Validation set: used to perform pruning Build decision tree based on training data Until pruning does not reduce validation set performance Compute perf. for pruning each nodes (& its children) Greedily remove nodes that do not reduce VS performance Yields smaller tree with best performance 36
Performance Measures Compute accuracy on: 37
Performance Measures Compute accuracy on: Validation set k-fold cross-validation 38
Performance Measures Compute accuracy on: Validation set k-fold cross-validation Weighted classification error cost: Weight some types of errors more heavily 39
Performance Measures Compute accuracy on: Validation set k-fold cross-validation Weighted classification error cost: Weight some types of errors more heavily Minimum description length: 40
Performance Measures Compute accuracy on: Validation set k-fold cross-validation Weighted classification error cost: Weight some types of errors more heavily Minimum description length: Favor good accuracy on compact models MDL = error(tree) + model_size(tree) 41
Rule Post-Pruning Convert tree to rules 42
Rule Post-Pruning Convert tree to rules Prune rules independently 43
Rule Post-Pruning Convert tree to rules Prune rules independently Sort final rule set 44
Rule Post-Pruning Convert tree to rules Prune rules independently Sort final rule set Probably most widely used method (toolkits) 45
Modeling Features Different types of features need different tests Binary: Test branches on 46
Modeling Features Different types of features need different tests Binary: Test branches on true/false Discrete: Branches 47
Modeling Features Different types of features need different tests Binary: Test branches on true/false Discrete: Branches for each discrete value Continuous? 48
Modeling Features Different types of features need different tests Binary: Test branches on true/false Discrete: Branches for each discrete value Continuous? Need to discretize 49
Modeling Features Different types of features need different tests Binary: Test branches on true/false Discrete: Branches for each discrete value Continuous? Need to discretize Enumerate all values 50
Modeling Features Different types of features need different tests Binary: Test branches on true/false Discrete: Branches for each discrete value Continuous? Need to discretize Enumerate all values not possible or desirable Pick value x Branches: value = x 51
Modeling Features Different types of features need different tests Binary: Test branches on true/false Discrete: Branches for each discrete value Continuous? Need to discretize Enumerate all values not possible or desirable Pick value x Branches: value = x How can we pick split points? 52
Picking Splits Need split useful, sufficient split points What’s a good strategy? 53
Picking Splits Need split useful, sufficient split points What’s a good strategy? Approach: Sort all values for the feature in training data 54
Picking Splits Need split useful, sufficient split points What’s a good strategy? Approach: Sort all values for the feature in training data Identify adjacent instances of different classes Candidate split points between those instances 55
Picking Splits Need split useful, sufficient split points What’s a good strategy? Approach: Sort all values for the feature in training data Identify adjacent instances of different classes Candidate split points between those instances Select candidate with highest information gain 56
Advanced Topics Missing features: What do you do if an instance lacks a feature value?
Advanced Topics Missing features: What do you do if an instance lacks a feature value? Feature costs: How do you model different costs for features?
Advanced Topics Missing features: What do you do if an instance lacks a feature value? Feature costs: How do you model different costs for features? Regression trees: How do you build trees with real-valued predictions?
Missing Features Problem: What if you don’t know the value for a feature?
Missing Features Problem: What if you don’t know the value for a feature? Not binary presence/absence
Missing Features Problem: What if you don’t know the value for a feature? Not binary presence/absence Create synthetic value:
Missing Features Problem: What if you don’t know the value for a feature? Not binary presence/absence Create synthetic value: ‘blank’: allow a distinguished value ‘blank’
Missing Features Problem: What if you don’t know the value for a feature? Not binary presence/absence Create synthetic value: ‘blank’: allow a distinguished value ‘blank’ most common value: assign most common value of feature in training set at that node
Missing Features Problem: What if you don’t know the value for a feature? Not binary presence/absence Create synthetic value: ‘blank’: allow a distinguished value ‘blank’ most common value: assign most common value of feature in training set at that node common value by class: assign most common value of feature in training set at that node for that class
Missing Features Problem: What if you don’t know the value for a feature? Not binary presence/absence Create synthetic value: ‘blank’: allow a distinguished value ‘blank’ most common value: assign most common value of feature in training set at that node common value by class: assign most common value of feature in training set at that node for that class Assign prob p i to each possible value v i of A Assign a fraction (p i ) of example to each descendant in tree
Features with Cost Issue: Obtaining a value for a feature can be expensive
Features with Cost Issue: Obtaining a value for a feature can be expensive i.e. Medical diagnosis: Feature value is result of some diagnostic test
Features with Cost Issue: Obtaining a value for a feature can be expensive i.e. Medical diagnosis: Feature value is result of some diagnostic test Goal: Build best tree with lowest expected cost Approach: Modify feature selection
Features with Cost Issue: Obtaining a value for a feature can be expensive i.e. Medical diagnosis: Feature value is result of some diagnostic test Goal: Build best tree with lowest expected cost Approach: Modify feature selection Replace information gain with measure including cost Tan & Schlimmer (1990)
Regression Trees Leaf nodes provide real-valued predictions
Regression Trees Leaf nodes provide real-valued predictions i.e. level of sunburn, rather than binary Height of pitch accent, rather than +/-
Regression Trees Leaf nodes provide real-valued predictions i.e. level of sunburn, rather than binary Height of pitch accent, rather than +/- Leaf nodes provide Value or linear function E.g. mean of nodes on that branch
Regression Trees Leaf nodes provide real-valued predictions i.e. level of sunburn, rather than binary Height of pitch accent, rather than +/- Leaf nodes provide Value or linear function E.g. mean of nodes on that branch What measure of inhomogeneity? Variance, standard deviation,…
Decision Trees: Strengths Simplicity (conceptual) Feature selection Handling of diverse features: Binary, discrete, continuous Fast decoding Perspicuousness (Interpretability) 75
Decision Trees: Weaknesses Features Assumed independent If want group effect, must model explicitly E.g. make new feature AorB Feature tests conjunctive Inefficiency of training: complex, multiple calculations Lack of formal guarantees: greedy training, non-optimal trees Inductive bias: Rectangular decision boundaries Sparse data problems: splits at each node Lack of stability/robustness 76
Decision Trees Train: Build tree by forming subsets of least disorder Predict: Traverse tree based on feature tests Assign leaf node sample label Pros: Robust to irrelevant features, some noise, fast prediction, perspicuous rule reading Cons: Poor feature combination, dependency, optimal tree build intractable 77