Today’s Topics Dealing with Noise Overfitting (the key issue in all of ML) A ‘Greedy’ Algorithm for Pruning D-Trees Generating IF-THEN Rules from D-Trees Rule Pruning 9/22/15CS Fall 2015 (© Jude Shavlik), Lecture 6, Week 31

CS Fall 2015 (© Jude Shavlik), Lecture 6, Week 2 Noise: Major Issue in ML Worst Case of Noise +, - at same point in feature space Causes of Noise 1. Too few features (“hidden variables”) or too few possible values 2. Incorrectly reported/measured/judged feature values 3. Mis-classified instances 9/22/152

CS Fall 2015 (© Jude Shavlik), Lecture 6, Week 2 Noise - Major Issue in ML (cont.) Overfitting Producing an ‘awkward’ concept because of a few ‘noisy’ points Bad performance on future ex’s?Better performance? 9/22/153

CS Fall 2015 (© Jude Shavlik), Lecture 6, Week 2 Overfitting Viewed in Terms of Function-Fitting (can exactly fit N points with an N-1 degree polynomial) Data = Red Line + Noise Model f(x) x Underfitting? Overfitting? 9/22/154

CS Fall 2015 (© Jude Shavlik), Lecture 6, Week 2 Definition of Overfitting Assuming large enough test set so that it is representative, concept C overfit the training data if there exists a simpler concept S so that but > < Training set accuracy of C Training set accuracy of S Test set accuracy of C Test set accuracy of S 9/22/155

CS Fall 2015 (© Jude Shavlik), Lecture 6, Week 2 Remember! It is easy to learn/fit the training data What’s hard is generalizing well to future (‘test set’) data! Overfitting avoidance (reduction, really) is the key issue in ML Easy to think ‘spurious correlations’ are meaningful signals 9/22/156

See a Pattern? 9/22/15CS Fall 2015 (© Jude Shavlik), Lecture 6, Week 2 Lecture 1, Slide 7 The first 10 digits of Pi: What comes next in Pi? 3 (already used) After that? 5 “35” rounds to “4” (in fractional part of number) “4” has since been added! Picture taken (by me) June 2015 in Lambeau Field Atrium, Green Bay, WI Presumably a ‘spurious correlation’

CS Fall 2015 (© Jude Shavlik), Lecture 6, Week 2 Can One Underfit? Sure, if not fully fitting the training set Eg, just return majority category (+ or -) in the trainset as the learned model But also if not enough data to illustrate important distinctions Eg, color may be important, but all examples seen are red, so no reason to include color and make more complex model 9/22/158

CS Fall 2015 (© Jude Shavlik), Lecture 6, Week 2 Overfitting + Noise Using the strict definition of overfitting presented earlier, is it possible to overfit noise-free data? (Remember: overfitting the key ML issue, not just a decision-tree topic) 9/22/159

CS Fall 2015 (© Jude Shavlik), Lecture 6, Week 2 Example of Overfitting Noise-Free Data Let –Correct concept = A  B –Feature C be true 50% of the time, for both + and – examples –Prob(pos example) = 0.66 –Training set +: A B C D E, A B C ¬D E, A B C D ¬E -: A ¬B ¬C D ¬E, ¬A B ¬C ¬D E 9/22/1510

CS Fall 2015 (© Jude Shavlik), Lecture 6, Week 2 Example (concluded) Tree Trainset Accuracy TestSet Accuracy 100%50% Pruned 60%66% C + - FT + 9/22/1511

CS Fall 2015 (© Jude Shavlik), Lecture 6, Week 2 ID3 & Noisy Data To avoid overfitting, could allow splitting to stop before all ex’s are of one class –Early stopping was Quinlan’s original idea Stop if further splitting not justified by a statistical test (just skim text’s material on the  2 test) –But post-pruning now seen as better More robust to weaknesses of greedy algo’s (eg, post-pruning benefits from seeing the full tree; a node may look bad when building tree, but not in hindsight) 9/22/1512

CS Fall 2015 (© Jude Shavlik), Lecture 6, Week 2 ID3 & Noisy Data (cont.) Recap: Build complete tree, then use some ‘spare’ (tuning) examples to decide which parts of tree can be pruned - called Reduced [tuneset] Error Pruning 9/22/1513

CS Fall 2015 (© Jude Shavlik), Lecture 6, Week 2 ID3 & Noisy Data (cont.) See which dropped subtree leads to highest tune-set accuracy Repeat (ie, another greedy algo) Better tuneset accuracy? discard? 9/22/1514

CS Fall 2015 (© Jude Shavlik), Lecture 6, Week 2 Greedily Pruning D-Trees Sample (Hill Climbing) Search Space best Stop here if node’s best child is not an improvement 9/15/15 Note in pruning we’re reversing the tree- building process 15

CS Fall 2015 (© Jude Shavlik), Lecture 6, Week 2 Greedily Pruning D-trees - Pseudocode 1.Run ID3 to fully fit TRAIN’ Set, measure accuracy on TUNE 2.Consider all subtrees where ONE interior node removed and replaced by leaf - label with majority category in pruned subtree IF progress on TUNE choose best subtree ELSE (ie, if no improvement) quit 3.Go to 2 + 9/22/1516

CS Fall 2015 (© Jude Shavlik), Lecture 6, Week 2 Train/Tune/Test Accuracies (same sort of curves for other tuned param’s in other algo’s) 100% Accuracy Tune Test Train Ideal tree to choose Chosen pruned tree Amount of Pruning 9/22/1517

CS Fall 2015 (© Jude Shavlik), Lecture 6, Week 2 The General Tradeoff in Greedy Algorithms (more later) Efficiency vs. Optimality R AB C D F E Initial Tree Assume True Best Cuts Discard C’s & F’s subtrees Single Best Cut Discard B’s subtrees - irrevocable Greedy Search: Powerful, General- Purpose, Trick–of-the-Trade 9/22/1518

9/22/15CS Fall 2015 (© Jude Shavlik), Lecture 6, Week 2 Generating IF-THEN Rules from Trees Antecedent: Conjunction of all decisions leading to terminal node Consequent: Label of terminal node Red COLOR ? SIZE ? Blue Big Small Green - 19

9/22/15CS Fall 2015 (© Jude Shavlik), Lecture 6, Week 2 Generating Rules (cont) Previous slide’s tree generates these rules If Color=Green  Output = - If Color=Blue  Output = + If Color=Red and Size=Big  + If Color=Red and Size=Small  - Note 1. Can ‘clean up’ the rule set (next slide) 2. Decision trees learn disjunctive concepts 20

9/22/15CS Fall 2015 (© Jude Shavlik), Lecture 6, Week 2 Rule Post-Pruning (Another Greedy Algorithm) 1.Induce a decision tree 2.Convert to rules (see earlier slide) 3.Consider dropping any one rule antecedent –Delete the one that improves tuning set accuracy the most –Repeat as long as progress being made 21

9/22/15CS Fall 2015 (© Jude Shavlik), Lecture 6, Week 2 Rule Post-Pruning (cont) Advantages –Allows an intermediate node to be pruned from some rules but retained in others –Can correct poor early decisions in tree construction –Final concept more understandable Also applicable to ML algo’s that directly learn rules (eg, ILP, MLNs) 22 But note that the final rules will overlap one another – so need a ‘conflict resolution’ scheme

CS Fall 2015 (© Jude Shavlik), Lecture 6, Week 2 Training with Noisy Data If we can clean up the training data, should we do so? –No (assuming one can’t clean up the testing data when the learned concept will be used) –Better to train with the same type of data as will be experienced when the result of learning is put into use –Recall hadBankcruptcy was best indicator of “good candidate for credit card” story! 9/22/1523

CS Fall 2015 (© Jude Shavlik), Lecture 6, Week 2 Aside: A Rose by Any Other Name … Tuning sets also called –Pruning sets (in d-tree algorithms) –Validation sets (in general), but sometimes in the literature (eg, stats community) AI’s test sets called validation (and AI’s tuning sets called test sets!) 9/22/1524