CS 540 - Fall 2015 (© Jude Shavlik), Lecture 7, Week 3 CS 540 Fall 2015 (Shavlik) 4/24/2017 Today’s Topics Ensembles Decision Forests (actually, Random Forests) Bagging and Boosting Decision Stumps 9/22/15 CS 540 - Fall 2015 (© Jude Shavlik), Lecture 7, Week 3

Ensembles (Bagging, Boosting, and all that) Old View Learn one good model New View Learn a good set of models Probably best example of interplay between ‘theory & practice’ in machine learning Naïve Bayes, k-NN, neural net, d-tree, SVM, etc 9/22/15 CS 540 - Fall 2015 (© Jude Shavlik), Lecture 7, Week 3

Ensembles of Neural Networks (or any supervised learner) OUTPUT Combiner Network Network Network INPUT Ensembles often produce accuracy gains of 5-10 percentage points! Can combine “classifiers” of various types Eg, decision trees, rule sets, neural networks, etc. 9/22/15 CS 540 - Fall 2015 (© Jude Shavlik), Lecture 7, Week 3

Three Explanations of Why Ensembles Help Statistical (sample effects) Computational (limited cycles for search) Representational (wrong hypothesis space) From: Dietterich, T. G. (2002). Ensemble Learning. In The Handbook of Brain Theory and Neural Networks, Second edition, (M.A. Arbib, Ed.), Cambridge, MA: The MIT Press, 2002. 405-408 Key  true concept  learned models search path Concept Space Considered 9/22/15 CS 540 - Fall 2015 (© Jude Shavlik), Lecture 7, Week 3

Combining Multiple Models Three ideas for combining predictions Simple (unweighted) votes Standard choice Weighted votes eg, weight by tuning-set accuracy Learn a combining function Prone to overfitting? ‘Stacked generalization’ (Wolpert) 9/22/15 CS 540 - Fall 2015 (© Jude Shavlik), Lecture 7, Week 3

Random Forests (Breiman, Machine Learning 2001; related to Ho, 1995) A variant of something called BAGGING (‘multi-sets’) Algorithm Repeat k times Draw with replacement N examples, put in train set Build d-tree, but in each recursive call Choose (w/o replacement) i features Choose best of these i as the root of this (sub)tree Do NOT prune In HW2, we’ll give you 101 ‘bootstrapped’ samples of the Thoracic Surgery Dataset Let N = # of examples F = # of features i = some number << F 9/22/15 CS 540 - Fall 2015 (© Jude Shavlik), Lecture 7, Week 3

CS 540 - Fall 2015 (© Jude Shavlik), Lecture 7, Week 3 Using Random Forests After training we have K decision trees How to use on TEST examples? Some variant of If at least L of these K trees say ‘true’ then output ‘true’ How to choose L ? Use a tune set to decide 9/22/15 CS 540 - Fall 2015 (© Jude Shavlik), Lecture 7, Week 3

CS 540 - Fall 2015 (© Jude Shavlik), Lecture 7, Week 3 More on Random Forests Increasing i Increases correlation among individual trees (BAD) Also increases accuracy of individual trees (GOOD) Can also use tuning set to choose good value for i Overall, random forests Are very fast (eg, 50K examples, 10 features, 10 trees/min on 1 GHz CPU back in 2004) Deal well with large # of features Reduce overfitting substantially; NO NEED TO PRUNE! Work very well in practice 9/22/15 CS 540 - Fall 2015 (© Jude Shavlik), Lecture 7, Week 3

A Relevant Early Paper on ENSEMBLES Hansen & Salamen, PAMI:20, 1990 If (a) the combined predictors have errors that are independent from one another And (b) prob any given model correct predicts any given testset example is > 50%, then 9/22/15 CS 540 - Fall 2015 (© Jude Shavlik), Lecture 7, Week 3

Some More Relevant Early Papers CS 540 Fall 2015 (Shavlik) 4/24/2017 Some More Relevant Early Papers Schapire, Machine Learning:5, 1990 (‘Boosting’) If you have an algorithm that gets > 50% on any distribution of examples, you can create an algorithm that gets > (100% - ), for any  > 0 Need an infinite (or very large, at least) source of examples - Later extensions (eg, AdaBoost) address this weakness Also see Wolpert, ‘Stacked Generalization,’ Neural Networks, 1992 9/22/15 CS 540 - Fall 2015 (© Jude Shavlik), Lecture 7, Week 3

Some Methods for Producing ‘Uncorrelated’ Members of an Ensemble K times randomly choose (with replacement) N examples from a training set of size N Give each training set to a std ML algo ‘Bagging’ by Brieman (Machine Learning, 1996) Want unstable algorithms (so learned models vary) Reweight examples each cycle (if wrong, increase weight; else decrease weight) ‘AdaBoosting’ by Freund & Schapire (1995, 1996) 9/22/15 CS 540 - Fall 2015 (© Jude Shavlik), Lecture 7, Week 3

CS 540 - Fall 2015 (© Jude Shavlik), Lecture 7, Week 3 Empirical Studies (from Freund & Schapire; reprinted in Dietterich’s AI Magazine paper) Error Rate of C4.5 Error Rate of Bagged (Boosted) C4.5 (Each point one data set) Boosting and Bagging helped almost always! Error Rate of AdaBoost Error Rate of Bagging On average, Boosting slightly better? ID3 successor 9/22/15 CS 540 - Fall 2015 (© Jude Shavlik), Lecture 7, Week 3

Some More Methods for Producing “Uncorrelated” Members of an Ensemble Directly optimize accuracy + diversity Opitz & Shavlik (1995; used genetic algo’s) Melville & Mooney (2004-5) Different number of hidden units in a neural network, different k in k -NN, tie-breaking scheme, example ordering, diff ML algos, etc Various people See 2005-2008 papers of Rich Caruana’s group for large-scale empirical studies of ensembles 9/22/15 CS 540 - Fall 2015 (© Jude Shavlik), Lecture 7, Week 3

Boosting/Bagging/etc Wrapup An easy to use and usually highly effective technique always consider it (Bagging, at least) when applying ML to practical problems Does reduce ‘comprehensibility’ of models see work by Craven & Shavlik though (‘rule extraction’) Increases runtime, but cycles usually much cheaper than examples (and easily parallelized) 9/22/15 CS 540 - Fall 2015 (© Jude Shavlik), Lecture 7, Week 3

Decision “Stumps” (formerly part of HW; try on your own!) Holte (ML journal) compared: Decision trees with only one decision (decision stumps) vs Trees produced by C4.5 (with pruning algorithm used) Decision ‘stumps’ do remarkably well on UC Irvine data sets Archive too easy? Some datasets seem to be Decision stumps are a ‘quick and dirty control for comparing to new algorithms But ID3/C4.5 easy to use and probably a better control 9/22/15 CS 540 - Fall 2015 (© Jude Shavlik), Lecture 7, Week 3

C4.5 Compared to 1R (‘Decision Stumps’) Testset Accuracy C4.5 Compared to 1R (‘Decision Stumps’) Dataset C4.5 1R BC 72.0% 68.7% CH 99.2% GL 63.2% 67.6% G2 74.3% 53.8% HD 73.6% 72.9% HE 81.2% 76.3% HO 83.6% 81.0% HY 99.1% 97.2% IR 93.8% 93.5% LA 77.2% 71.5% LY 77.5% 70.7% MU 100.0% 98.4% SE 97.7% 95.0% SO 97.5% VO 95.6% 95.2% V1 89.4% 86.8% See Holte paper in Machine Learning for key (eg, HD=heart disease) 9/22/15 CS 540 - Fall 2015 (© Jude Shavlik), Lecture 7, Week 3