1. CLASSIFICATION: Ensemble Methods
Combines multiple models
Construct multiple classifiers from training set
Aggregate their predictions on testing set
Meta-algorithm

2. CLASSIFICATION: Ensemble Methods
Improves stability and accuracy
Reduces variance
Helps avoid overfitting
Compensates for poor learning algorithms
Uses more computation

3. ENSEMBLE METHODS: Examples
Bagging (bootstrap aggregation)
Bagging with MetaCost
Random forests
Boosting
Stacked generalization
Usually used on different learning algorithms
Bayesian model combination

4. ENSEMBLE METHODS: Bagging
Randomly create samples (with replacement) from a data set
Create classifiers (same type) for each sample
Run classifiers on testing sample
Use majority voting to determine classification of testing sample

5. ENSEMBLE METHODS: Bagging with MetaCost
Used when each model can output probability estimates
Probability estimates used to obtain expected cost of each prediction
Classifies training instances to minimize the expected cost
Learns new classifier

6. ENSEMBLE METHODS: Random Forests
Modification of applying bagging to tree learners
Uses only random subsets of features at each split
Promotes tree diversity

7. ENSEMBLE METHODS: Boosting
Seeks models that complement one another
Combines models of same type
New models constructed to better handle those instances incorrectly handled by previous models – focuses on hard to classify examples
Uses weighted averaging often adaptively

8. ENSEMBLE METHODS: Stacked Generalization
Introduced by David Wolpert, 1992
Other algorithms trained from training set
Stacking (“level-1”) algorithm uses predicitions from base (“level-0”) algorithms as inputs

9. ENSEMBLE METHODS: Stacked Generalization
Employs j-fold cross validation of training set
Train and test each of the level-0 algorithms using the split training data to create the level-0 models
Test each model on each split to create level-1 data

10. ENSEMBLE METHODS: Stacked Generalization

11. ENSEMBLE METHODS: Stacked Generalization
Can be used for both supervised and unsupervised learning
Best performers in Netflix competition were forms of stacked generalization
Can even create multiple levels of stacking(“level-2”, etc.)

12. ENSEMBLE METHODS: Stacked Generalization
Best performers in Netflix competition were forms of stacked generalization
Can even create multiple layers (“stacked stacking”)
Works best with class probabilities (Tang and Witten, 1999)

13. ENSEMBLE METHODS: Bayesian Model Combination
Built upon Bayes Model Averaging and Bayes Optimal Classifier
Bayes Optimal Classifier
Ensemble (using Bayes’ rule) of all hypotheses in hypothesis space
On average, it is the ideal ensemble

14. ENSEMBLE METHODS: Bayesian Model Combination
Bayes Model Averaging
Approximates Bayes optimal classifer
Samples from hypothesis space
Monte Carlo sampling
Tends to promote overfitting
Performs worse in practice than simpler techniques (eg bagging)

15. ENSEMBLE METHODS: Bayesian Model Combination
Bayes Model Combination
Correction to Bayes Model Averaging
Uses model weightings to create samples
Overcomes drawback of BMA giving weight to single model
Better performance than BMA or bagging