1. Boosting
Rong Jin

2. Inefficiency with BaggingD
Bagging … D1 D2 Dk
Boostrap Sampling h1 h2 hk
Inefficient boostrap sampling:
Every example has equal chance to be sampled
No distinction between “easy” examples and “difficult” examples
Inefficient model combination:
A constant weight for each classifier
No distinction between accurate classifiers and inaccurate classifiers

3. Improve the Efficiency of Bagging
Better sampling strategy
Focus on the examples that are difficult to classify
Better combination strategy
Accurate model should be assigned larger weights

4. Intuition + +  May overfitting !! Mistakes Classifier3 Classifier2
 No training mistakes !!
 May overfitting !! +
Mistakes X1 Y1 +
Mistakes X1 Y1 X3 Y3
Training Examples X1 Y1 X2 Y2 X3 Y3 X4 Y4
Pop up note: no mistakes but possible overfitting on training data
Road map before boosting
Clearify the old works and the proposed methods
Less words !!!

5. AdaBoost Algorithm

6. AdaBoost Example: t=ln2
x1, y1
x2, y2
x3, y3
x4, y4
x5, y5
1/5
D0:
x5, y5
x3, y3
x1, y1
Sample h1
Training
x1, y1
x2, y2
x3, y3
x4, y4
x5, y5
Update Weights h1  
Sample
x3, y3
x1, y1
2/7
1/7
D1: h2
Training
x1, y1
x2, y2
x3, y3
x4, y4
x5, y5   h2
Update Weights
2/9
1/9
4/9
D2:
Sample …

7. How To Choose t in AdaBoost?
How to construct the best distribution Dt+1(i)
Dt+1(i) should be significantly different from Dt(i)
Dt+1(i) should create a situation that classifier ht performs poorly

8. How To Choose t in AdaBoost?

9. Optimization View for Choosing t
ht(x): x{1,-1}; a base (weak) classifier
HT(x): a linear combination of basic classifiers
Goal: minimize training error
Approximate error swith a exponential function

10. AdaBoost: Greedy Optimization
Fix HT-1(x), and solve hT(x) and t

11. Empirical Study of AdaBoost
AdaBoosting decision trees
Generate 50 decision trees by AdaBoost
Linearly combine decision trees using the weights of AdaBoost
In general:
AdaBoost = Bagging > C4.5
AdaBoost usually needs less number of classifiers than Bagging

12. Bia-Variance Tradeoff for AdaBoost
AdaBoost can reduce both variance and bias simultaneously
variance
bias
single decision tree
Bagging decision tree
AdaBoosting decision trees