1. Combining Base Learners
Meta-Learning Algo …
Algo Algo Algo Algo N

2. Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)
Rationale
No Free Lunch theorem: There is no algorithm that is always the most accurate.
Generate a group of algorithms which when combined display higher accuracy.

3. Data and Algo Variation…
Different Datasets …
Different Algorithms

4. Bagging Bagging can easily be extended to regression.
Bagging is most efficient when the base-learner is unstable.
Bagging typically increases accuracy.
Interpretability is lost.

5. Bagging

6. “Breiman's work bridged the gap between statisticians and
computer scientists, particularly in the field of machine learning.
Perhaps his most important contributions were his work
on classification and regression trees and ensembles
of trees fit to bootstrap samples.
Bootstrap aggregation was given the name bagging by
Breiman. Another of Breiman's ensemble approaches is the
random forest.” (Extracted from Wikipedia).

7. Boosting
Boosting tries to combine weak learners into a strong learner.
Originally all examples have the same weight, but in following
iterations examples wrongly classified increase their weight.
Boosting can be applied to any learner.

8. Boosting

9. Boosting Initialize all weights wi to 1/N (N: no. of examples)
error = 0
Repeat (until error > 0.5 or max. iterations reached)
Train classifier and get hypothesis ht(x)
Compute error as the sum of weights for misclassified exs.
error = Σ wi if wi is incorrectly classified.
Set αt = log ( 1-error / error )
Updates weights wi = [ wi e - (αt yi ht(xi) ] / Z
Output f(x) = sign ( Σt αt ht(x) )

10. Boosting
Misclassified Example
Increase Weights

11. Data and Algo Variation…
Different Datasets …
Different Algorithms

12. Mixture of Experts Voting where weights are input-dependent (gating)
Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)
Mixture of Experts
Voting where weights are input-dependent (gating)
(Jacobs et al., 1991)
Experts or gating
can be nonlinear

13. Mixture of Experts Robert Jacobs Michael Jordan
Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)
Mixture of Experts
Robert Jacobs
University of Rochester
Michael Jordan
UC Berkeley

14. Stacking Variations are among learners.
The predictions of the base learners form a new meta-dataset.
A testing example is first transformed into a new meta-example
and then classified.
Several variations have been proposed around stacking.

15. Stacking

16. Cascade Generalization
Variations are among learners.
Classifiers are used in sequence rather than in parallel as
in stacking.
The prediction of the first classifier is added to the
example feature vector to form an extended dataset.
The process can go on through many iterations.

17. Cascade Generalization

18. Cascading Like boosting, distribution changes across datasets.
But unlike boosting we will vary the classifiers.
Classification is based on prediction confidence.
Cascading creates rules that account for most instances
catching exceptions at the final step.

19. Cascading

20. Error-Correcting Output Codes
Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)
Error-Correcting Output Codes
K classes; L problems (Dietterich and Bakiri, 1995)
Code matrix W codes classes in terms of learners
One per class
L=K
Pairwise
L=K(K-1)/2

21. Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)
Full code L=2(K-1)-1
With reasonable L, find W such that the Hamming distance between rows and columns is maximized.