Combining Base Learners Meta-Learning Algo … Algo 1 Algo 2 Algo 3 Algo N

Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) Rationale No Free Lunch theorem: There is no algorithm that is always the most accurate. Generate a group of algorithms which when combined display higher accuracy.

Data and Algo Variation … Different Datasets … Different Algorithms

Bagging Bagging can easily be extended to regression. Bagging is most efficient when the base-learner is unstable. Bagging typically increases accuracy. Interpretability is lost.

Bagging

“Breiman's work bridged the gap between statisticians and computer scientists, particularly in the field of machine learning. Perhaps his most important contributions were his work on classification and regression trees and ensembles of trees fit to bootstrap samples. Bootstrap aggregation was given the name bagging by Breiman. Another of Breiman's ensemble approaches is the random forest.” (Extracted from Wikipedia).

Boosting Boosting tries to combine weak learners into a strong learner. Originally all examples have the same weight, but in following iterations examples wrongly classified increase their weight. Boosting can be applied to any learner.

Boosting

Boosting Initialize all weights wi to 1/N (N: no. of examples) error = 0 Repeat (until error > 0.5 or max. iterations reached) Train classifier and get hypothesis ht(x) Compute error as the sum of weights for misclassified exs. error = Σ wi if wi is incorrectly classified. Set αt = log ( 1-error / error ) Updates weights wi = [ wi e - (αt yi ht(xi) ] / Z Output f(x) = sign ( Σt αt ht(x) )

Boosting Misclassified Example Increase Weights

Data and Algo Variation … Different Datasets … Different Algorithms

Mixture of Experts Voting where weights are input-dependent (gating) Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) Mixture of Experts Voting where weights are input-dependent (gating) (Jacobs et al., 1991) Experts or gating can be nonlinear

Mixture of Experts Robert Jacobs Michael Jordan Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) Mixture of Experts Robert Jacobs University of Rochester Michael Jordan UC Berkeley

Stacking Variations are among learners. The predictions of the base learners form a new meta-dataset. A testing example is first transformed into a new meta-example and then classified. Several variations have been proposed around stacking.

Stacking

Cascade Generalization Variations are among learners. Classifiers are used in sequence rather than in parallel as in stacking. The prediction of the first classifier is added to the example feature vector to form an extended dataset. The process can go on through many iterations.

Cascade Generalization

Cascading Like boosting, distribution changes across datasets. But unlike boosting we will vary the classifiers. Classification is based on prediction confidence. Cascading creates rules that account for most instances catching exceptions at the final step.

Cascading

Error-Correcting Output Codes Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) Error-Correcting Output Codes K classes; L problems (Dietterich and Bakiri, 1995) Code matrix W codes classes in terms of learners One per class L=K Pairwise L=K(K-1)/2

Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) Full code L=2(K-1)-1 With reasonable L, find W such that the Hamming distance between rows and columns is maximized.