AdaBoost Reference Yoav Freund and Robert E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 55(1): , 1997.
“Two heads are better than one.” 三人寄れば文殊の知恵 Boosting is a technique of constructing a strong classifier by combining a collection of weak classifiers.
T1 T2 T3 T4 目標属性 (Objective Attribute) Training dataset
Basic Idea of AdaBoost Initially assign an equal weight to each record. Iterate the following steps: 1.Generate a hypothesis the error ratio of which is smaller than that of random guessing. 2. Increase weights of mis-predicted records relatively higher than the others. We call a classifier “a hypothesis” following the terms in the cited paper.
T1 T2 T3 T4 Ob Weightif T1=1 New then Ob=0 Weight else Ob= hypothesis represents the degree of the weight. Size of
T1 T2 T3 T4 Ob Weightif T3=1 New then Ob=1 Weight else Ob= another hypothesis
T1 T2 T3 T4 Ob Weightif T4=1 New then Ob=1 Weight else Ob= another hypothesis
AdaBoost performs weighted majority voting by hypotheses. Hypotheses if T1=1 if T3=1 if T4=1 Simple then Ob=0 then Ob=1 then Ob=1 Majority T1 T2 T3 T4 Ob else Ob=1 else Ob=0else Ob=0Voting
Input Training dataset Initial weight Weak learner WeakLearn that always ouputs a hypothesis whose error ratio is less than ½. T: number of iterations to generate weak hypothesis for each i =1,2, …, N Details of AdaBoost
2: Call WeakLearn to generate such a weak hypothesis h t that 1: Compute the distribution p i t of each record by normalizing weights 3: Revise weights
Revision of weights
Output: Final hypothesis h f (Weighted Majority Voting among h t )
The error of h f for the initial distribution.
Lemma 2
Q.E.D