Adaboost Team G Youngmin Jun (ym_jun@naver.com) Shekhroz Khudoyarov (shekhrozx@gmail.com) Jaewan Choi (jwchoi@rayman.sejong.ac.kr) Alexandre Larzat (alexandre.larzat@gmail.com)

Contents Introduction Explanation of Adaboost Mathematical Formula Image Explanation How to solve

Overview of boosting Introduced by Schapire and Freund in 1990s. “Boosting”: convert a weak learning algorithm into a strong one. Main idea: Combine many weak classifiers to produce a powerful committee. Algorithms: AdaBoost: adaptive boosting Gentle AdaBoost BrownBoost Gradient Tree Boosting XGBoost R. Schapire and Y. Freund won the 2003 Godel Prize (one of the most prestigious awards in theoretical computer science) Prize winning paper (which introduced AdaBoost): "A decision theoretic generalization of on-line learning and an application to Boosting,“ Journal of Computer and System Sciences, 1997, 55: 119-139.

What is the Adaboost? Boosting is an approach to machine learning based on the idea of creating a highly accurate prediction rule by combining many relatively weak and inaccurate rules. AdaBoost, short for Adaptive Boosting, is a machine learning meta-algorithm. It ca n be used in conjunction with many other types of learning algorithms to improve performance. The output of the other learning algorithms ('weak learners') is com bined into a weighted sum that represents the final output of the boosted classifie r.

Adaboost Terms Classifier : h [-1, +1] Error rate line Learner = Hypothesis = Classifier Weak Learner: < 50% error over any distribution Strong Classifier: thresholded linear combination of weak learner outputs Classifier : h [-1, +1] 0.5 1.0 Weak classifier Strong classifier Error rate line Discarded Part

Introduction of Adaboost Weight

The Main Algorithm of Adaboost Each weights are Initialized as 1/m Repeats a “J” loop (Iteration) Finds a Classifier that has a min error Multiplies Dj(i) weight to wrong recognition errors, to calculate the whole error value If the error value is more than 0.5 or same than stop, because it is not good than the random value Else if the error value is less than 0.5, calculate the 𝛼 𝑗 = 1 2 log 1− 𝜖 𝑗 𝜖 𝑗 value If the 𝛼 𝑗 is set the Dj+1(i) value is updated Repeat this progress until “J” By linear combination of the weak classifiers, we can get a strong classifier

The Main Algorithm of Adaboost

Algorithm of Adaboost Given a training set with two classes: 𝑇={ 𝑥 1 , 𝑦 1 , 𝑥 2 , 𝑦 2 ,…,( 𝑥 𝑛 , 𝑦 𝑛 )} Where 𝑥 𝑖 ∈ 𝑅 𝑛 , 𝑦 𝑖 ∈{−1, 1}. The procedure of Adaboost can be described as following: Input: training set 𝑇 Output: the final classifier 𝐺(𝑥)

Algorithm of Adaboost Initialize weights of training examples: 𝐷 1 = 𝑤 11 ,…,𝑤 1 𝑖 ,…, 𝑤 1𝑛 , 𝑤 1𝑖 = 1 𝑛 ,𝑖=1,2,…𝑛 For 𝑚=1,2,…,𝑀 (Where M is the number of weak classifiers) Fit a classifier 𝐺 𝑚 𝑥 to the training data using weights 𝑤 𝑖 Compute misclassification error of 𝐺 𝑚 𝑥 : 𝑒 𝑚 =𝑃 𝐺 𝑚 𝑥 𝑖 ≠ 𝑦 𝑖 = 𝑖=1 𝑛 𝑤 𝑚𝑖 𝐼( 𝐺 𝑚 𝑥 𝑖 ≠ 𝑦 𝑖 ) (1)

Algorithm of Adaboost Compute the weight 𝛼 𝑚 for this classifier 𝐺 𝑚 𝑥 𝛼 𝑚 = 1 2 ln⁡ 1− 𝑒 𝑚 𝑒 𝑚 Update weights of training examples: 𝐷 𝑚+1 = 𝑤 𝑚+1,1 ,…, 𝑤 𝑚+1,𝑖 ,…, 𝑤 𝑚+1,𝑛 𝑤 𝑚+1,𝑖 = 𝑤 𝑚,𝑖 𝑍 𝑚 exp − 𝛼 𝑚 𝑦 𝑖 𝐺 𝑚 𝑥 𝑖 where 𝑍 𝑚 = 𝑖=1 𝑛 𝑤 𝑚𝑖 exp − 𝛼 𝑚 𝑦 𝑖 𝐺 𝑚 𝑥 𝑖 is regularization term and renormalize to 𝑤 𝑖 to sum to 1. (2) (3) (4) (5)

Algorithm of Adaboost The final classifier 𝐺(𝑥) is weighted sum of on each M iterations’ 𝛼 value and classifier output. 𝐺 𝑥 =𝑠𝑖𝑔𝑛 𝑓 𝑥 =𝑠𝑖𝑔𝑛( 𝑚=1 𝑀 𝛼 𝑚 𝐺 𝑚 (𝑥)) 𝛼 𝑚 stands for the weight of the m-th classifier. According to Equation (2), 𝛼 𝑚 ≥0 when 𝑒 𝑚 ≤ 1 2 . In addition, 𝛼 𝑚 increase with the decrease of 𝑒 𝑚 . Therefore, the classifiers with lower classification error have higher weights in the final classifier. (6)

Example of the classify – circle, triangle Example of Adaboost Example of the classify – circle, triangle

Example of Adaboost 𝜀 1 =0.30 𝛼 1 =0.42

Example of Adaboost 𝜀 2 =0.21 𝛼 2 =0.65

Example of Adaboost 𝜀 3 =0.14 𝛼 3 =0.92

Example of Adaboost 𝜀 2 =0.21 𝜀 1 =0.30 𝜀 3 =0.14 𝛼 2 =0.65 𝛼 1 =0.42 𝛼 3 =0.92

Example Training data : First iteration : 1 2 3 4 5 6 7 8 9 Y -1 𝝎 𝒊 0.1 First iteration : Best treshold is 2.5 with ℎ 1 𝑥 = +1 𝑖𝑓 𝑥<2.5 −1 𝑖𝑓 𝑥>2.5 𝐸 1 =3∗0.1=0.3 𝛼 1 = 0.5∗ ln 1−0.3 0.3 =0.423649 𝜔 𝑖 (𝑠+1) = 𝜔 𝑖 (𝑠) ∗𝑒 −0.423649 = 𝜔 𝑖 (𝑠) ∗0.65465, 𝑖𝑓 𝑐𝑜𝑟𝑟𝑒𝑐𝑡 𝜔 𝑖 (𝑠) ∗𝑒 +0.423649 = 𝜔 𝑖 (𝑠) ∗1.52753, 𝑖𝑓 𝑤𝑟𝑜𝑛𝑔

Example Updated data : 𝐻 𝑥 =𝑠𝑖𝑔𝑛(0.423649∗ ℎ 1 (𝑥)) 3 errors X 1 2 3 4 1 2 3 4 5 6 7 8 9 Y -1 𝝎 𝒊 updated Prenorm 0.065465 0.152753 Z = 7*0.065465 + 3*0,152753 = 0.9165168 0.07143 0.16667 𝐻 𝑥 =𝑠𝑖𝑔𝑛(0.423649∗ ℎ 1 (𝑥)) 3 errors

Example Training data : Second iteration : 1 2 3 4 5 6 7 8 9 Y -1 𝝎 𝒊 0.07143 0.16667 Second iteration : Best treshold is 8.5 (tiniest error) with ℎ 2 𝑥 = +1 𝑖𝑓 𝑥<8.5 −1 𝑖𝑓 𝑥>8.5 𝐸 2 =3∗0.07143=0.21429 𝛼 2 = 0.5∗ ln 1−0.21429 0.21429 = 0.64963 𝜔 𝑖 (𝑠+1) = 𝜔 𝑖 (𝑠) ∗ 𝑒 −0.64963 = 𝜔 𝑖 𝑠 ∗0.52224, 𝑖𝑓 𝑐𝑜𝑟𝑟𝑒𝑐𝑡 𝜔 𝑖 (𝑠) ∗ 𝑒 +0.64962 = 𝜔 𝑖 𝑠 ∗1.91481, 𝑖𝑓 𝑤𝑟𝑜𝑛𝑔

Example Updated data : 𝐻 𝑥 =𝑠𝑖𝑔𝑛(0.423649∗ ℎ 1 (𝑥)+0.64963∗ ℎ 2 (𝑥)) 1 2 3 4 5 6 7 8 9 Y -1 𝝎 𝒊 updated Prenorm 0.037304 0,136775 0.08704 Z = 4*0.037304 + 3*0,136775 + 3*0.08704 = 0.820661 0.045456 0.166664 0.106061 𝐻 𝑥 =𝑠𝑖𝑔𝑛(0.423649∗ ℎ 1 (𝑥)+0.64963∗ ℎ 2 (𝑥)) Still 3 errors

Example Training data : Third iteration : 1 2 3 4 5 6 7 8 9 Y -1 𝝎 𝒊 0.045456 0.166664 0.106061 Third iteration : Best treshold is 5.5 (tiniest error) with ℎ 3 𝑥 = +1 𝑖𝑓 𝑥>5.5 −1 𝑖𝑓 𝑥<5.5 𝐸 3 =4∗0.045456= 0.181824 𝛼 3 = 0.5∗ ln 1−0.181824 0.181824 = 0.752019 𝜔 𝑖 (𝑠+1) = 𝜔 𝑖 (𝑠) ∗𝑒 −0.752019 = 𝜔 𝑖 (𝑠) ∗0.471414, 𝑖𝑓 𝑐𝑜𝑟𝑟𝑒𝑐𝑡 𝜔 𝑖 (𝑠) ∗𝑒 +0.752019 = 𝜔 𝑖 (𝑠) ∗2.12129, 𝑖𝑓 𝑤𝑟𝑜𝑛𝑔

Example Training data : 1 2 3 4 5 6 7 8 9 Y -1 𝐻 𝑥 =𝑠𝑖𝑔𝑛(0.423649∗ ℎ 1 (𝑥)+0.64963∗ ℎ 2 𝑥 +0.752019∗ ℎ 3 (𝑥)) 0 error

Reference https://infinitescript.com/2016/09/adaboost/ https://en.wikipedia.org/wiki/Boosting_(machine_learning) https://www.analyticsvidhya.com/blog/2015/11/quick-introduction-boosting-algorithms- machine-learning/ https://medium.com/@deepvalidation/%EA%B5%B0%EC%A4%91%EC%9D%80- %EB%98%91%EB%98%91%ED%95%98%EB%8B%A42-adaboost-ba0122500034 https://quantdare.com/what-is-the-difference-between-bagging-and-boosting/ Yoav Freund, Robert Schapire, a short Introduction to Boosting Robert Schapire, the boosting approach to machine learning; Princeton University Yoav Freund, Robert Schapire, A decision-theoretic generalization of on-line learning and a pplication to boosting Pengyu Hong, Statistical Machine Learning lecture notes.

Thank You for Your Listening! Thank you for listening to my presentation