A Statistician’s Games * : Bootstrap, Bagging and Boosting * Please refer to “Game theory, on-line prediction and boosting” by Y. Freund and R. Schapire,

Slides:

Advertisements

Similar presentations

Ensemble Learning – Bagging, Boosting, and Stacking, and other topics

Advertisements

Ensemble Learning Reading: R. Schapire, A brief introduction to boosting.

On-line learning and Boosting

Boosting Rong Jin.

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Other Classification Techniques 1.Nearest Neighbor Classifiers 2.Support Vector Machines.

Ensemble Methods An ensemble method constructs a set of base classifiers from the training data Ensemble or Classifier Combination Predict class label.

CMPUT 466/551 Principal Source: CMU

Longin Jan Latecki Temple University

Review of : Yoav Freund, and Robert E

Introduction to Boosting Slides Adapted from Che Wanxiang( 车万翔 ) at HIT, and Robin Dhamankar of Many thanks!

Sparse vs. Ensemble Approaches to Supervised Learning

Ensemble Learning what is an ensemble? why use an ensemble?

Boosting Rong Jin. Inefficiency with Bagging D Bagging … D1D1 D2D2 DkDk Boostrap Sampling h1h1 h2h2 hkhk Inefficiency with boostrap sampling: Every example.

A Brief Introduction to Adaboost

Ensemble Learning: An Introduction

Adaboost and its application

Introduction to Boosting Aristotelis Tsirigos SCLT seminar - NYU Computer Science.

Bayesian Framework EE 645 ZHAO XIN. A Brief Introduction to Bayesian Framework The Bayesian Philosophy Bayesian Neural Network Some Discussion on Priors.

Sparse vs. Ensemble Approaches to Supervised Learning

Boosting Main idea: train classifiers (e.g. decision trees) in a sequence. a new classifier should focus on those cases which were incorrectly classified.

For Better Accuracy Eick: Ensemble Learning

Machine Learning CS 165B Spring 2012

1 Bayesian methods for parameter estimation and data assimilation with crop models Part 2: Likelihood function and prior distribution David Makowski and.

Bias and Variance Two ways to measure the match of alignment of the learning algorithm to the classification problem involve the bias and variance. Bias.

CSSE463: Image Recognition Day 27 This week This week Last night: k-means lab due. Last night: k-means lab due. Today: Classification by “boosting” Today:

Machine Learning1 Machine Learning: Summary Greg Grudic CSCI-4830.

Boosting Neural Networks Published by Holger Schwenk and Yoshua Benggio Neural Computation, 12(8): , Presented by Yong Li.

CS 391L: Machine Learning: Ensembles

Lecture 7 Ensemble Algorithms MW 4:00PM-5:15PM Dr. Jianjun Hu CSCE822 Data Mining and Warehousing University of South.

LOGO Ensemble Learning Lecturer: Dr. Bo Yuan

Benk Erika Kelemen Zsolt

Boosting of classifiers Ata Kaban. Motivation & beginnings Suppose we have a learning algorithm that is guaranteed with high probability to be slightly.

Ensemble Learning Spring 2009 Ben-Gurion University of the Negev.

Learning with AdaBoost

E NSEMBLE L EARNING : A DA B OOST Jianping Fan Dept of Computer Science UNC-Charlotte.

Classification (slides adapted from Rob Schapire) Eran Segal Weizmann Institute.

CSSE463: Image Recognition Day 33 This week This week Today: Classification by “boosting” Today: Classification by “boosting” Yoav Freund and Robert Schapire.

Ensemble Methods in Machine Learning

Classification Ensemble Methods 1

1 January 24, 2016Data Mining: Concepts and Techniques 1 Data Mining: Concepts and Techniques — Chapter 7 — Classification Ensemble Learning.

Classification and Prediction: Ensemble Methods Bamshad Mobasher DePaul University Bamshad Mobasher DePaul University.

T-BAG: Bootstrap Aggregating the TAGE Predictor Ibrahim Burak Karsli, Resit Sendag University of Rhode Island.

Decision Trees IDHairHeightWeightLotionResult SarahBlondeAverageLightNoSunburn DanaBlondeTallAverageYesnone AlexBrownTallAverageYesNone AnnieBlondeShortAverageNoSunburn.

1 Introduction to Predictive Learning Electrical and Computer Engineering LECTURE SET 8 Combining Methods and Ensemble Learning.

… Algo 1 Algo 2 Algo 3 Algo N Meta-Learning Algo.

ECE 8443 – Pattern Recognition ECE 8527 – Introduction to Machine Learning and Pattern Recognition Objectives: Bagging and Boosting Cross-Validation ML.

1 Chapter 8: Model Inference and Averaging Presented by Hui Fang.

Boosting ---one of combining models Xin Li Machine Learning Course.

1 Machine Learning Lecture 8: Ensemble Methods Moshe Koppel Slides adapted from Raymond J. Mooney and others.

Adaboost (Adaptive boosting) Jo Yeong-Jun Schapire, Robert E., and Yoram Singer. "Improved boosting algorithms using confidence- rated predictions."

Machine Learning: Ensemble Methods

Reading: R. Schapire, A brief introduction to boosting

The Boosting Approach to Machine Learning

COMP61011 : Machine Learning Ensemble Models

Ensemble Learning Introduction to Machine Learning and Data Mining, Carla Brodley.

Combining Base Learners

Adaboost Team G Youngmin Jun

Predictive distributions

Machine Learning Ensemble Learning: Voting, Boosting(Adaboost)

Lecture 18: Bagging and Boosting

Multiple Decision Trees ISQS7342

Ensemble learning.

Lecture 06: Bagging and Boosting

Model Combination.

Ensemble learning Reminder - Bagging of Trees Random Forest

CS 391L: Machine Learning: Ensembles

Presentation transcript:

A Statistician’s Games * : Bootstrap, Bagging and Boosting * Please refer to “Game theory, on-line prediction and boosting” by Y. Freund and R. Schapire, Proceedings of 9th Conference on Computational Learning Theory. Yaochu Jin Future Technology Research Honda R&D Europe (Germany) March 21, 2000

Bootstrap -- Problem Description The bootstrap was introduced as a general method for assessing the statistical accuracy of an estimator Given data: x = ( x 1,..., x n ) Have an estimator:  = s(x) ? How to assess the accuracy of 

Bootstrap -- the Idea Bootstrap estimate of the standard error:

Bootstrap -- Pros and Cons Easy to implement Need a large number of independent bootstrap samples (B>=1000) Uncertainty of the estimate 1) Jackknife-after-Bootstrap(JAB) 2) Weighted JAB

Bagging is Not Related to Begging Using bootstrap techniques to improve the estimator Bagging -- Bootstrap aggregating

Bagging -- the Idea The final estimate:  = (  1 +   B )/B

Bagging -- Pros and Cons The estimator can be significantly improved if the learning algorithm is unstable Degrade the performance of stable procedures Reduce the variance, bias unchanged

Adaptive Bagging Reduce both variance and bias

Boosting To boost a “weak” learning algorithm into a “strong” learning algorithm A week learning algorithm can be inaccurate rules of thumb that is slightly better than random guess

AdaBoost Initialize Distribution D 1 (i) = 1/n Calculate error  t Choose weight  t = 1/2ln(1-  t /  t ) Update distribution The final estimate:  = (  1  1 +  2   n  B )/B

AdaBoost -- Pros and Cons Reduce both variance and bias Need large number of estimators (B>=1000) Sensitive to noise Theoretical guarantee (maximizes the likelihood) Easy to implement (compared to Bayesian methods) Relation to Support Vector Machines

Further Information on B 3 ftp://ftp.stat.berkeley.edu/pub/users/breiman/