Decision Trees IDHairHeightWeightLotionResult SarahBlondeAverageLightNoSunburn DanaBlondeTallAverageYesnone AlexBrownTallAverageYesNone AnnieBlondeShortAverageNoSunburn.

Slides:

Advertisements

Similar presentations

Ensemble Learning Reading: R. Schapire, A brief introduction to boosting.

Advertisements

Lectures 17,18 – Boosting and Additive Trees Rice ECE697 Farinaz Koushanfar Fall 2006.

Boosting Rong Jin.

Ensemble Methods An ensemble method constructs a set of base classifiers from the training data Ensemble or Classifier Combination Predict class label.

CMPUT 466/551 Principal Source: CMU

Longin Jan Latecki Temple University

Decision Trees IDHairHeightWeightLotionResult SarahBlondeAverageLightNoSunburn DanaBlondeTallAverageYesnone AlexBrownTallAverageYesNone AnnieBlondeShortAverageNoSunburn.

Introduction to Boosting Slides Adapted from Che Wanxiang( 车万翔 ) at HIT, and Robin Dhamankar of Many thanks!

Sparse vs. Ensemble Approaches to Supervised Learning

Ensemble Learning what is an ensemble? why use an ensemble?

2D1431 Machine Learning Boosting.

Boosting Rong Jin. Inefficiency with Bagging D Bagging … D1D1 D2D2 DkDk Boostrap Sampling h1h1 h2h2 hkhk Inefficiency with boostrap sampling: Every example.

A Brief Introduction to Adaboost

Ensemble Learning: An Introduction

Adaboost and its application

Three kinds of learning

Introduction to Boosting Aristotelis Tsirigos SCLT seminar - NYU Computer Science.

Machine Learning: Ensemble Methods

Sparse vs. Ensemble Approaches to Supervised Learning

Boosting Main idea: train classifiers (e.g. decision trees) in a sequence. a new classifier should focus on those cases which were incorrectly classified.

Ensemble Learning (2), Tree and Forest

For Better Accuracy Eick: Ensemble Learning

Machine Learning CS 165B Spring 2012

Chapter 10 Boosting May 6, Outline Adaboost Ensemble point-view of Boosting Boosting Trees Supervised Learning Methods.

Bias and Variance Two ways to measure the match of alignment of the learning algorithm to the classification problem involve the bias and variance. Bias.

ENSEMBLE LEARNING David Kauchak CS451 – Fall 2013.

CS 391L: Machine Learning: Ensembles

LOGO Ensemble Learning Lecturer: Dr. Bo Yuan

ECE 8443 – Pattern Recognition Objectives: Bagging and Boosting Cross-Validation ML and Bayesian Model Comparison Combining Classifiers Resources: MN:

Benk Erika Kelemen Zsolt

Boosting of classifiers Ata Kaban. Motivation & beginnings Suppose we have a learning algorithm that is guaranteed with high probability to be slightly.

CS Fall 2015 (© Jude Shavlik), Lecture 7, Week 3

BOOSTING David Kauchak CS451 – Fall Admin Final project.

Combining multiple learners Usman Roshan. Bagging Randomly sample training data Determine classifier C i on sampled data Goto step 1 and repeat m times.

Ensemble Methods: Bagging and Boosting

Ensembles. Ensemble Methods l Construct a set of classifiers from training data l Predict class label of previously unseen records by aggregating predictions.

Ensemble Learning Spring 2009 Ben-Gurion University of the Negev.

CS Ensembles1 Ensembles. 2 A “Holy Grail” of Machine Learning Automated Learner Just a Data Set or just an explanation of the problem Hypothesis.

CLASSIFICATION: Ensemble Methods

ISQS 6347, Data & Text Mining1 Ensemble Methods. ISQS 6347, Data & Text Mining 2 Ensemble Methods Construct a set of classifiers from the training data.

E NSEMBLE L EARNING : A DA B OOST Jianping Fan Dept of Computer Science UNC-Charlotte.

Ensemble Methods.  “No free lunch theorem” Wolpert and Macready 1995.

Ensemble Methods in Machine Learning

Konstantina Christakopoulou Liang Zeng Group G21

Classification Ensemble Methods 1

1 January 24, 2016Data Mining: Concepts and Techniques 1 Data Mining: Concepts and Techniques — Chapter 7 — Classification Ensemble Learning.

Classification and Prediction: Ensemble Methods Bamshad Mobasher DePaul University Bamshad Mobasher DePaul University.

ID3 Algorithm Amrit Gurung. Classification Library System Organise according to special characteristics Faster retrieval New items sorted easily Related.

… Algo 1 Algo 2 Algo 3 Algo N Meta-Learning Algo.

ECE 8443 – Pattern Recognition ECE 8527 – Introduction to Machine Learning and Pattern Recognition Objectives: Bagging and Boosting Cross-Validation ML.

Combining multiple learners Usman Roshan. Decision tree From Alpaydin, 2010.

Tree and Forest Classification and Regression Tree Bagging of trees Boosting trees Random Forest.

1 Machine Learning Lecture 8: Ensemble Methods Moshe Koppel Slides adapted from Raymond J. Mooney and others.

Ensemble Classifiers.

Combining Models Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya.

Machine Learning: Ensemble Methods

Reading: R. Schapire, A brief introduction to boosting

Bagging and Random Forests

Trees, bagging, boosting, and stacking

COMP61011 : Machine Learning Ensemble Models

Ensemble Learning Introduction to Machine Learning and Data Mining, Carla Brodley.

A “Holy Grail” of Machine Learing

Combining Base Learners

Introduction to Data Mining, 2nd Edition

Ensemble learning.

Model Combination.

Ensemble learning Reminder - Bagging of Trees Random Forest

Recitation 10 Oznur Tastan

CS 391L: Machine Learning: Ensembles

Advisor: Dr.vahidipour Zahra salimian Shaghayegh jalali Dec 2017

Presentation transcript:

Decision Trees IDHairHeightWeightLotionResult SarahBlondeAverageLightNoSunburn DanaBlondeTallAverageYesnone AlexBrownTallAverageYesNone AnnieBlondeShortAverageNoSunburn EmilyRedAverageHeavyNoSunburn PeteBrownTallHeavyNoNone JohnBrownAverageHeavyNoNone KatieBlondeShortLightYesNone

Ensemble methods Use multiple models to obtain better predictive performance Ensembles combine multiple hypotheses to form a (hopefully) better hypothesis Combine multiple weak learners to produce a strong learner Typically much more computation, since you are training multiple learners

Ensemble learners Typically combine multiple fast learners (like decision trees) Tend to overfit Tend to get better results since there is deliberately introduced significant diversity among models  Diversity does not mean reduced performance Note that empirical studies have shown that random forests do better than an ensemble of decision trees  Random forest is an ensemble of decisions trees that do not minimize entropy to choose tree nodes.

Bayes optimal classifier is an ensemble learner

Bagging: Bootstrap aggregating Each model in the ensemble votes with equal weight Train each model with a random training set Random forests do better than bagged entropy reducing DTs

Bootstrap estimation Repeatedly draw n samples from D For each set of samples, estimate a statistic The bootstrap estimate is the mean of the individual estimates Used to estimate a statistic (parameter) and its variance

Bagging For i = 1.. M  Draw n * <n samples from D with replacement  Learn classifier C i Final classifier is a vote of C 1.. C M Increases classifier stability/reduces variance

Boosting Incremental Build new models that try to do better on previous model's mis-classifications  Can get better accuracy  Tends to overfit Adaboost is canonical boosting algorithm

Boosting (Schapire 1989) Randomly select n 1 < n samples from D without replacement to obtain D 1  Train weak learner C 1 Select n 2 < n samples from D with half of the samples misclassified by C 1 to obtain D 2  Train weak learner C 2 Select all samples from D that C 1 and C 2 disagree on  Train weak learner C 3 Final classifier is vote of weak learners

Adaboost Learner = Hypothesis = Classifier Weak Learner: < 50% error over any distribution Strong Classifier: thresholded linear combination of weak learner outputs

Discrete Adaboost

Real Adaboost

Comparison