INTRODUCTION TO Machine Learning

Slides:

Advertisements

Similar presentations

Ensemble Learning – Bagging, Boosting, and Stacking, and other topics

Advertisements

Combining Multiple Learners Ethem Chp. 15 Haykin Chp. 7, pp

Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0) ETHEM ALPAYDIN © The MIT Press, 2010

CHAPTER 2: Supervised Learning. Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 2 Learning a Class from Examples.

INTRODUCTION TO Machine Learning 2nd Edition

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

INTRODUCTION TO Machine Learning ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

INTRODUCTION TO Machine Learning 3rd Edition

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

INTRODUCTION TO Machine Learning 2nd Edition

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

INTRODUCTION TO Machine Learning ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

INTRODUCTION TO Machine Learning ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

Ensemble Learning what is an ensemble? why use an ensemble?

2D1431 Machine Learning Boosting.

A Brief Introduction to Adaboost

Ensemble Learning: An Introduction

INTRODUCTION TO Machine Learning ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

INTRODUCTION TO Machine Learning ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

INTRODUCTION TO Machine Learning ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

INTRODUCTION TO Machine Learning ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

INTRODUCTION TO Machine Learning ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

For Better Accuracy Eick: Ensemble Learning

Machine Learning CS 165B Spring 2012

Zhangxi Lin ISQS Texas Tech University Note: Most slides are from Decision Tree Modeling by SAS Lecture Notes 6 Ensembles of Trees.

CS Fall 2015 (© Jude Shavlik), Lecture 7, Week 3

Combining multiple learners Usman Roshan. Bagging Randomly sample training data Determine classifier C i on sampled data Goto step 1 and repeat m times.

Ensemble Methods: Bagging and Boosting

Ensemble Learning Spring 2009 Ben-Gurion University of the Negev.

Tony Jebara, Columbia University Advanced Machine Learning & Perception Instructor: Tony Jebara.

INTRODUCTION TO Machine Learning 3rd Edition

Concept learning, Regression Adapted from slides from Alpaydin’s book and slides by Professor Doina Precup, Mcgill University.

INTRODUCTION TO MACHINE LEARNING 3RD EDITION ETHEM ALPAYDIN © The MIT Press, Lecture.

INTRODUCTION TO MACHINE LEARNING 3RD EDITION ETHEM ALPAYDIN © The MIT Press, Lecture.

Ensemble Methods.  “No free lunch theorem” Wolpert and Macready 1995.

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

INTRODUCTION TO MACHINE LEARNING 3RD EDITION ETHEM ALPAYDIN © The MIT Press, Lecture.

CHAPTER 10: Logistic Regression. Binary classification Two classes Y = {0,1} Goal is to learn how to correctly classify the input into one of these two.

… Algo 1 Algo 2 Algo 3 Algo N Meta-Learning Algo.

Combining multiple learners Usman Roshan. Decision tree From Alpaydin, 2010.

Boosting ---one of combining models Xin Li Machine Learning Course.

INTRODUCTION TO MACHINE LEARNING 3RD EDITION ETHEM ALPAYDIN © The MIT Press, Lecture.

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

Combining Models Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya.

Ch 14. Combining Models Pattern Recognition and Machine Learning, C. M

Ensemble Learning Introduction to Machine Learning and Data Mining, Carla Brodley.

A “Holy Grail” of Machine Learing

INTRODUCTION TO Machine Learning

Combining Base Learners

Data Mining Practical Machine Learning Tools and Techniques

INTRODUCTION TO Machine Learning

Introduction to Data Mining, 2nd Edition

Machine Learning Ensemble Learning: Voting, Boosting(Adaboost)

INTRODUCTION TO Machine Learning

INTRODUCTION TO Machine Learning

Model Combination.

Ensemble learning Reminder - Bagging of Trees Random Forest

INTRODUCTION TO Machine Learning

INTRODUCTION TO Machine Learning 2nd Edition

INTRODUCTION TO Machine Learning 3rd Edition

INTRODUCTION TO Machine Learning

Presentation transcript:

INTRODUCTION TO Machine Learning Lecture Slides for INTRODUCTION TO Machine Learning ETHEM ALPAYDIN © The MIT Press, 2004 alpaydin@boun.edu.tr http://www.cmpe.boun.edu.tr/~ethem/i2ml

CHAPTER 15: Combining Multiple Learners

Rationale No Free Lunch thm: There is no algorithm that is always the most accurate Generate a group of base-learners which when combined has higher accuracy Different learners use different Algorithms Hyperparameters Representations (Modalities) Training sets Subproblems Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Voting Linear combination Classification Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Bayesian perspective: If dj are iid Bias does not change, variance decreases by L Average over randomness Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Error-Correcting Output Codes K classes; L problems (Dietterich and Bakiri, 1995) Code matrix W codes classes in terms of learners One per class L=K Pairwise L=K(K-1)/2 Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Subproblems may be more difficult than one-per-K Full code L=2(K-1)-1 With reasonable L, find W such that the Hamming distance btw rows and columns are maximized. Voting scheme Subproblems may be more difficult than one-per-K Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Bagging Use bootstrapping to generate L training sets and train one base-learner with each (Breiman, 1996) Use voting (Average or median with regression) Unstable algorithms profit from bagging Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

AdaBoost Generate a sequence of base-learners each focusing on previous one’s errors (Freund and Schapire, 1996) Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Mixture of Experts Voting where weights are input-dependent (gating) (Jacobs et al., 1991) Experts or gating can be nonlinear Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Stacking Combiner f () is another learner (Wolpert, 1992) Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Cascading Use dj only if preceding ones are not confident Cascade learners in order of complexity Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)