CSE P573 Applications of Artificial Intelligence Decision Trees

Slides:

Advertisements

Similar presentations

On the Optimality of Probability Estimation by Random Decision Trees Wei Fan IBM T.J.Watson.

Advertisements

Random Forest Predrag Radenković 3237/10

Ensemble Learning Reading: R. Schapire, A brief introduction to boosting.

Machine Learning III Decision Tree Induction

My name is Dustin Boswell and I will be presenting: Ensemble Methods in Machine Learning by Thomas G. Dietterich Oregon State University, Corvallis, Oregon.

Data Mining Classification: Alternative Techniques

Ensemble Methods An ensemble method constructs a set of base classifiers from the training data Ensemble or Classifier Combination Predict class label.

Boosting Ashok Veeraraghavan. Boosting Methods Combine many weak classifiers to produce a committee. Resembles Bagging and other committee based methods.

Longin Jan Latecki Temple University

Bayesian Learning Rong Jin. Outline MAP learning vs. ML learning Minimum description length principle Bayes optimal classifier Bagging.

Sparse vs. Ensemble Approaches to Supervised Learning

Learning from Observations Copyright, 1996 © Dale Carnegie & Associates, Inc. Chapter 18 Fall 2005.

Learning from Observations Copyright, 1996 © Dale Carnegie & Associates, Inc. Chapter 18 Fall 2004.

Ensemble Learning: An Introduction

Machine Learning IV Ensembles CSE 473. © Daniel S. Weld 2 Machine Learning Outline Machine learning: Supervised learning Overfitting Ensembles of classifiers.

Introduction to Boosting Aristotelis Tsirigos SCLT seminar - NYU Computer Science.

Examples of Ensemble Methods

Machine Learning: Ensemble Methods

Sparse vs. Ensemble Approaches to Supervised Learning

Ensemble Learning (2), Tree and Forest

For Better Accuracy Eick: Ensemble Learning

Ensembles of Classifiers Evgueni Smirnov

Issues with Data Mining

By Wang Rui State Key Lab of CAD&CG

Machine Learning Chapter 3. Decision Tree Learning

Machine Learning1 Machine Learning: Summary Greg Grudic CSCI-4830.

ENSEMBLE LEARNING David Kauchak CS451 – Fall 2013.

CS 391L: Machine Learning: Ensembles

ECE 8443 – Pattern Recognition Objectives: Bagging and Boosting Cross-Validation ML and Bayesian Model Comparison Combining Classifiers Resources: MN:

Learning from Observations Chapter 18 Through

1 Ensembles An ensemble is a set of classifiers whose combined results give the final decision. test feature vector classifier 1classifier 2classifier.

Ensembles. Ensemble Methods l Construct a set of classifiers from training data l Predict class label of previously unseen records by aggregating predictions.

CLASSIFICATION: Ensemble Methods

ISQS 6347, Data & Text Mining1 Ensemble Methods. ISQS 6347, Data & Text Mining 2 Ensemble Methods Construct a set of classifiers from the training data.

CS 8751 ML & KDDDecision Trees1 Decision tree representation ID3 learning algorithm Entropy, Information gain Overfitting.

Machine Learning II Decision Tree Induction CSE 573.

CS 5751 Machine Learning Chapter 3 Decision Tree Learning1 Decision Trees Decision tree representation ID3 learning algorithm Entropy, Information gain.

Ensemble Methods in Machine Learning

Random Forests Ujjwol Subedi. Introduction What is Random Tree? ◦ Is a tree constructed randomly from a set of possible trees having K random features.

Classification Ensemble Methods 1

Decision Trees IDHairHeightWeightLotionResult SarahBlondeAverageLightNoSunburn DanaBlondeTallAverageYesnone AlexBrownTallAverageYesNone AnnieBlondeShortAverageNoSunburn.

ECE 8443 – Pattern Recognition ECE 8527 – Introduction to Machine Learning and Pattern Recognition Objectives: Bagging and Boosting Cross-Validation ML.

Learning From Observations Inductive Learning Decision Trees Ensembles.

CSE 446: Ensemble Learning Winter 2012 Daniel Weld Slides adapted from Tom Dietterich, Luke Zettlemoyer, Carlos Guestrin, Nick Kushmerick, Padraig Cunningham.

1 Machine Learning Lecture 8: Ensemble Methods Moshe Koppel Slides adapted from Raymond J. Mooney and others.

1 Ensembles An ensemble is a set of classifiers whose combined results give the final decision. test feature vector classifier 1classifier 2classifier.

Ensemble Classifiers.

Machine Learning: Ensemble Methods

Data Mining Practical Machine Learning Tools and Techniques

Reading: R. Schapire, A brief introduction to boosting

Presented By S.Yamuna AP/CSE

Machine Learning: Ensembles

Ensemble Learning Introduction to Machine Learning and Data Mining, Carla Brodley.

Issues in Decision-Tree Learning Avoiding overfitting through pruning

Data Mining Lecture 11.

A “Holy Grail” of Machine Learing

Data Mining Practical Machine Learning Tools and Techniques

Introduction to Data Mining, 2nd Edition

CSE P573 Applications of Artificial Intelligence Bayesian Learning

Machine Learning Chapter 3. Decision Tree Learning

CSE 573 Introduction to Artificial Intelligence Decision Trees

Machine Learning Chapter 3. Decision Tree Learning

Machine Learning CSE 454.

Ensemble learning.

Model Combination.

Ensemble learning Reminder - Bagging of Trees Random Forest

Lecture 14 Learning Inductive inference

Ensembles An ensemble is a set of classifiers whose combined results give the final decision. test feature vector classifier 1 classifier 2 classifier.

Machine Learning: Lecture 5

Presentation transcript:

CSE P573 Applications of Artificial Intelligence Decision Trees Henry Kautz Autumn 2004

Today’s Agenda Inductive learning Decision trees break Bayesian learning

Inductive Learning

Is it necessarily the answer? Bingo! Is it necessarily the answer?

Is it necessarily the answer? Bingo! Is it necessarily the answer?

Bias in Learning Hypothesis space Preferences over hypothesis Other prior knowledge Without bias learning is impossible!

Decision Trees

Variable-sized hypothesis space Number of possible hypotheses grows with depth of tree

How can this algorithm be viewed as a state-space search problem?

Ensembles of Classifiers Idea: instead of training one classifier (decision tree) Train k classifiers and let them vote Only helps if classifiers disagree with each other Trained on different data Use different learning methods Amazing fact: can help a lot!

How voting helps If individual area is 0.3 Assume errors are independent Assume majority vote Probability majority is wrong = area under bionomial dist Prob 0.2 0.1 Number of classifiers in error If individual area is 0.3 Area under curve for 11 wrong is 0.026 Order of magnitude improvement!

Constructing Ensembles Bagging Run classifier k times on m examples drawn randomly with replacement from the original set of n examples Cross-validated committees Divide examples into k disjoint sets Train on k sets corresponding to original minus 1/k-th Boosting (Shapire) Maintain a probability distribution over set of training examples On each iteration, use distribution to sample Use error rate to modify distribution Create harder and harder learning problems

Summary Inductive learning Decision trees Ensembles Representation Tree growth Heuristics Overfitting and pruning Scaling up Ensembles