Decision Trees Greg Grudic

Slides:

Advertisements

Similar presentations

Random Forest Predrag Radenković 3237/10

Advertisements

1 Machine Learning: Lecture 3 Decision Tree Learning (Based on Chapter 3 of Mitchell T.., Machine Learning, 1997)

CHAPTER 9: Decision Trees

Machine Learning III Decision Tree Induction

CHAPTER 2: Supervised Learning. Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 2 Learning a Class from Examples.

CPSC 502, Lecture 15Slide 1 Introduction to Artificial Intelligence (AI) Computer Science cpsc502, Lecture 15 Nov, 1, 2011 Slide credit: C. Conati, S.

Introduction Training Complexity, Pruning CART vs. ID3 vs. C4.5

K-NEAREST NEIGHBORS AND DECISION TREE Nonparametric Supervised Learning.

Decision Tree Learning 主講人：虞台文大同大學資工所智慧型多媒體研究室.

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

Decision Tree Rong Jin. Determine Milage Per Gallon.

Sparse vs. Ensemble Approaches to Supervised Learning

Greg GrudicIntro AI1 Introduction to Artificial Intelligence CSCI 3202 Fall 2007 Introduction to Classification Greg Grudic.

Decision tree LING 572 Fei Xia 1/10/06. Outline Basic concepts Main issues Advanced topics.

Three kinds of learning

INTRODUCTION TO Machine Learning ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

1 Nearest Neighbor Learning Greg Grudic (Notes borrowed from Thomas G. Dietterich and Tom Mitchell) Intro AI.

Decision Trees Rich Caruana. A Simple Decision Tree ©Tom Mitchell, McGraw Hill, 1997.

1 Decision Trees Greg Grudic (Notes borrowed from Thomas G. Dietterich and Tom Mitchell) Modified by Longin Jan Latecki Some slides by Piyush Rai Intro.

Sparse vs. Ensemble Approaches to Supervised Learning

INTRODUCTION TO Machine Learning ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

Ordinal Decision Trees Qinghua Hu Harbin Institute of Technology

Review Rong Jin. Comparison of Different Classification Models  The goal of all classifiers Predicating class label y for an input x Estimate p(y|x)

Decision Tree Learning

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

Data Mining Joyeeta Dutta-Moscato July 10, Wherever we have large amounts of data, we have the need for building systems capable of learning information.

Mohammad Ali Keyvanrad

Chapter 10. Sampling Strategy for Building Decision Trees from Very Large Databases Comprising Many Continuous Attributes Jean-Hugues Chauchat and Ricco.

Computational Intelligence: Methods and Applications Lecture 19 Pruning of decision trees Włodzisław Duch Dept. of Informatics, UMK Google: W Duch.

Machine Learning Lecture 10 Decision Tree Learning 1.

Learning from Observations Chapter 18 Through

Categorical data. Decision Tree Classification Which feature to split on? Try to classify as many as possible with each split (This is a good split)

Decision Tree Learning Debapriyo Majumdar Data Mining – Fall 2014 Indian Statistical Institute Kolkata August 25, 2014.

1 Learning Chapter 18 and Parts of Chapter 20 AI systems are complex and may have many parameters. It is impractical and often impossible to encode all.

For Wednesday No reading Homework: –Chapter 18, exercise 6.

For Monday No new reading Homework: –Chapter 18, exercises 3 and 4.

Concept learning, Regression Adapted from slides from Alpaydin’s book and slides by Professor Doina Precup, Mcgill University.

Decision Trees Binary output – easily extendible to multiple output classes. Takes a set of attributes for a given situation or object and outputs a yes/no.

1 Decision Tree Learning Original slides by Raymond J. Mooney University of Texas at Austin.

INTRODUCTION TO MACHINE LEARNING 3RD EDITION ETHEM ALPAYDIN © The MIT Press, Lecture.

1 CSCI 3202: Introduction to AI Decision Trees Greg Grudic (Notes borrowed from Thomas G. Dietterich and Tom Mitchell) Intro AIDecision Trees.

Exercises Decision Trees In decision tree learning, the information gain criterion helps us select the best attribute to split the data at every node.

DECISION TREE Ge Song. Introduction ■ Decision Tree: is a supervised learning algorithm used for classification or regression. ■ Decision Tree Graph:

Classification and Regression Trees

1 Decision Trees Greg Grudic (Notes borrowed from Thomas G. Dietterich and Tom Mitchell) [Edited by J. Wiebe] Decision Trees.

Outline Decision tree representation ID3 learning algorithm Entropy, Information gain Issues in decision tree learning 2.

Machine Learning Lecture 1: Intro + Decision Trees Moshe Koppel Slides adapted from Tom Mitchell and from Dan Roth.

Eco 6380 Predictive Analytics For Economists Spring 2016 Professor Tom Fomby Department of Economics SMU.

Decision Tree Learning DA514 - Lecture Slides 2 Modified and expanded from: E. Alpaydin-ML (chapter 9) T. Mitchell-ML.

INTRODUCTION TO MACHINE LEARNING 3RD EDITION ETHEM ALPAYDIN © The MIT Press, Lecture.

Combining Models Foundations of Algorithms and Machine Learning (CS60020), IIT KGP, 2017: Indrajit Bhattacharya.

Machine Learning Inductive Learning and Decision Trees

k-Nearest neighbors and decision tree

Artificial Intelligence

Decision Trees (suggested time: 30 min)

Decision Tree Saed Sayad 9/21/2018.

ECE 471/571 – Lecture 12 Decision Tree.

Machine Learning Today: Reading: Maria Florina Balcan

Machine Learning Chapter 3. Decision Tree Learning

Classification Slides by Greg Grudic, CSCI 3202 Fall 2007

Machine Learning: Lecture 3

Machine Learning Chapter 3. Decision Tree Learning

Statistical Learning Dong Liu Dept. EEIS, USTC.

Learning Chapter 18 and Parts of Chapter 20

Classification with CART

INTRODUCTION TO Machine Learning

CS639: Data Management for Data Science

Learning Chapter 18 and Parts of Chapter 20

INTRODUCTION TO Machine Learning

Presentation transcript:

Decision Trees Greg Grudic (Notes borrowed from Thomas G. Dietterich and Tom Mitchell) Modified by Longin Jan Latecki Some slides by Piyush Rai Intro AI Decision Trees

Outline Decision Tree Representations Entropy, Information Gain ID3 and C4.5 learning algorithms (Quinlan 1986) CART learning algorithm (Breiman et al. 1985) Entropy, Information Gain Overfitting Intro AI Decision Trees

Training Data Example: Goal is to Predict When This Player Will Play Tennis? Intro AI Decision Trees

Intro AI Decision Trees

Intro AI Decision Trees

Intro AI Decision Trees

Intro AI Decision Trees

Learning Algorithm for Decision Trees What happens if features are not binary? What about regression? Intro AI Decision Trees

Choosing the Best Attribute A1 and A2 are “attributes” (i.e. features or inputs). Number + and – examples before and after a split. - Many different frameworks for choosing BEST have been proposed! - We will look at Entropy Gain. Intro AI Decision Trees

Entropy Intro AI Decision Trees

Entropy is like a measure of purity… Intro AI Decision Trees

Entropy Intro AI Decision Trees

Intro AI Decision Trees

Information Gain Intro AI Decision Trees

Intro AI Decision Trees

Intro AI Decision Trees

Intro AI Decision Trees

Training Example Intro AI Decision Trees

Selecting the Next Attribute Intro AI Decision Trees

Intro AI Decision Trees

Non-Boolean Features Features with multiple discrete values Multi-way splits Test for one value versus the rest Group values into disjoint sets Real-valued features Use thresholds Regression Splits based on mean squared error metric Intro AI Decision Trees

Hypothesis Space Search You do not get the globally optimal tree! Search space is exponential. Intro AI Decision Trees

Overfitting Intro AI Decision Trees

Overfitting in Decision Trees Intro AI Decision Trees

Validation Data is Used to Control Overfitting Prune tree to reduce error on validation set Intro AI Decision Trees