1 Decision Trees Greg Grudic (Notes borrowed from Thomas G. Dietterich and Tom Mitchell) Modified by Longin Jan Latecki Some slides by Piyush Rai Intro.

Slides:

Advertisements

Similar presentations

DECISION TREES. Decision trees  One possible representation for hypotheses.

Advertisements

1 Machine Learning: Lecture 3 Decision Tree Learning (Based on Chapter 3 of Mitchell T.., Machine Learning, 1997)

CHAPTER 9: Decision Trees

CHAPTER 2: Supervised Learning. Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 2 Learning a Class from Examples.

Data Mining Techniques: Classification. Classification What is Classification? –Classifying tuples in a database –In training set E each tuple consists.

Introduction Training Complexity, Pruning CART vs. ID3 vs. C4.5

K-NEAREST NEIGHBORS AND DECISION TREE Nonparametric Supervised Learning.

Decision Tree Learning 主講人：虞台文大同大學資工所智慧型多媒體研究室.

Overview Previous techniques have consisted of real-valued feature vectors (or discrete-valued) and natural measures of distance (e.g., Euclidean). Consider.

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

Decision Tree Rong Jin. Determine Milage Per Gallon.

Sparse vs. Ensemble Approaches to Supervised Learning

Decision tree LING 572 Fei Xia 1/10/06. Outline Basic concepts Main issues Advanced topics.

Three kinds of learning

INTRODUCTION TO Machine Learning ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

1 Nearest Neighbor Learning Greg Grudic (Notes borrowed from Thomas G. Dietterich and Tom Mitchell) Intro AI.

Example of a Decision Tree categorical continuous class Splitting Attributes Refund Yes No NO MarSt Single, Divorced Married TaxInc NO < 80K > 80K.

Decision Trees Rich Caruana. A Simple Decision Tree ©Tom Mitchell, McGraw Hill, 1997.

Sparse vs. Ensemble Approaches to Supervised Learning

Ordinal Decision Trees Qinghua Hu Harbin Institute of Technology

Machine Learning Lecture 10 Decision Trees G53MLE Machine Learning Dr Guoping Qiu1.

Decision Tree Learning

Decision tree LING 572 Fei Xia 1/16/06.

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

Data Mining: Classification

Machine Learning Chapter 3. Decision Tree Learning

Mohammad Ali Keyvanrad

Computational Intelligence: Methods and Applications Lecture 19 Pruning of decision trees Włodzisław Duch Dept. of Informatics, UMK Google: W Duch.

Machine Learning Lecture 10 Decision Tree Learning 1.

Comparing Univariate and Multivariate Decision Trees Olcay Taner Yıldız Ethem Alpaydın Department of Computer Engineering Bogazici University

Learning from Observations Chapter 18 Through

Categorical data. Decision Tree Classification Which feature to split on? Try to classify as many as possible with each split (This is a good split)

Decision Tree Learning Debapriyo Majumdar Data Mining – Fall 2014 Indian Statistical Institute Kolkata August 25, 2014.

1 Learning Chapter 18 and Parts of Chapter 20 AI systems are complex and may have many parameters. It is impractical and often impossible to encode all.

Longin Jan Latecki Temple University

For Wednesday No reading Homework: –Chapter 18, exercise 6.

Machine Learning, Decision Trees, Overfitting Machine Learning Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 14,

For Monday No new reading Homework: –Chapter 18, exercises 3 and 4.

MACHINE LEARNING 10 Decision Trees. Motivation  Parametric Estimation  Assume model for class probability or regression  Estimate parameters from all.

Concept learning, Regression Adapted from slides from Alpaydin’s book and slides by Professor Doina Precup, Mcgill University.

Machine Learning II Decision Tree Induction CSE 573.

1 Decision Tree Learning Original slides by Raymond J. Mooney University of Texas at Austin.

Decision Trees, Part 1 Reading: Textbook, Chapter 6.

INTRODUCTION TO MACHINE LEARNING 3RD EDITION ETHEM ALPAYDIN © The MIT Press, Lecture.

1 CSCI 3202: Introduction to AI Decision Trees Greg Grudic (Notes borrowed from Thomas G. Dietterich and Tom Mitchell) Intro AIDecision Trees.

Exercises Decision Trees In decision tree learning, the information gain criterion helps us select the best attribute to split the data at every node.

DECISION TREE Ge Song. Introduction ■ Decision Tree: is a supervised learning algorithm used for classification or regression. ■ Decision Tree Graph:

Decision Tree Learning Presented by Ping Zhang Nov. 26th, 2007.

ECE 471/571 – Lecture 20 Decision Tree 11/19/15. 2 Nominal Data Descriptions that are discrete and without any natural notion of similarity or even ordering.

Classification and Regression Trees

1 Decision Trees Greg Grudic (Notes borrowed from Thomas G. Dietterich and Tom Mitchell) [Edited by J. Wiebe] Decision Trees.

Outline Decision tree representation ID3 learning algorithm Entropy, Information gain Issues in decision tree learning 2.

Eco 6380 Predictive Analytics For Economists Spring 2016 Professor Tom Fomby Department of Economics SMU.

Decision Tree Learning DA514 - Lecture Slides 2 Modified and expanded from: E. Alpaydin-ML (chapter 9) T. Mitchell-ML.

INTRODUCTION TO MACHINE LEARNING 3RD EDITION ETHEM ALPAYDIN © The MIT Press, Lecture.

Machine Learning Inductive Learning and Decision Trees

k-Nearest neighbors and decision tree

Decision Trees (suggested time: 30 min)

Ch9: Decision Trees 9.1 Introduction A decision tree:

Decision Trees Greg Grudic

ECE 471/571 – Lecture 12 Decision Tree.

Machine Learning Chapter 3. Decision Tree Learning

Classification Slides by Greg Grudic, CSCI 3202 Fall 2007

Machine Learning: Lecture 3

Machine Learning Chapter 3. Decision Tree Learning

Statistical Learning Dong Liu Dept. EEIS, USTC.

Classification with CART

INTRODUCTION TO Machine Learning

Presentation transcript:

1 Decision Trees Greg Grudic (Notes borrowed from Thomas G. Dietterich and Tom Mitchell) Modified by Longin Jan Latecki Some slides by Piyush Rai Intro AIDecision Trees

2 Outline Decision Tree Representations –ID3 and C4.5 learning algorithms (Quinlan 1986) –CART learning algorithm (Breiman et al. 1985) Entropy, Information Gain Overfitting Intro AIDecision Trees

3 Training Data Example: Goal is to Predict When This Player Will Play Tennis? Intro AIDecision Trees

4Intro AIDecision Trees

5Intro AIDecision Trees

6Intro AIDecision Trees

7Intro AIDecision Trees

8 Learning Algorithm for Decision Trees What happens if features are not binary? What about regression? Intro AIDecision Trees

9 Choosing the Best Attribute Intro AIDecision Trees - Many different frameworks for choosing BEST have been proposed! - We will look at Entropy Gain. Number + and – examples before and after a split. A1 and A2 are “attributes” (i.e. features or inputs).

10 Entropy Intro AIDecision Trees

11Intro AIDecision Trees Entropy is like a measure of impurity…

12 Entropy Intro AIDecision Trees

Intro AIDecision Trees13

14 Information Gain Intro AIDecision Trees

Intro AIDecision Trees15

Intro AIDecision Trees16

Intro AIDecision Trees17

18 Training Example Intro AIDecision Trees

19 Selecting the Next Attribute Intro AIDecision Trees

20Intro AIDecision Trees

21 Non-Boolean Features Features with multiple discrete values –Multi-way splits –Test for one value versus the rest –Group values into disjoint sets Real-valued features –Use thresholds Regression –Splits based on mean squared error metric Intro AIDecision Trees

22 Hypothesis Space Search Intro AIDecision Trees You do not get the globally optimal tree! - Search space is exponential.

23 Overfitting Intro AIDecision Trees

24 Overfitting in Decision Trees Intro AIDecision Trees

25 Validation Data is Used to Control Overfitting Prune tree to reduce error on validation set Intro AIDecision Trees

Homework Which feature will be at the root node of the decision tree trained for the following data? In other words which attribute makes a person most attractive? Intro AIDecision Trees26 HeightHairEyesAttractive? smallblondebrownNo talldarkbrownNo tallblondeblueYes talldarkBlueNo smalldarkBlueNo tallredBlueYes tallblondebrownNo smallblondeblueYes