Machine Learning: Decision Trees Homework 4 assigned courtesy: Geoffrey Hinton, Yann LeCun, Tan, Steinbach, Kumar.

Slides:

Advertisements

Similar presentations

Data Mining Classification: Alternative Techniques

Advertisements

DECISION TREES. Decision trees  One possible representation for hypotheses.

CHAPTER 9: Decision Trees

Data Mining Classification: Basic Concepts, Decision Trees, and Model Evaluation Lecture Notes for Chapter 4 Part I Introduction to Data Mining by Tan,

Bab /44 Bab 4 Classification: Basic Concepts, Decision Trees & Model Evaluation Part 1 Classification With Decision tree.

Classification: Definition Given a collection of records (training set ) –Each record contains a set of attributes, one of the attributes is the class.

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Classification: Definition l Given a collection of records (training set) l Find a model.

1 Data Mining Classification Techniques: Decision Trees (BUSINESS INTELLIGENCE) Slides prepared by Elizabeth Anglo, DISCS ADMU.

Data Mining Classification: Basic Concepts, Decision Trees, and Model Evaluation Lecture Notes for Chapter 4 Introduction to Data Mining by Tan, Steinbach,

Classification Kuliah 4 4/29/2015. Classification: Definition  Given a collection of records (training set )  Each record contains a set of attributes,

Data Mining Classification This lecture node is modified based on Lecture Notes for Chapter 4/5 of Introduction to Data Mining by Tan, Steinbach, Kumar,

Data Mining Classification: Alternative Techniques

Data Mining Classification: Alternative Techniques

Lecture Notes for Chapter 4 (2) Introduction to Data Mining

C4.5 - pruning decision trees

Classification Techniques: Decision Tree Learning

Lecture Notes for Chapter 4 Part III Introduction to Data Mining

Classification: Basic Concepts and Decision Trees.

Lecture Notes for Chapter 4 Introduction to Data Mining

Classification: Decision Trees, and Naïve Bayes etc. March 17, 2010 Adapted from Chapters 4 and 5 of the book Introduction to Data Mining by Tan, Steinbach,

Data Mining Classification: Alternative Techniques

Lecture outline Classification Decision-tree classification.

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

Decision Tree Rong Jin. Determine Milage Per Gallon.

Ensemble Learning: An Introduction

Tree-based methods, neutral networks

Lecture 5 (Classification with Decision Trees)

INTRODUCTION TO Machine Learning ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

© Vipin Kumar CSci 8980 Fall CSci 8980: Data Mining (Fall 2002) Vipin Kumar Army High Performance Computing Research Center Department of Computer.

Example of a Decision Tree categorical continuous class Splitting Attributes Refund Yes No NO MarSt Single, Divorced Married TaxInc NO < 80K > 80K.

© Tan,Steinbach, Kumar Introduction to Data Mining 1/17/ Data Mining Classification: Alternative Techniques Figures for Chapter 5 Introduction to.

Classification.

DATA MINING : CLASSIFICATION. Classification : Definition  Classification is a supervised learning.  Uses training sets which has correct answers (class.

Machine Learning Chapter 3. Decision Tree Learning

Mohammad Ali Keyvanrad

DATA MINING LECTURE 9 Classification Basic Concepts Decision Trees.

1 Data Mining Lecture 3: Decision Trees. 2 Classification: Definition l Given a collection of records (training set ) –Each record contains a set of attributes,

Chapter 4 Classification. 2 Classification: Definition Given a collection of records (training set ) –Each record contains a set of attributes, one of.

Classification Basic Concepts, Decision Trees, and Model Evaluation

Lecture 7. Outline 1. Overview of Classification and Decision Tree 2. Algorithm to build Decision Tree 3. Formula to measure information 4. Weka, data.

Machine Learning: Decision Trees Homework 4 assigned

Review - Decision Trees

Decision Tree Learning Debapriyo Majumdar Data Mining – Fall 2014 Indian Statistical Institute Kolkata August 25, 2014.

Data Mining Classification: Basic Concepts, Decision Trees, and Model Evaluation COSC 4368.

MACHINE LEARNING 10 Decision Trees. Motivation  Parametric Estimation  Assume model for class probability or regression  Estimate parameters from all.

Christoph Eick: Learning Models to Predict and Classify 1 Learning from Examples Example of Learning from Examples  Classification: Is car x a family.

Practical Issues of Classification Underfitting and Overfitting –Training errors –Generalization (test) errors Missing Values Costs of Classification.

Data Mining Classification: Basic Concepts, Decision Trees, and Model Evaluation Lecture Notes for Chapter 4 Introduction to Data Mining by Tan, Steinbach,

Classification: Basic Concepts, Decision Trees. Classification: Definition l Given a collection of records (training set ) –Each record contains a set.

Decision Trees Example of a Decision Tree categorical continuous class Refund MarSt TaxInc YES NO YesNo Married Single, Divorced < 80K> 80K Splitting.

Lecture Notes for Chapter 4 Introduction to Data Mining

Classification: Basic Concepts, Decision Trees. Classification Learning: Definition l Given a collection of records (training set) –Each record contains.

Data Mining Classification: Basic Concepts, Decision Trees, and Model Evaluation Lecture Notes for Chapter 4 Introduction to Data Mining By Tan, Steinbach,

Outline Decision tree representation ID3 learning algorithm Entropy, Information gain Issues in decision tree learning 2.

Data Mining CH6 Implementation: Real machine learning schemes(2) Reporter: H.C. Tsai.

Decision Tree Learning DA514 - Lecture Slides 2 Modified and expanded from: E. Alpaydin-ML (chapter 9) T. Mitchell-ML.

INTRODUCTION TO MACHINE LEARNING 3RD EDITION ETHEM ALPAYDIN © The MIT Press, Lecture.

Lecture Notes for Chapter 4 Introduction to Data Mining

C4.5 - pruning decision trees

Ch9: Decision Trees 9.1 Introduction A decision tree:

Christoph Eick: Learning Models to Predict and Classify 1 Learning from Examples Example of Learning from Examples  Classification: Is car x a family.

EECS 647: Introduction to Database Systems

Introduction to Data Mining, 2nd Edition by

Lecture Notes for Chapter 4 Introduction to Data Mining

Introduction to Data Mining, 2nd Edition by

Introduction to Data Mining, 2nd Edition by

Basic Concepts and Decision Trees

CSCI N317 Computation for Scientific Applications Unit Weka

INTRODUCTION TO Machine Learning 2nd Edition

Presentation transcript:

Machine Learning: Decision Trees Homework 4 assigned courtesy: Geoffrey Hinton, Yann LeCun, Tan, Steinbach, Kumar

Fully worked-out example 2 Weekend (Example) WeatherHomeworkMoney Decision (Category) W1SunnyYesRichCinema W2SunnyNoRichTennis W3WindyYesRichCinema W4RainyYesPoorCinema W5RainyNoRichStay in W6RainyYesPoorCinema W7WindyNoPoorCinema W8WindyNoRichShopping W9WindyYesRichCinema W10SunnyNoRichTennis

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Decision Tree Based Classification l Advantages: –Inexpensive to construct –Extremely fast at classifying unknown records –Easy to interpret for small-sized trees –Accuracy is comparable to other classification techniques for many simple data sets

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Tree Induction l Greedy strategy. –Split the records based on an attribute test that optimizes certain criterion. l Issues –Determine how to split the records  How to specify the attribute test condition?  How to determine the best split? –Determine when to stop splitting

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ The “training set/testing set" paradigm Slide credit: Leon Bottou (2) Learning the model

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ The “validation set” Slide credit: Leon Bottou

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Underfitting and Overfitting Overfitting Underfitting: when model is too simple, both training and test errors are large

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Overfitting due to Noise Decision boundary is distorted by noise point

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Notes on Overfitting l Overfitting results in decision trees that are more complex than necessary l Training error no longer provides a good estimate of how well the tree will perform on previously unseen records l Need new ways for estimating errors

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ How to Address Overfitting l Pre-Pruning (Early Stopping Rule) –Stop the algorithm before it becomes a fully-grown tree –Typical stopping conditions for a node:  Stop if all instances belong to the same class  Stop if all the attribute values are the same –More restrictive conditions:  Stop if number of instances is less than some user-specified threshold  Stop if class distribution of instances are independent of the available features (e.g., using  2 test)  Stop if expanding the current node does not improve impurity measures (e.g., information gain).

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ How to Address Overfitting… l Post-pruning –Grow decision tree to its entirety –Trim the nodes of the decision tree in a bottom-up fashion –If generalization error improves after trimming, replace sub-tree by a leaf node. –Class label of leaf node is determined from majority class of instances in the sub-tree

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Decision Boundary Border line between two neighboring regions of different classes is known as decision boundary Decision boundary is parallel to axes because test condition involves a single attribute at-a-time

© Tan,Steinbach, Kumar Introduction to Data Mining 4/18/ Oblique Decision Trees x + y < 1 Class = + Class = Test condition may involve multiple attributes More expressive representation Finding optimal test condition is computationally expensive