Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 1.

Slides:



Advertisements
Similar presentations
Data Mining Lecture 9.
Advertisements

1 Machine Learning: Lecture 3 Decision Tree Learning (Based on Chapter 3 of Mitchell T.., Machine Learning, 1997)
CPSC 502, Lecture 15Slide 1 Introduction to Artificial Intelligence (AI) Computer Science cpsc502, Lecture 15 Nov, 1, 2011 Slide credit: C. Conati, S.
Machine Learning in Real World: C4.5
Paper By - Manish Mehta, Rakesh Agarwal and Jorma Rissanen
Huffman code and ID3 Prof. Sin-Min Lee Department of Computer Science.
Data Mining Techniques: Classification. Classification What is Classification? –Classifying tuples in a database –In training set E each tuple consists.
Decision Tree Approach in Data Mining
Decision Tree Learning 主講人:虞台文 大同大學資工所 智慧型多媒體研究室.
Chapter 7 – Classification and Regression Trees
Chapter 7 – Classification and Regression Trees
Decision Trees IDHairHeightWeightLotionResult SarahBlondeAverageLightNoSunburn DanaBlondeTallAverageYesnone AlexBrownTallAverageYesNone AnnieBlondeShortAverageNoSunburn.
Decision Trees Jeff Storey. Overview What is a Decision Tree Sample Decision Trees How to Construct a Decision Tree Problems with Decision Trees Decision.
Lecture outline Classification Decision-tree classification.
ID3 Algorithm Abbas Rizvi CS157 B Spring What is the ID3 algorithm? ID3 stands for Iterative Dichotomiser 3 Algorithm used to generate a decision.
Classification Continued
Decision Trees IDHairHeightWeightLotionResult SarahBlondeAverageLightNoSunburn DanaBlondeTallAverageYesnone AlexBrownTallAverageYesNone AnnieBlondeShortAverageNoSunburn.
Decision Tree Pruning Methods Validation set – withhold a subset (~1/3) of training data to use for pruning –Note: you should randomize the order of training.
Classification.
Chapter 7 Decision Tree.
Fall 2004 TDIDT Learning CS478 - Machine Learning.
Machine Learning Chapter 3. Decision Tree Learning
Lecture 5 Note: Some slides and/or pictures are adapted from Lecture slides / Books of Dr Zafar Alvi. Text Book - Aritificial Intelligence Illuminated.
Learning what questions to ask. 8/29/03Decision Trees2  Job is to build a tree that represents a series of questions that the classifier will ask of.
Mohammad Ali Keyvanrad
EIE426-AICV1 Machine Learning Filename: eie426-machine-learning-0809.ppt.
Decision Trees & the Iterative Dichotomiser 3 (ID3) Algorithm David Ramos CS 157B, Section 1 May 4, 2006.
Chapter 9 – Classification and Regression Trees
For Friday No reading No homework. Program 4 Exam 2 A week from Friday Covers 10, 11, 13, 14, 18, Take home due at the exam.
For Wednesday No reading Homework: –Chapter 18, exercise 6.
Learning with Decision Trees Artificial Intelligence CMSC February 20, 2003.
For Monday No new reading Homework: –Chapter 18, exercises 3 and 4.
CS 8751 ML & KDDDecision Trees1 Decision tree representation ID3 learning algorithm Entropy, Information gain Overfitting.
Decision Trees. What is a decision tree? Input = assignment of values for given attributes –Discrete (often Boolean) or continuous Output = predicated.
MACHINE LEARNING 10 Decision Trees. Motivation  Parametric Estimation  Assume model for class probability or regression  Estimate parameters from all.
1 Universidad de Buenos Aires Maestría en Data Mining y Knowledge Discovery Aprendizaje Automático 5-Inducción de árboles de decisión (2/2) Eduardo Poggi.
CS 5751 Machine Learning Chapter 3 Decision Tree Learning1 Decision Trees Decision tree representation ID3 learning algorithm Entropy, Information gain.
Decision Trees Binary output – easily extendible to multiple output classes. Takes a set of attributes for a given situation or object and outputs a yes/no.
1 Decision Tree Learning Original slides by Raymond J. Mooney University of Texas at Austin.
Data Mining Practical Machine Learning Tools and Techniques By I. H. Witten, E. Frank and M. A. Hall Chapter 6.2: Classification Rules Rodney Nielsen Many.
Decision Tree Learning
Decision Tree Learning Presented by Ping Zhang Nov. 26th, 2007.
Decision Trees IDHairHeightWeightLotionResult SarahBlondeAverageLightNoSunburn DanaBlondeTallAverageYesnone AlexBrownTallAverageYesNone AnnieBlondeShortAverageNoSunburn.
ID3 Algorithm Amrit Gurung. Classification Library System Organise according to special characteristics Faster retrieval New items sorted easily Related.
Learning with Decision Trees Artificial Intelligence CMSC February 18, 2003.
1 Classification: predicts categorical class labels (discrete or nominal) classifies data (constructs a model) based on the training set and the values.
Decision Trees.
DECISION TREES Asher Moody, CS 157B. Overview  Definition  Motivation  Algorithms  ID3  Example  Entropy  Information Gain  Applications  Conclusion.
Outline Decision tree representation ID3 learning algorithm Entropy, Information gain Issues in decision tree learning 2.
Medical Decision Making Learning: Decision Trees Artificial Intelligence CMSC February 10, 2005.
Data Mining CH6 Implementation: Real machine learning schemes(2) Reporter: H.C. Tsai.
Review of Decision Tree Learning Bamshad Mobasher DePaul University Bamshad Mobasher DePaul University.
Chapter 6 Decision Tree.
Machine Learning Inductive Learning and Decision Trees
DECISION TREES An internal node represents a test on an attribute.
Decision Trees.
C4.5 algorithm Let the classes be denoted {C1, C2,…, Ck}. There are three possibilities for the content of the set of training samples T in the given node.
C4.5 - pruning decision trees
C4.5 algorithm Let the classes be denoted {C1, C2,…, Ck}. There are three possibilities for the content of the set of training samples T in the given node.
Artificial Intelligence
Ch9: Decision Trees 9.1 Introduction A decision tree:
Classification and Prediction
Learning with Identification Trees
Machine Learning Chapter 3. Decision Tree Learning
Machine Learning: Lecture 3
Machine Learning Chapter 3. Decision Tree Learning
Machine Learning in Practice Lecture 17
©Jiawei Han and Micheline Kamber
Decision Trees Jeff Storey.
Presentation transcript:

Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 1

Resources  Artificial Intelligence, 3 rd Edition, Patrick Henry Winston, Ch. 21  h21.pdf h21.pdf  Artificial Intelligence: A Modern Approach, 3 rd Edition, Russell, Norvig, Ch. 18.3,pg Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 2

Identification Tree  Type of Decision Tree  The Winston book call their methods SPROUTER and PRUNER, but it’s basically simplified example of an algorithm called ‘ID3’ Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 3

Identification Tree Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 4 Name (Sample ID) HairHeightWeightLotionResult SarahBlondeAverageLightNoSunburned DanaBlondeTallAverageYesNone AlexBrownShortAverageYesNone AnnieBlondeShortAverageNoSunburned EmilyRedAverageHeavyNoSunburned PeteBrownTallHeavyNoNone JohnBrownAverageHeavyNoNone KatieBlondeShortLightYesNone  Sunburn Dataset  Select one attribute to be predicted/identified  All other attributes used to identify the selected target attributed, or classification

Identification Tree  Predict Sunburns  More than one tree can correctly identify the dataset  Some trees generalize information better  Smaller trees tend to be better (Occam’s Razor)  The smallest identification tree consistent with the samples is the one most likely to identify unknown objects correctly  How to we construct the smallest/‘best’ tree? Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 5

Identification Tree  Computationally impractical to find the smallest tree when many tests are required  Use a procedure that builds small trees, but is NOT guaranteed to build the SMALLEST possible tree. Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 6

Identification Tree  Split the samples based on the best attribute  A single attribute that comes closest to correctly grouping the samples based on the target classification  Number of samples in homogeneous sets Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno

Identification Tree  Select best attribute, and repeat with remaining attributes  Must repeat for each heterogeneous branch  Only split the samples that went down that branch  The next attribute you select for one branch may be different from the attribute you select for another branch, even if they share the same parent node Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 8

Identification Tree  In real data, unlikely to get ANY homogeneous branches  Need a measure of inhomogeneity/disorder/entropy  Minimize disorder/entropy (or maximize Information Gain)  Many different measurements/calculations that can be used  Example: Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 9 Entropy(S)

Identification Tree  Results using new disorder measurement Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 10 Hair Attribute Disorder Calculation All Disorder Calculations (first Node) All Disorder Calculations (second Node)

Identification Tree  Information Gain  Expected reduction in entropy due to sorting Sample Set S on attribute A Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 11

Identification Tree  SPROUTER algorithm Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 12

Tree to Rules  Each path, from root to leaf, is a rule  The value of each attribute node are the antecedents  The leaf value is the consequence Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 13

Simplify Rules  For each rule, drop antecedents if it won’t change what the rule does on all the samples Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 14

Eliminate Rules  Once all individual rules have been simplified, you can eliminate unnecessary rules  Create a “default rule” eliminates the most rules  In the event of a tie, make up some metric to break the tie  Examples: Covers the most common consequent in the sample set Leaves the simplest rules Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 15 Most common consequent Simplest rules

Eliminate Rules Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 16

Decision Tree Algorithms  ID3 (Iterative Dichotomiser 3)  Gets stuck on local optimums, Greedy  Not good on attributes with continuous values  C4.5/J4.8  Extension of ID3  Better handling of attributes with continuous values  Can handle training data where some attribute values are missing/unknown  Handling attributes with different costs  Pruning Tree after creation  C5.0/See5.0  Commercial, closed-source  Not talking about this, but it exists Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 17

C4.5  Pruning  Helps avoid over fitting  Prepruning  Deciding not to split a set of sample any further based on some heuristic, during tree construction  Usually based on some statistical test  Chi-squared  Postpruning  Subtree Replacement  Subtree Raising Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 18

C4.5  Continuous Values  For an attribute with continuous values, sort all samples based on that attribute  Mark a ‘split point’ between samples where the classification changes  Calculate information gain on all split points  Select split point with highest information gain and use for that attribute Evolutionary Computing Systems Lab (ECSL), University of Nevada, Reno 19