Comparing Univariate and Multivariate Decision Trees Olcay Taner Yıldız Ethem Alpaydın Department of Computer Engineering Bogazici University E-mail:

Slides:

Advertisements

Similar presentations

DECISION TREES. Decision trees  One possible representation for hypotheses.

Advertisements

Random Forest Predrag Radenković 3237/10

CHAPTER 9: Decision Trees

Learning Rules from Data

C4.5 - pruning decision trees

Classification Techniques: Decision Tree Learning

A Quick Overview By Munir Winkel. What do you know about: 1) decision trees 2) random forests? How could they be used?

Decision Trees Instructor: Qiang Yang Hong Kong University of Science and Technology Thanks: Eibe Frank and Jiawei Han.

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

Radosław Wesołowski Tomasz Pękalski, Michal Borkowicz, Maciej Kopaczyński

1 Classification with Decision Trees Instructor: Qiang Yang Hong Kong University of Science and Technology Thanks: Eibe Frank and Jiawei.

Classification: Decision Trees

1 Classification with Decision Trees I Instructor: Qiang Yang Hong Kong University of Science and Technology Thanks: Eibe Frank and Jiawei.

Tree-based methods, neutral networks

© Vipin Kumar CSci 8980 Fall CSci 8980: Data Mining (Fall 2002) Vipin Kumar Army High Performance Computing Research Center Department of Computer.

Classification Continued

Decision Trees an Introduction.

Three kinds of learning

INTRODUCTION TO Machine Learning ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

Example of a Decision Tree categorical continuous class Splitting Attributes Refund Yes No NO MarSt Single, Divorced Married TaxInc NO < 80K > 80K.

Classification.

© Prentice Hall1 DATA MINING Introductory and Advanced Topics Part II Margaret H. Dunham Department of Computer Science and Engineering Southern Methodist.

Ordinal Decision Trees Qinghua Hu Harbin Institute of Technology

Classification: Decision Trees 2 Outline  Top-Down Decision Tree Construction  Choosing the Splitting Attribute  Information Gain and Gain Ratio.

Ensemble Learning (2), Tree and Forest

Decision Tree Learning

Chapter 7 Decision Tree.

A Comparison of Discriminant Functions and Decision Tree Induction Techniques for Evaluation of Antenatal Fetal Risk Assessment Nilgün Güler, Olcay Taner.

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

Data Mining: Classification

Fall 2004 TDIDT Learning CS478 - Machine Learning.

Mohammad Ali Keyvanrad

Computational Intelligence: Methods and Applications Lecture 19 Pruning of decision trees Włodzisław Duch Dept. of Informatics, UMK Google: W Duch.

Data Mining Practical Machine Learning Tools and Techniques Chapter 4: Algorithms: The Basic Methods Section 4.3: Decision Trees Rodney Nielsen Many of.

Combining multiple learners Usman Roshan. Bagging Randomly sample training data Determine classifier C i on sampled data Goto step 1 and repeat m times.

For Wednesday No reading Homework: –Chapter 18, exercise 6.

For Monday No new reading Homework: –Chapter 18, exercises 3 and 4.

CS 8751 ML & KDDDecision Trees1 Decision tree representation ID3 learning algorithm Entropy, Information gain Overfitting.

MACHINE LEARNING 10 Decision Trees. Motivation  Parametric Estimation  Assume model for class probability or regression  Estimate parameters from all.

CS 5751 Machine Learning Chapter 3 Decision Tree Learning1 Decision Trees Decision tree representation ID3 learning algorithm Entropy, Information gain.

1 Decision Tree Learning Original slides by Raymond J. Mooney University of Texas at Austin.

Decision Trees Example of a Decision Tree categorical continuous class Refund MarSt TaxInc YES NO YesNo Married Single, Divorced < 80K> 80K Splitting.

Decision Tree Learning

Decision Tree Learning Presented by Ping Zhang Nov. 26th, 2007.

Neural Trees Olcay Taner Yıldız, Ethem Alpaydın Boğaziçi University Computer Engineering Department

Decision Trees.

Classification and Regression Trees

Combining multiple learners Usman Roshan. Decision tree From Alpaydin, 2010.

DECISION TREES Asher Moody, CS 157B. Overview  Definition  Motivation  Algorithms  ID3  Example  Entropy  Information Gain  Applications  Conclusion.

Outline Decision tree representation ID3 learning algorithm Entropy, Information gain Issues in decision tree learning 2.

Eco 6380 Predictive Analytics For Economists Spring 2016 Professor Tom Fomby Department of Economics SMU.

Ensemble Learning, Boosting, and Bagging: Scaling up Decision Trees (with thanks to William Cohen of CMU, Michael Malohlava of 0xdata, and Manish Amde.

Tree and Forest Classification and Regression Tree Bagging of trees Boosting trees Random Forest.

BY International School of Engineering {We Are Applied Engineering} Disclaimer: Some of the Images and content have been taken from multiple online sources.

Decision Trees by Muhammad Owais Zahid

Decision Tree Learning DA514 - Lecture Slides 2 Modified and expanded from: E. Alpaydin-ML (chapter 9) T. Mitchell-ML.

Review of Decision Tree Learning Bamshad Mobasher DePaul University Bamshad Mobasher DePaul University.

INTRODUCTION TO MACHINE LEARNING 3RD EDITION ETHEM ALPAYDIN © The MIT Press, Lecture.

Rodney Nielsen Many / most of these slides were adapted from: I. H. Witten, E. Frank and M. A. Hall Data Mining Practical Machine Learning Tools and Techniques.

10. Decision Trees and Markov Chains for Gene Finding.

Chapter 6 Decision Tree.

DECISION TREES An internal node represents a test on an attribute.

Artificial Intelligence

Trees, bagging, boosting, and stacking

Ch9: Decision Trees 9.1 Introduction A decision tree:

Data Science Algorithms: The Basic Methods

Statistical Learning Dong Liu Dept. EEIS, USTC.

Classification with CART

INTRODUCTION TO Machine Learning

INTRODUCTION TO Machine Learning 2nd Edition

Presentation transcript:

Comparing Univariate and Multivariate Decision Trees Olcay Taner Yıldız Ethem Alpaydın Department of Computer Engineering Bogazici University

Univariate Trees (ID3) Constructs decision trees top-down manner. Select the best attribute to test at the root node by using a statistical test. Descendants of the root node are created for each possible value of the attribute. Two for numeric attributes as x i a, m for symbolic attributes as x i = a k, k = 1, …, m.

Partition Merit Criteria –Information Gain Entropy = Sum i (p i logp i ) –Weak Theory Learning Measure –Gini Index Avoiding Overfitting –Pre-pruning –Post-pruning ID3 Continued

Univariate versus Multivariate

Classification and Regression Trees (CART) Each instance is first normalized. Algorithm takes a set of coefficients W=(w 1,…, w n ) and searches for the best split of the form v=Sum i (w i x i )  c for i=1 to n. Algorithm cycles through the attributes x 1,…, x n at each step doing a search for an improved split. At each cycle CART searches for the best split of the form v-  (x i +  )  c. The search for  is carried out for  = -0.25, 0.0, Best of  and  are used to update linear combination.

CART continued Univariate vs Multivariate Splits Symbolic and Numeric Features conversion Color: (red, green, blue) red: 100 green:010 blue:001 Feature Selection –The most important single variable is the one whose deletion causes the greatest deterioration.

Conclusions for ID3 For three partition merit criteria (Entropy, Weak Theory Learning Measure, Gini Index) there is no significant difference in accuracy, node size and learning time difference between them. Pruning increases accuracy and post-pruning is better than pre-pruning in case of accuracy and node size at the expense of more computation time.

Conclusions for CART When feature selection is applied, CART accuracy is statistically significantly increased and node size is decreased in 13 datasets out of 15. Multivariate method CART does not always increase accuracy and does not always lower node size.

Questions