Fast Effective Rule Induction

Slides:

Advertisements

Similar presentations

Data Mining Classification: Alternative Techniques

Advertisements

1 CS 391L: Machine Learning: Rule Learning Raymond J. Mooney University of Texas at Austin.

DECISION TREES. Decision trees  One possible representation for hypotheses.

1 Machine Learning: Lecture 3 Decision Tree Learning (Based on Chapter 3 of Mitchell T.., Machine Learning, 1997)

Rule-Based Classifiers. Rule-Based Classifier Classify records by using a collection of “if…then…” rules Rule: (Condition)  y –where Condition is a conjunctions.

From Decision Trees To Rules

3-1 Decision Tree Learning Kelby Lee. 3-2 Overview ¨ What is a Decision Tree ¨ ID3 ¨ REP ¨ IREP ¨ RIPPER ¨ Application.

Paper By - Manish Mehta, Rakesh Agarwal and Jorma Rissanen

Learning Rules from Data

RIPPER Fast Effective Rule Induction

Decision Tree Approach in Data Mining

Iterative Optimization and Simplification of Hierarchical Clusterings Doug Fisher Department of Computer Science, Vanderbilt University Journal of Artificial.

Decision Tree Learning 主講人：虞台文大同大學資工所智慧型多媒體研究室.

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

Decision Tree Rong Jin. Determine Milage Per Gallon.

Iterative Optimization of Hierarchical Clusterings Doug Fisher Department of Computer Science, Vanderbilt University Journal of Artificial Intelligence.

© Vipin Kumar CSci 8980 Fall CSci 8980: Data Mining (Fall 2002) Vipin Kumar Army High Performance Computing Research Center Department of Computer.

INTRODUCTION TO Machine Learning ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

Covering Algorithms. Trees vs. rules From trees to rules. Easy: converting a tree into a set of rules –One rule for each leaf: –Antecedent contains a.

© Vipin Kumar CSci 8980 Fall CSci 8980: Data Mining (Fall 2002) Vipin Kumar Army High Performance Computing Research Center Department of Computer.

Basic Data Mining Techniques

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

Fall 2004 TDIDT Learning CS478 - Machine Learning.

Mohammad Ali Keyvanrad

Weka Project assignment 3

Decision Trees DefinitionDefinition MechanismMechanism Splitting FunctionSplitting Function Issues in Decision-Tree LearningIssues in Decision-Tree Learning.

Scaling up Decision Trees. Decision tree learning.

Decision-Tree Induction & Decision-Rule Induction

Bab 5 Classification: Alternative Techniques Part 1 Rule-Based Classifer.

CS690L Data Mining: Classification

Rule Induction Overview Generic separate-and-conquer strategy CN2 rule induction algorithm Improvements to rule induction.

MACHINE LEARNING 10 Decision Trees. Motivation  Parametric Estimation  Assume model for class probability or regression  Estimate parameters from all.

1 Universidad de Buenos Aires Maestría en Data Mining y Knowledge Discovery Aprendizaje Automático 5-Inducción de árboles de decisión (2/2) Eduardo Poggi.

Decision Trees Binary output – easily extendible to multiple output classes. Takes a set of attributes for a given situation or object and outputs a yes/no.

Class1 Class2 The methods discussed so far are Linear Discriminants.

By Ankur Khator Gaurav Sharma Arpit Mathur 01D05014 SPAM FILTERING.

Data Mining Practical Machine Learning Tools and Techniques By I. H. Witten, E. Frank and M. A. Hall Chapter 6.2: Classification Rules Rodney Nielsen Many.

Learning to Detect Faces A Large-Scale Application of Machine Learning (This material is not in the text: for further information see the paper by P.

CSC 8520 Spring Paula Matuszek DecisionTreeFirstDraft Paula Matuszek Spring,

Classification with Decision Trees and Rules. Copyright © Andrew W. Moore Density Estimation – looking ahead Compare it against the two other major kinds.

RULE-BASED CLASSIFIERS

Outline Decision tree representation ID3 learning algorithm Entropy, Information gain Issues in decision tree learning 2.

Ensemble Learning, Boosting, and Bagging: Scaling up Decision Trees (with thanks to William Cohen of CMU, Michael Malohlava of 0xdata, and Manish Amde.

Data Mining CH6 Implementation: Real machine learning schemes(2) Reporter: H.C. Tsai.

INTRODUCTION TO MACHINE LEARNING 3RD EDITION ETHEM ALPAYDIN © The MIT Press, Lecture.

CSE573 Autumn /11/98 Machine Learning Administrative –Finish this topic –The rest of the time is yours –Final exam Tuesday, Mar. 17, 2:30-4:20.

DECISION TREES An internal node represents a test on an attribute.

CSPs: Search and Arc Consistency Computer Science cpsc322, Lecture 12

Computational Intelligence: Methods and Applications

Decision Trees.

Decision Tree Learning

Rule Induction for Classification Using

CSPs: Search and Arc Consistency Computer Science cpsc322, Lecture 12

Data Science Algorithms: The Basic Methods

Data Mining Classification: Alternative Techniques

Issues in Decision-Tree Learning Avoiding overfitting through pruning

Rule Learning for Go Introduction Data Extraction Bad Move Problems

CSPs: Search and Arc Consistency Computer Science cpsc322, Lecture 12

Machine Learning Chapter 3. Decision Tree Learning

Machine Learning: Lecture 3

Classification Algorithms

Rule Learning Hankui Zhuo April 28, 2018.

Data Mining Rule Classifiers

Machine Learning Chapter 3. Decision Tree Learning

Data Mining Rule Classifiers

INTRODUCTION TO Machine Learning

Chapter 7: Transformations

INTRODUCTION TO Machine Learning 2nd Edition

Using Bayesian Network in the Construction of a Bi-level Multi-classifier. A Case Study Using Intensive Care Unit Patients Data B. Sierra, N. Serrano,

Decision Trees Jeff Storey.

Presentation transcript:

Fast Effective Rule Induction By William W. Cohen

Overview Rule Based Learning Rule Learning Algorithm Pruning Techniques Modifications to IREP Evolution of Ripper Conclusion

Goal of the Paper The goal of this paper is to develop a rule learning algorithm that perform efficiently on a large noisy datasets and are competitive in generalization performance with more mature symbolic learning methods, such as decision trees.

Concepts to Refresh Overfit and simplify strategy - Separate and Conquer Pruning

Separate and Conquer General Idea: 1. Learn one rule that covers certain number of positive examples 2. Remove those examples covered by the rule 3. Repeat until no positive examples are left.

Sequential Covering Algorithm Sequential-Covering(class,attributes,examples,threshold T) RuleSet = 0 Rule = Learn-one-rule(class,attributes,examples) While (performance(Rule) > T) do a. RuleSet += Rule b. Examples = Examples \ {examples classified correctly by Rule} c. Rule = Learn-one-rule(class,attributes,examples) Sort RuleSet based on the performance of the rules Return RuleSet

Pruning Why do we need pruning? Techniques of pruning: 1. Reduced Error Pruning 2. Grow 3. Incremental Reduced Error Pruning

IREP Algorithm

How to build a rule in IREP? First the uncovered examples are randomly partitioned into two subsets, a growing set and a pruning set. Next a rule is grown.The implementation of a Grow rule is a propositional version of FOIL.

Grow Rule It begins with an empty conjunction of lconditions and considers adding to this any condition of the form An=v,Ac<=@ or Ac>=@ where An is a nominal attribute and v is a legal value for An or Ac is a continuous attribute and 2 is some value for Ac that occurs in the training data.

Grow Rule Grow rule repeatedly adda the conditions that maximizes FOIL’s information gain criterion until the rule covers no negative examples from the growing dataset.

Pruning After growing,the rule is immediately pruned by deleting any final sequence of conditions from the rule, and chooses the deletion that maximizes the function v(Rule,PrunePos,PruneNeg)= p+(N-n) / P+N

IREP IREP algorithm works for - Two-class problems - Multiple classes - Handles missing attributes

Experiments with IREP The First Graph

CPU times for C4.5,IREP and RIPPER2

Improvements to IREP Improvement in IREP needs modifications 1.The Real Value Metric 2. The stopping criterion 3. Rule optimization

Evolution of RIPPER First IREP* is used to obtain the initial rule set.This rule set is next optimized and finally rules are added to cover any remaining positive examples using IREP*.This leads to a new algorithm , namely RIPPER