1. Learning Rules from Data
Olcay Taner Yıldız, Ethem Alpaydın
Department of Computer Engineering
Boğaziçi University

2. Rule Induction? Derive meaningful rules from data
Mainly used in classification problems
Attribute types (Continuous, Discrete)
Name
Income
Owns a house?
Marital status
Default
Ali
25,000 $
Yes
Married No
Veli
18,000 $ No
Married
Yes

3. Rules Disjunction: conjunctions are binded via OR's
Conjunction: propositions are binded via AND's
Proposition: relation on an attribute
Attribute is Continuous (defines a subinterval)
Attribute is Discrete (= one of the values of the attribute)

4. How to generate rules? Rule Induction Techniques Via Trees
C4.5Rules
Directly from Data
Ripper

5. C4.5Rules (Quinlan, 93) Create decision tree using C4.5
Convert the decision tree to a ruleset by writing each path from root to the leaves as a rule

6. C4.5Rules q1 x1 > q1 x2 > q2 y = 0 y = 1 yes no q2 x2 : savings
x1 : yearly-income q1
x1 > q1
x2 > q2
y = 0
y = 1
yes no OK
DEFAULT q2
Rules:
IF (x1 > q1) AND (x2 > q2) THEN Y = OK
IF (x1 < q1) OR ((x1 > q1) AND (x2 < q2)) THEN
Y = DEFAULT

7. RIPPER (Cohen, 95) Learn rule for each class Two Steps
Objective class is positive, other classes are negative
Two Steps
Initialization
Learn rules one by one
Immediately prune learned rule
Optimization
Since search is greeedy, pass k times over the rules to optimize

8. RIPPER (Initialization)
Split (pos, neg) into growset and pruneset
Rule := grow conjunction with growset
Add propositions one by one
IF (x1 > q1) AND (x2 > q2) AND (x2 < q3) THEN Y = OK
Rule := prune conjunction with pruneset
Remove propositions
IF (x1 > q1) AND (x2 < q3) THEN Y = OK
If MDL < Best MDL + 64
Add conjunction
Else
Break

9. Ripper (Optimization)
Repeat K times
For each rule
IF (x1 > q1) AND (x2 < q3) THEN Y = OK
Generate revisionrule by adding propositions
IF (x1 > q1) AND (x2 < q3) AND (x1 > q3) THEN Y = OK
Generate replacementrule by regrowing
IF (x1 > q4) THEN Y = OK
Compare current rule with revisionrule and replacementrule
Take the best according to MDL

10. Minimum Description Length
Description Length of a Ruleset
Description Length of Rules
S = ||k|| + k log2 (n / k) + (n – k) log2 (k / (n – k))
Description Length of Exceptions
S = log2 (|D| + 1)
+ fp (-log2 (e / 2C))
+ (C – fp) (-log2 (1 - e / 2C))
+ fn (-log2 (fn / 2U))
+ (U – fn) (-log2 (1 - fn / 2U))

11. Ripper*
Finding best condition is done by trying all possible split points (time-consuming)
Shortcut: Linear Discriminant Analysis
Split point is calculated analytically
To be more robust
Instances further than 3  are removed
If number of examples < 20, shortcut not used

12. Experiments 29 datasets from UCI repository are used
10 fold cross-validation
Comparison done using one-sided t test
Comparison of three algorithms C4.5Rules, Ripper, Ripper*
Comparison based on
Error Rate
Complexity of the rulesets
Learning Time

13. Error Rate (I)
Ripper and its variant have better performance than C4.5Rules
Ripper* has similar performance compared to Ripper
C4.5Rules has advantage when the number of rules are small (Exhaustive Search)

14. Error Rate (II)
C4.5Rules
Ripper
Ripper*
Total - 4 7 12 2 10 1 5 9

15. Ruleset Complexity (I)
Ripper and Ripper* produce significantly small number of rules compared to C4.5Rules
C4.5Rules starts with an unpruned tree, which is a large amount of rule to start

16. Ruleset Complexity (II)
C4.5Rules
Ripper
Ripper*
Total - 1 26 10 27 2 3 11

17. Learning Time (I)
Ripper* better than Ripper, which is better than C4.5Rules
C4.5Rules O(N3)
Ripper O(Nlog2N)
Ripper* O(NlogN)

18. Learning Time (II)
C4.5Rules
Ripper
Ripper*
Total - 2 23 25 13 27

19. Conclusion
Comparison of two rule induction algorithms C4.5Rules and Ripper
Proposed a shortcut in learning conditions using LDA (Ripper*)
Ripper is better than C4.5Rules
Ripper* improves learning time of Ripper without decreasing its performance