1. Data Science Algorithms: The Basic Methods
Covering Algorithms WFH: Data Mining, Chapter 4.4
Rodney Nielsen
Many of these slides were adapted from: I. H. Witten, E. Frank and M. A. Hall

2. Algorithms: The Basic Methods
Inferring rudimentary rules
Naïve Bayes, probabilistic model
Constructing decision trees
Constructing rules
Association rule learning
Linear models
Instance-based learning
Clustering

3. Covering Algorithms Convert decision tree into a rule set
Straightforward, but rule set overly complex
More effective conversions are not trivial
Instead, can generate rule set directly
For each class in turn find rule set that covers all instances in it (excluding instances not in the class)
Called a covering approach:
At each stage a rule is identified that “covers” some of the instances

4. Example: Generating a Rule
Possible rule set for class “b”:
Could add more rules, get “perfect” rule set
If true then class = a
If x > 1.2 and y > 2.6 then class = a
If x > 1.2 then class = a
Student Q: Name an example where you would want to generate "sensible" rules as opposed to perfect rules.
If x  1.2 then class = b
If x > 1.2 and y  2.6 then class = b

5. Rules vs. Trees Corresponding decision tree:
(produces exactly the same
predictions)
But: rule sets can be more clear when decision trees suffer from replicated subtrees
Note: In multiclass situations, covering algorithm concentrates on one class at a time whereas decision tree learner takes all classes into account

6. Simple Covering Algorithm
Generates a rule by adding tests that maximize rule’s accuracy
Similar to situation in decision trees: problem of selecting an attribute to split on
But: decision tree inducer maximizes overall purity
Each new test reduces rule’s coverage:

7. Selecting a Test Goal: maximize accuracy
t total number of instances covered by rule
p positive examples of the class covered by rule
(t – p number of errors made by rule)
Select test that maximizes accuracy of rule: p/t
We are finished when p/t = 1 or the set of instances can’t be split any further
Student Q: It seems like separate-and-conquer strategy has its positive in its simplicity while the advantage of the divide-and-conquer approach is its accuracy. How does it make sense for one to consider the separate-and-conquer strategy at the expense of accuracy?

8. Example: Contact Lens Data
Rule we seek:
Possible tests/conditions:
If ? then recommendation = hard
Tear production rate = Normal
0/12
Tear production rate = Reduced
Astigmatism = yes
Astigmatism = no
1/12
Spectacle prescription = Hypermetrope
3/12
Spectacle prescription = Myope
1/8
Age = Presbyopic
Age = Pre-presbyopic
2/8
Age = Young
4/12
4/12

9. Modified Rule and Resulting Data
Rule with best test added:
Instances covered by rule:
If astigmatism = yes then recommendation = hard
None
Reduced
Yes
Hypermetrope
Pre-presbyopic
Normal
Myope
Presbyopic
Hard
Young
Recommended lenses
Tear production rate
Astigmatism
Spectacle prescription
Age

10. Further Refinement Current state: Possible tests:
If astigmatism = yes and ? then recommendation = hard
Tear production rate = Normal
0/6
Tear production rate = Reduced
1/6
Spectacle prescription = Hypermetrope
3/6
Spectacle prescription = Myope
1/4
Age = Presbyopic
Age = Pre-presbyopic
2/4
Age = Young
4/6

11. Modified Rule and Resulting Data
Rule with best test added:
Instances covered by modified rule:
If astigmatism = yes and tear production rate = normal then recommendation = hard
None
Normal
Yes
Hypermetrope
Pre-presbyopic
Hard
Myope
Presbyopic
Young
Recommended lenses
Tear production rate
Astigmatism
Spectacle prescription
Age

12. Further Refinement Current state: Possible tests:
Tie between the first and the fourth test
We choose the one with greater coverage
If astigmatism = yes and tear production rate = normal and ? then recommendation = hard
1/3
Spectacle prescription = Hypermetrope
3/3
Spectacle prescription = Myope
1/2
Age = Presbyopic
Age = Pre-presbyopic
2/2
Age = Young

13. The Result
Final rule:
Second rule for recommending “hard lenses”: (built from instances not covered by first rule)
These two rules cover all “hard lenses”:
Process is repeated with other two classes
If astigmatism = yes and tear production rate = normal and spectacle prescription = myope then recommendation = hard
If age = young and astigmatism = yes and tear production rate = normal then recommendation = hard

14. Pseudo-Code for PRISM
For each class C
Initialize E to the instance set
While E contains instances in class C
Create a rule R with an empty left-hand side that predicts class C
Until p/t = 1.0 OR no_more_attributes do
For each attribute A not mentioned in R, and each value v,
Consider adding the condition A = v to the left-hand side of R
Select A and v to maximize the accuracy p/t
(break ties by choosing the condition with the largest p)
Add condition (A = v) to R
Remove the instances covered by R from E
Student Q: What kind of rule can you add to a test in order to prevent it from covering negative data examples?
Student Q: PRISM only constructs "perfect" rules by ignoring any rule with less than 100% accuracy. What kind of effect would this have on the running time, and why may it not be the appropriate method for certain situations?

15. Rules vs. Decision Lists
PRISM with outer loop removed generates a decision list set for one class
Subsequent rules are designed for examples that are not covered by previous rules
But: order doesn’t matter because all rules predict the same class
Outer loop considers all classes separately
No order dependence implied
Problems:
Overlapping rules
No guarantee all new exs. covered: Default rule required

16. Separate and Conquer
Methods like PRISM (for dealing with one class) are separate-and-conquer algorithms:
First, identify a useful rule
Then, separate out all the instances it covers
Finally, “conquer” the remaining instances
Difference with divide-and-conquer methods:
Subset covered by rule doesn’t need to be explored any further

17. Student Questions
Student Q: What is an alternative approach to divide-and-conquer classification methods?
Student Q: Since rule generation is usually more understandable to humans trying to extrapolate from the data, is rule generation considered good at describing the patterns in the data in comparison to our previously talked about algorithms?
Student Q: How do you evaluate the error rate of rule on a test validation dataset and decide if its good enough to keep it?
Student Q: How do you evaluate the error rate of rule on test
set and decide if its good enough to keep it?
Student Q: When pruning a decision tree, does it make sense to prune based on the lowest p/t?
Student Q: The rules derived from a decision tree may be much more numerous than necessary, may contain redundant terms and may not provide easily understandable classification rules if the tree is too large. Is there a way to deal with this large tree dilemma?
Student Q: Is a rule-generating method a potentially better option than a decision tree when looking for a minority class?
Student Q: Section 6.2 talks a lot about how post pruning is better than pre pruning for rule generalization but makes a point to bring forth the flaws in each method. It ends with only one method truly working but being overly complex. Couldn't a form of pre-pruning be used by using the information gain? Set a threshold for the lowest amount of information gain acceptable and then generate rules while leaving out rules under the information gain threshold?