ER-Miner: A New Method to Mine Essential Rules and Constrained Essential Rules Donghui Zhang CCIS, Northeastern University Unpublished work of our group

Outline Association rule mining Association rule mining Existing work on mining essential rules Existing work on mining essential rules Our new scheme – ER-Miner Our new scheme – ER-Miner Count the number of strong rules – ER-counter Count the number of strong rules – ER-counter Mining constrained essential rules Mining constrained essential rules Performance Performance

What Is Association Mining? Given a transactional database (TDB), where each transaction contains some items, Given a transactional database (TDB), where each transaction contains some items, Find association rules, e.g. {bread} {milk, egg}. Find association rules, e.g. {bread} {milk, egg}. Strong rule: confidence and support are high. Strong rule: confidence and support are high.

Why Is Association Mining an Essential Task in Data Mining? Foundation for many essential data mining tasks Foundation for many essential data mining tasks Association, correlation, causalityAssociation, correlation, causality Sequential patterns, temporal or cyclic association, partial periodicity, spatial and multimedia associationSequential patterns, temporal or cyclic association, partial periodicity, spatial and multimedia association Associative classification, cluster analysis, iceberg cube, fascicles (semantic data compression)Associative classification, cluster analysis, iceberg cube, fascicles (semantic data compression) Broad applications Broad applications Basket data analysis, cross-marketing, catalog design, sale campaign analysis Basket data analysis, cross-marketing, catalog design, sale campaign analysis Web log (click stream) analysis, DNA sequence analysis, etc. Web log (click stream) analysis, DNA sequence analysis, etc.

Basic Concepts: Frequent Itemsets and Association Rules Itemset X={x 1, …, x k } Itemset X={x 1, …, x k } Find all the rules X Y with min confidence and support Find all the rules X Y with min confidence and support support, s, probability that a transaction contains XYsupport, s, probability that a transaction contains XY confidence, c, conditional probability that a transaction having X also contains Y.confidence, c, conditional probability that a transaction having X also contains Y. Let min_support = 50%, min_conf = 50%: A C (50%, 66.7%) C A (50%, 100%) Customer buys diaper Customer buys both Customer buys beer Transaction- id Items bought 10 A, B, C 20 A, C 30 A, D 40 B, E, F

Mining Association Rules an Example For rule A C: support = support({A,C}) = 50% confidence = support({A,C})/support({A}) = 66.6% Min. support 50% Min. confidence 50% Transaction- id Items bought 10 A, B, C 20 A, C 30 A, D 40 B, E, F Frequent itemset Support{A}75% {B}50% {C}50% {A, C} 50%

Mining Association Rules A Two-Phase Approach Frequent Itemset Generation. Scan through TDB and find all itemsets whose support is above minsupport. Frequent Itemset Generation. Scan through TDB and find all itemsets whose support is above minsupport. Apriori, FP-Growth, …Apriori, FP-Growth, … Rule Generation. Given frequent itemset S, for every subset S, try to generate S S (controlling parameter is minconfidence). Rule Generation. Given frequent itemset S, for every subset S, try to generate S S (controlling parameter is minconfidence).

Outline Association rule mining Association rule mining Existing work on mining essential rules Existing work on mining essential rules Our new scheme – ER-Miner Our new scheme – ER-Miner Count the number of strong rules – ER-counter Count the number of strong rules – ER-counter Mining constrained essential rules Mining constrained essential rules Performance Performance

A Critical Observation RuleSupportConfidence A BC sup(ABC) sup(ABC)/sup(A) AB C sup(ABC) sup(ABC)/sup(AB) AC B sup(ABC) sup(ABC)/sup(AC) A B sup(AB) sup(AB)/sup(A) A C sup(AC) sup(AC)/sup(A) A BC has smaller support and confidence than the other rules, independent to the TDB. A BC has smaller support and confidence than the other rules, independent to the TDB. Rules AB C, AC B, A B and A C are redundant with regard to A BC. Rules AB C, AC B, A B and A C are redundant with regard to A BC. While mining association rules, a large percentage of rules may be redundant. While mining association rules, a large percentage of rules may be redundant.

Formal Definition of Essential Rule Definition 1 Rule r 1 implies another rule r 2 if support(r 1 )support(r 2 ) and confidence(r 1 ) confidence(r 2 ) independent to TDB. Definition 1 Rule r 1 implies another rule r 2 if support(r 1 )support(r 2 ) and confidence(r 1 ) confidence(r 2 ) independent to TDB. Denote as r 1 r 2 Denote as r 1 r 2 Definition 2 Rule r 1 is an essential rule if r 1 is strong and r 2 s.t. r 2 r 1. Definition 2 Rule r 1 is an essential rule if r 1 is strong and r 2 s.t. r 2 r 1.

Generate Strong Rules from an Essential Rule Theorem 1 Rule X Y X Y, iff X X, Y Y, and XY XY. Theorem 1 Rule X Y X Y, iff X X, Y Y, and XY XY. Given an essential rule A BCD, all rules it can imply can be derived by deleting a non-empty subset (e.g. {BC}) from its consequent, and add part of the deleted subset (e.g., {BC}) to its antecedent. Given an essential rule A BCD, all rules it can imply can be derived by deleting a non-empty subset (e.g. {BC}) from its consequent, and add part of the deleted subset (e.g., {BC}) to its antecedent. E.g., both ABC D and A D can be implied by A BCD. E.g., both ABC D and A D can be implied by A BCD.

Whats the Benefits of Mining Essential Rules? A strong rule where the consequent contains k items will cause other 3 k - 2 k -1 strong rules to be redundant. A strong rule where the consequent contains k items will cause other 3 k - 2 k -1 strong rules to be redundant. E.g., A BCD will cause 18 rules to be redundant. E.g., A BCD will cause 18 rules to be redundant. The set of essential rules is much more compact, and we have simple methods to derive the redundant rules. The set of essential rules is much more compact, and we have simple methods to derive the redundant rules.

Existing Work – Adjacency Lattice Aggarwal and Yu [AY98] proposed to pre- compute the in-memory structure adjacency lattice. Aggarwal and Yu [AY98] proposed to pre- compute the in-memory structure adjacency lattice. Every node corresponds to an itemset, whose support is no less than a small threshold (primary support). Every node corresponds to an itemset, whose support is no less than a small threshold (primary support). A direct link points from itemset X to Y if X Y and |X|=|Y|-1. A direct link points from itemset X to Y if X Y and |X|=|Y|-1. At query time, any minsupport provided by user is no less than the primary support. At query time, any minsupport provided by user is no less than the primary support.

Existing Work – Adjacency Lattice Null A 1%B 2%C 2%D 1% AB 0.5%AC 0.7%BC 0.4%BD 0.6% ABC 0.3%

Existing Work – Mining Essential Rules The existing algorithm to mine essential rules works as follows. The existing algorithm to mine essential rules works as follows. Given an itemset X and minconfidence, start from node X, browse the lattice upward to find every ancestor X s.t. Given an itemset X and minconfidence, start from node X, browse the lattice upward to find every ancestor X s.t. Confidence(X X - X) minconfidenceConfidence(X X - X) minconfidence This is not true for any parent node of XThis is not true for any parent node of X We report X X – X as an essential rule. We report X X – X as an essential rule.

Existing Work – Adjacency Lattice Null A 1%B 2%C 2%D 1% AB 0.5%AC 0.7%BC 0.4%BD 0.6% ABC 0.3% minconfidence=60% Input = {ABC} ERs: AB C BC A

Existing Work – Limitations Only when the antecedent union the consequent is EQUAL to some input itemset. Only when the antecedent union the consequent is EQUAL to some input itemset. E.g., if the input is {ABC}, can not find essential rule A C. E.g., if the input is {ABC}, can not find essential rule A C. In order to find all essential rules, needs to take as input all the frequent itemsets. In order to find all essential rules, needs to take as input all the frequent itemsets. A frequent itemset with n items have 2 n subsets, all of them are frequent! (except for ) A frequent itemset with n items have 2 n subsets, all of them are frequent! (except for )

Outline Association rule mining Association rule mining Existing work on mining essential rules Existing work on mining essential rules Our new scheme – ER-Miner Our new scheme – ER-Miner Count the number of strong rules – ER-counter Count the number of strong rules – ER-counter Mining constrained essential rules Mining constrained essential rules Performance Performance

Basic Concept: Max Itemset Given a TDB, a max itemset is a frequent itemset, where any superset is not frequent. Given a TDB, a max itemset is a frequent itemset, where any superset is not frequent. # max itemsets is much smaller than # frequent itemset. # max itemsets is much smaller than # frequent itemset.

Our New Scheme – ER-Miner ER-Miner takes as input the set of max itemsets. ER-Miner takes as input the set of max itemsets. ER-Miner is based on the ERM-tree ER-Miner is based on the ERM-tree Pruning techniques Pruning techniques Duplicate avoidance that minimize the number of rules we examine. Duplicate avoidance that minimize the number of rules we examine.

Lattice of rules Defined with: Defined with: set = set of all rulesset = set of all rules op = implicationop = implication Root (virtual): ABC. Root (virtual): ABC. To generate a child rule: either delete an item from the consequent, or move it to the antecedent. To generate a child rule: either delete an item from the consequent, or move it to the antecedent.

Example of a Lattice of rules (ABC) A(BC) C(AB) B(AC) AC(B)AB(C)A(B)A(C)B(A)B(C)BC(A)C(A)C(B) Idea: browse top-down; prune a sub-tree whenever an essential rule is found.

Lattice ERM-tree (ABC) A(BC) C(AB) B(AC) AC(B)AB(C)A(B)A(C) B(A)B(C)BC(A)AB(C) C(A)C(B)AC(B)BC(A)

Duplicate Elimination by the Moving Principle AB(C) was generated from A(BC) by moving B (from consequent to antecedent), while AB(C) was generated from B(AC) by moving A. AB(C) was generated from A(BC) by moving B (from consequent to antecedent), while AB(C) was generated from B(AC) by moving A. Moving principle: Move an item only if the item is bigger than all items in the antecedent. Moving principle: Move an item only if the item is bigger than all items in the antecedent.

Example of a Preliminary version of ERM-tree (ABC) A(BC) C(AB) B(AC) AC(B)AB(C)A(B)A(C) B(A)B(C)BC(A)AB(C) C(A)C(B)AC(B)BC(A)

The Deletion Principle keep track of the recent deleted item r. keep track of the recent deleted item r. Pass r to the child nodes to be generated later. Pass r to the child nodes to be generated later. When expanding a node, only delete an item if it is larger than r. When expanding a node, only delete an item if it is larger than r. A(BCD) A(CD)A(BD) A(D) delete Bdelete C delete Bdelete C r=C

The Order Principle Apply deletion before moving. Apply deletion before moving. Within each category process smaller items first. Within each category process smaller items first. A(BCD) AB(CD)A(BD) AB(D) move Bdelete C move Bdelete C After deletion is processed, here r=D

Examine the ERM-tree By incorporating the three principles, our ERM-tree does not have duplicates, while contains all rules from a given max itemset. By incorporating the three principles, our ERM-tree does not have duplicates, while contains all rules from a given max itemset. We do not pre-build the whole tree. Instead, our ERM-Tree-Examine algorithm examines each node in a breath-first order. We do not pre-build the whole tree. Instead, our ERM-Tree-Examine algorithm examines each node in a breath-first order. In this way, we guarantee: In this way, we guarantee: If r 1 r 2, r 1 is examined before r 2 ;If r 1 r 2, r 1 is examined before r 2 ; Each rule is examined at most once.Each rule is examined at most once.

Algorithm ERM-Tree-Examine Input: a max itemset M, an examine procedure. Input: a max itemset M, an examine procedure. enqueue [(M), -1] into a queue Q;enqueue [(M), -1] into a queue Q; while Q is not emptywhile Q is not empty dequeue an element [H(T), r] from Q; dequeue an element [H(T), r] from Q; examine H T unless H= ; examine H T unless H= ; if |T|=1, continue loop; if |T|=1, continue loop; for every item m in T in increasing order for every item m in T in increasing order if H and m>rif H and m>r change r to m; change r to m; enqueue [H(T-m), r] into Q; enqueue [H(T-m), r] into Q; end if end for for every item m in T in increasing order for every item m in T in increasing order if h H, m>hif h H, m>h enqueue [H{m}(T-m), r] into Q enqueue [H{m}(T-m), r] into Q end if end for end while

Example of ERM-tree (ABC), -1 A(BC), C C(AB), B B(AC), C AC(B),CAB(C),CA(B),CA(C),CB(A),CB(C),CBC(A),C C(A),B C(B),B Each node is of the form [X(Y), r], where X(Y) represents a rule X Y, and r is the recent deleted item. The root contains a virtual rule Y, and its r initially equals to -1. Each node is of the form [X(Y), r], where X(Y) represents a rule X Y, and r is the recent deleted item. The root contains a virtual rule Y, and its r initially equals to -1.

Mining Essential Rules from A Single Max Itemset Algorithm ER-Miner-Single mines essential rules from a single max itemset, by modifying ERM-Tree- Examine in the following ways. Algorithm ER-Miner-Single mines essential rules from a single max itemset, by modifying ERM-Tree- Examine in the following ways. Instantiate the examine procedure as the one to compute the confidence of a given rule;Instantiate the examine procedure as the one to compute the confidence of a given rule; Whenever a strong rule is identified, we report it as an essential rule and omit examining the sub-tree (i.e. prune the sub-tree).Whenever a strong rule is identified, we report it as an essential rule and omit examining the sub-tree (i.e. prune the sub-tree).

Local False Positive Elimination Local false positives can be eliminated by comparing it to the existing essential rules to see if any essential rule can imply it. Local false positives can be eliminated by comparing it to the existing essential rules to see if any essential rule can imply it. Since the number of essential rules is not large, even if we straightforwardly scanning through all essential rules, the performance is still acceptable. Since the number of essential rules is not large, even if we straightforwardly scanning through all essential rules, the performance is still acceptable. An optimized method is also proposed so that only part of the existing essential rules need to be compared. An optimized method is also proposed so that only part of the existing essential rules need to be compared. (ABC), -1 A(BC), C B(AC), C C(AB), B BC(A),C Local false essential rule BC A

Mining Essential Rules from Multiple Max Itemsets Given a TDB and a set of max itemsets, straightforwardly we can run ER-Miner- Single on each max itemset individually to find all essential rules. Given a TDB and a set of max itemsets, straightforwardly we can run ER-Miner- Single on each max itemset individually to find all essential rules. Two problem with the straightforward solution: Two problem with the straightforward solution: Duplicate examinationDuplicate examination Global false positivesGlobal false positives ER-Miner-Multiple integrates the solutions to the above two problems with ER-Miner- Single. ER-Miner-Multiple integrates the solutions to the above two problems with ER-Miner- Single.

Duplicate Examine Avoidance A sub-tree may exist in multiple ERM-trees Avoid such duplicate examines as follows. A sub-tree may exist in multiple ERM-trees Avoid such duplicate examines as follows. when we visit a node X(Y) of an ERM-tree, it is compared to previously examined max itemsets.when we visit a node X(Y) of an ERM-tree, it is compared to previously examined max itemsets. If some max itemset contains XY, omit examining this node (as well as the sub-tree).If some max itemset contains XY, omit examining this node (as well as the sub-tree). (ABCDF) (ABCDEH) (ABCEG) {ABCD}{ABCE} {ABC}

Global False Positive Elimination One ERM-tree may report an essential rule which implies previously identified essential rules from some other ERM-tree, (global false positives). One ERM-tree may report an essential rule which implies previously identified essential rules from some other ERM-tree, (global false positives). To eliminate the global false positives, each essential rule is compared to the existing essential rules got from other examined ERM-trees. To eliminate the global false positives, each essential rule is compared to the existing essential rules got from other examined ERM-trees. Only need to check whether an existing reported rule can be implied by the new reported rule. Only need to check whether an existing reported rule can be implied by the new reported rule. (ABCDF) (ABCDEH) B CD B CDH

Outline Association rule mining Association rule mining Existing work on mining essential rules Existing work on mining essential rules Our new scheme – ER-Miner Our new scheme – ER-Miner Count the number of strong rules – ER-counter Count the number of strong rules – ER-counter Mining constrained essential rules Mining constrained essential rules Performance Performance

Counting the Number of Strong Rules In OLAP environment, user may be also interested in knowing the number of strong rules to refine minsupport and minconfidence. In OLAP environment, user may be also interested in knowing the number of strong rules to refine minsupport and minconfidence. The set of essential rules along with the number of total strong rules will give users a better idea of how to adjust these two thresholds. The set of essential rules along with the number of total strong rules will give users a better idea of how to adjust these two thresholds.

Problem Definition Definition 3 Given a set of strong rules R={r 1,…,r n }, the derived set of R, denoted as S(r 1,…,r n ), is the set of all strong rules, each of which either belongs to R, or can be implied by some r iR. Definition 3 Given a set of strong rules R={r 1,…,r n }, the derived set of R, denoted as S(r 1,…,r n ), is the set of all strong rules, each of which either belongs to R, or can be implied by some r iR. Our ER-Counter takes as input the set of essential rules R={r 1,…,r n }, and returns|S(r 1,…,r n )|. Our ER-Counter takes as input the set of essential rules R={r 1,…,r n }, and returns|S(r 1,…,r n )|.

Non-intersected Derived Sets For n input strong rules R={r 1,…,r n }, any two derived sets, S(r i ) and S(r j ), do not intersect, |S(r 1,…,r n )|= i=1 (3 |Yi| -2 |Yi| ), where Y i is the consequent of r i. For n input strong rules R={r 1,…,r n }, any two derived sets, S(r i ) and S(r j ), do not intersect, |S(r 1,…,r n )|= i=1 (3 |Yi| -2 |Yi| ), where Y i is the consequent of r i. Especially, the size of a derived set with regard to a single rule r is 3 k -2 k, where k is the size of the consequent of r. Especially, the size of a derived set with regard to a single rule r is 3 k -2 k, where k is the size of the consequent of r.

If S(r i ) and S(r j ) intersect, it gives the upper bound to the exact number. If S(r i ) and S(r j ) intersect, it gives the upper bound to the exact number. For example, given two essential rules A BC and B AC, both rules imply AB C, and AB C is counted twice. For example, given two essential rules A BC and B AC, both rules imply AB C, and AB C is counted twice. Theorem 2 Given two strong rules r 1 : X 1 Y 1 and r 2 : X 2 Y 2, S(r 1 ) intersects with S(r 2 ) if only if r1 and r2 satisfy all three conditions: (1) Y 1Y 2, (2) X 1 -X 2Y 2, (3) X 2 -X 1 Y 1. Theorem 2 Given two strong rules r 1 : X 1 Y 1 and r 2 : X 2 Y 2, S(r 1 ) intersects with S(r 2 ) if only if r1 and r2 satisfy all three conditions: (1) Y 1Y 2, (2) X 1 -X 2Y 2, (3) X 2 -X 1 Y 1. Overlap Among Derived Sets

Theorem 3 Given two strong rules r 1 : X 1 Y 1 and r 2 : X 2 Y 2, if S(r 1 )S(r 2 ), then r= X 1X 2 Y 1Y 2, denoted as r 1r 2, must be a valid strong rule, and S(r)= S(r 1 )S(r 2 ). Theorem 3 Given two strong rules r 1 : X 1 Y 1 and r 2 : X 2 Y 2, if S(r 1 )S(r 2 ), then r= X 1X 2 Y 1Y 2, denoted as r 1r 2, must be a valid strong rule, and S(r)= S(r 1 )S(r 2 ). Then the number of derived rules of two intersected strong rules is |S(r 1 )|+|S(r 2 )|-|S(r 1r 2 )|. Then the number of derived rules of two intersected strong rules is |S(r 1 )|+|S(r 2 )|-|S(r 1r 2 )|.

Overlap Graph Each node represents an essential rule. If S(r i )S(r j ), there is an edge between r i and r j. Each node represents an essential rule. If S(r i )S(r j ), there is an edge between r i and r j. r1r1 r2r2 r4r4 r6r6 r7r7 r3r3 r5r5 r8r8

Overlap Among Derived Sets Consider the complex example, r 4 intersects with more than one rules, i.e. r 4 has more than one neighbors. Consider the complex example, r 4 intersects with more than one rules, i.e. r 4 has more than one neighbors. Recursively call ER-Counter to compute the exact number of overlapping strong rules between r 4 and its neighbors. Recursively call ER-Counter to compute the exact number of overlapping strong rules between r 4 and its neighbors. In this example, it equals to In this example, it equals to |S(r 4r 3, r 4r 5, r 4r 6 )|.

Algorithm ER-Counter Input: a set of essential rules {r 1,…,r n }. Input: a set of essential rules {r 1,…,r n }. sum = 0;sum = 0; for every rule r i in increasing orderfor every rule r i in increasing order sum = sum + 3 k -2 k, where k is the number of items in the consequent of r i. sum = sum + 3 k -2 k, where k is the number of items in the consequent of r i. Find the neighbors of r i among {r i+1,…,r n } in the conceptual overlap graph. Let the set of neighbors be {r j1,…,r jt }. Find the neighbors of r i among {r i+1,…,r n } in the conceptual overlap graph. Let the set of neighbors be {r j1,…,r jt }. if the set of neighbors is not empty if the set of neighbors is not empty sum = sum – ER-Counter(r ir j1, …, r ir jt )sum = sum – ER-Counter(r ir j1, …, r ir jt ) end if end for return sum;return sum;

Outline Association rule mining Association rule mining Existing work on mining essential rules Existing work on mining essential rules Our new scheme – ER-Miner Our new scheme – ER-Miner Count the number of strong rules – ER-counter Count the number of strong rules – ER-counter Mining constrained essential rules Mining constrained essential rules Performance Performance

Problem Definition The extended problem of finding constrained essential rules is: among all strong rules that satisfy given constraints, find those that cannot be implied by any other such rule. The extended problem of finding constrained essential rules is: among all strong rules that satisfy given constraints, find those that cannot be implied by any other such rule. Example of the constraint: the rule antecedent should contain soda. Example of the constraint: the rule antecedent should contain soda.

Mining Constrained Essential Rules Our ER-Miner scheme is extended as follows. Our ER-Miner scheme is extended as follows. Only those max itemsets that satisfy the itemset constraints will be chosen to generate ERM-trees.Only those max itemsets that satisfy the itemset constraints will be chosen to generate ERM-trees. When browsing the ERM-tree, at each node we check whether the constraint is satisfied prior to the confidence checking.When browsing the ERM-tree, at each node we check whether the constraint is satisfied prior to the confidence checking. Try to prune the sub-tree according to the constraint, if applicable.Try to prune the sub-tree according to the constraint, if applicable.

Outline Association rule mining Association rule mining Existing work on mining essential rules Existing work on mining essential rules Our new scheme – ER-Miner Our new scheme – ER-Miner Count the number of strong rules – ER-counter Count the number of strong rules – ER-counter Mining constrained essential rules Mining constrained essential rules Performance Performance

Performance Comparison