Presentation is loading. Please wait.

Presentation is loading. Please wait.

Association Rule Mining

Similar presentations


Presentation on theme: "Association Rule Mining"— Presentation transcript:

1 Association Rule Mining
CENG 514 Data Mining

2 What Is Frequent Pattern Analysis?
Motivation: Finding inherent regularities in data What products were often purchased together?— Beer and diapers?! What are the subsequent purchases after buying a PC? What kinds of DNA are sensitive to this new drug? Frequent pattern: a pattern (a set of items, subsequences, substructures, etc.) that occurs frequently in a data set First proposed by Agrawal, Imielinski, and Swami [AIS93] in the context of frequent itemsets and association rule mining June 26, 2018

3 Basic Concepts: Frequent Patterns and Association Rules
Transaction-id Items bought 10 A, B, D 20 A, C, D 30 A, D, E 40 B, E, F 50 B, C, D, E, F Itemset X = {x1, …, xk} Find all the rules X  Y with minimum support and confidence support, s, probability that a transaction contains X  Y confidence, c, conditional probability that a transaction having X also contains Y Customer buys diaper buys both buys beer Let supmin = 50%, confmin = 50% Freq. Pat.: {A:3, B:3, D:4, E:3, AD:3} Association rules: A  D (60%, 100%) D  A (60%, 75%) June 26, 2018

4 Use of Association Rules
Association rules do not represent any sort of causality or correlation between the two itemsets. X  Y does not mean X causes Y, so no Causality X  Y can be different from Y  X, unlike correlation Association rules assist in marketing, targeted advertising, floor planning, inventory control, churning management, homeland security, Basket data analysis, cross-marketing, catalog design, sale campaign analysis, Web log (click stream) analysis, and DNA sequence analysis. A 100-year old furniture giant (W…) claimed bankruptcy, who started the brochure shopping in the US. Spring 2005 CSE 572, CBS 598 by H. Liu

5 Closed Patterns and Max-Patterns
A long pattern contains a combinatorial number of sub-patterns, e.g., {a1, …, a100} contains (1001) + (1002) + … + (110000) = 2100 – 1 = 1.27*1030 sub-patterns! Solution: Mine closed patterns and max-patterns instead An itemset X is closed if it is frequent and there exists no super-pattern Y כ X, with the same support as X An itemset X is a max-pattern if it is frequent and there exists no frequent super-pattern Y כ X Closed pattern is a lossless compression of freq. patterns Reducing the # of patterns and rules An itemset X is generator if it is frequent and there exists no sub-pattern X כ Y with the same support as X June 26, 2018

6 Closed Patterns and Max-Patterns
Example DB = {<a1, …, a100>, < a1, …, a50>} Minimum support count is 1. What is the set of closed itemset? <a1, …, a100>: 1 < a1, …, a50>: 2 What is the set of max-patterns? What is the set of all frequent patterns? June 26, 2018

7 Methods for Mining Frequent Patterns
Apriori Rule (The downward closure property) Any subset of a frequent itemset must be frequent If {beer, diaper, nuts} is frequent, so is {beer, diaper} i.e., every transaction having {beer, diaper, nuts} also contains {beer, diaper} Basic FIM methods: Apriori Freq. pattern growth (ClosetFPgrowth—Han, Pei & Vertical data format approach (SPADE, CHARM) June 26, 2018

8 Apriori: A Candidate Generation-and-Test Approach
Apriori pruning principle: If there is any itemset which is infrequent, its superset should not be generated/tested! (Agrawal & Mannila, et KDD’ 94) Method: Initially, scan DB once to get frequent 1-itemset Generate length (k+1) candidate itemsets from length k frequent itemsets Test the candidates against DB Terminate when no frequent or candidate set can be generated June 26, 2018

9 The Apriori Algorithm—An Example
Supmin = 2 Itemset sup {A} 2 {B} 3 {C} {D} 1 {E} Database TDB Itemset sup {A} 2 {B} 3 {C} {E} L1 C1 Tid Items 10 A, C, D 20 B, C, E 30 A, B, C, E 40 B, E 1st scan C2 Itemset sup {A, B} 1 {A, C} 2 {A, E} {B, C} {B, E} 3 {C, E} C2 Itemset {A, B} {A, C} {A, E} {B, C} {B, E} {C, E} L2 2nd scan Itemset sup {A, C} 2 {B, C} {B, E} 3 {C, E} C3 L3 Itemset {B, C, E} 3rd scan Itemset sup {B, C, E} 2 June 26, 2018

10 The Apriori Algorithm Pseudo-code: Ck: Candidate itemset of size k Lk : frequent itemset of size k L1 = {frequent items}; for (k = 1; Lk !=; k++) do begin Ck+1 = candidates generated from Lk; for each transaction t in database do increment the count of all candidates in Ck that are contained in t Lk+1 = candidates in Ck+1 with min_support end return k Lk; June 26, 2018

11 Important Details of Apriori
How to generate candidates? Step 1: self-joining Lk Step 2: pruning Example of Candidate-generation L3={abc, abd, acd, ace, bcd} Self-joining: L3*L3 abcd from abc and abd acde from acd and ace Pruning: acde is removed because ade is not in L3 C4={abcd} June 26, 2018

12 How to Generate Candidates?
Suppose the items in Lk-1 are listed in an order Step 1: self-joining Lk-1 insert into Ck select p.item1, p.item2, …, p.itemk-1, q.itemk-1 from Lk-1 p, Lk-1 q where p.item1=q.item1, …, p.itemk-2=q.itemk-2, p.itemk-1 < q.itemk-1 Step 2: pruning forall itemsets c in Ck do forall (k-1)-subsets s of c do if (s is not in Lk-1) then delete c from Ck June 26, 2018

13 How to Count Supports of Candidates?
Why counting supports of candidates a problem? The total number of candidates can be very huge One transaction may contain many candidates Method: Candidate itemsets are stored in a hash-tree Leaf node contains a list of itemsets and counts Interior node contains a hash table Subset function: finds all the candidates contained in a transaction June 26, 2018

14 Example: Counting Supports of Candidates
Transaction: 2 3 4 5 6 7 1 4 5 1 3 6 1 2 4 4 5 7 1 2 5 4 5 8 1 5 9 3 4 5 3 5 6 3 5 7 6 8 9 3 6 7 3 6 8 June 26, 2018

15 Challenges of Frequent Pattern Mining
Multiple scans of transaction database Huge number of candidates Tedious workload of support counting for candidates Improving Apriori: general ideas Reduce passes of transaction database scans Shrink number of candidates Facilitate support counting of candidates June 26, 2018

16 Partition: Scan Database Only Twice
Any itemset that is potentially frequent in DB must be frequent in at least one of the partitions of DB Scan 1: partition database and find local frequent patterns Scan 2: consolidate global frequent patterns A. Savasere, E. Omiecinski, and S. Navathe. An efficient algorithm for mining association in large databases. In VLDB’95 June 26, 2018

17 Transaction reduction
A transaction that does not contain any frequent k-itemsets cannot contain any frequent (k+1)-itemsets. Therefore, such a transaction can be marked or removed from further consideration because subsequent scans of the database for j-itemsets, where j > k, will not require it. June 26, 2018

18 Sampling for Frequent Patterns
Select a sample of original database, mine frequent patterns within sample using Apriori (PL: potentially large itemsets) Find additional candidates using negative border function PL = BD-(PL) U PL (BD-(PL) : minimal set of itemsets that are not PL, but whose subsets are all in PL) Scan database once to verify frequent itemsets found in sample and to check only borders of closure of frequent patterns (first pass) If there are uncovered new items in the result of scan C, iteratively apply BD-(C) to expand border of closure and scan database again to find missed frequent patterns. (second pass – may or may not happen) H. Toivonen. Sampling large databases for association rules. In VLDB’96 June 26, 2018

19 Sampling for Frequent Patterns
Example: DB t1: {Bread, Jelly, Peanutbutter} t2: {Bread, Peanutbutter} t3: {Bread, Milk, Peanutbutter} t4: {Beer, Bread} t5: {Beer, Milk} Support threshold 40% Sample t1: {Bread, Jelly, Peanutbutter} t2: {Bread, Peanutbutter} Support threshold 10% June 26, 2018

20 Sampling for Frequent Patterns
Sample t1: {Bread, Jelly, Peanutbutter} Support threshold 10% t2: {Bread, Peanutbutter} -> support count 1 PL = { {Bread},{Jelly},{Peanutbutter},{Bread, Jelly}, {Bread, Peanutbutter}, {Jelly, Peanutbutter}, {Bread, Jelly, Peanutbutter}} BD-(PL) = {{Beer}, {Milk}} PL U BD-(PL) = { {Bread},{Jelly},{Peanutbutter},{Bread, Jelly}, {Bread, Peanutbutter}, {Jelly, Peanutbutter}, {Bread, Jelly, Peanutbutter}, {Beer}, {Milk} } C = { {Bread},{Peanutbutter}, {Bread, Peanutbutter}, {Beer}, {Milk} } (on DB with 40% support threshold) Need one more pass June 26, 2018

21 Sampling for Frequent Patterns
C = {{Bread},{Peanutbutter}, {Bread, Peanutbutter}, {Beer}, {Milk} } BD-(C) = { {Bread, Beer}, {Bread, Milk}, {Beer, Milk}, {Beer, Peanutbutter}, {Milk, Peanutbutter}} C = C U BD-(C) BD-(C) = { {Bread, Beer, Milk}, {Bread, Beer, Peanutbutter}, {Beer, Milk, Peanutbutter}, {Bread, Milk, Peanutbutter}} BD-(C) = { {Bread, Beer, Milk, Peanutbutter} } F = { {Bread},{Peanutbutter}, {Bread, Peanutbutter}, {Beer}, {Milk} } (on DB with 40% support threshold) June 26, 2018

22 Mining Frequent Patterns Without Candidate Generation
Build a compact representation so that no need scan db several times Find frequent itemsets in a recursive way June 26, 2018

23 Construct FP-tree from a Transaction Database
TID Items bought (ordered) frequent items 100 {f, a, c, d, g, i, m, p} {f, c, a, m, p} 200 {a, b, c, f, l, m, o} {f, c, a, b, m} 300 {b, f, h, j, o, w} {f, b} 400 {b, c, k, s, p} {c, b, p} 500 {a, f, c, e, l, p, m, n} {f, c, a, m, p} min_support = 3 {} f:4 c:1 b:1 p:1 c:3 a:3 m:2 p:2 m:1 Header Table Item frequency head f 4 c 4 a 3 b 3 m 3 p 3 Scan DB once, find frequent 1-itemset (single item pattern) Sort frequent items in frequency descending order Scan DB again, construct FP-tree June 26, 2018

24 Find Patterns Having P From P-conditional Database
Starting at the frequent item header table in the FP-tree Traverse the FP-tree by following the link of each frequent item p Accumulate all of transformed prefix paths of item p to form p’s conditional pattern base {} f:4 c:1 b:1 p:1 c:3 a:3 m:2 p:2 m:1 Header Table Item frequency head f 4 c 4 a 3 b 3 m 3 p 3 Conditional pattern bases item cond. pattern base c f:3 a fc:3 b fca:1, f:1, c:1 m fca:2, fcab:1 p fcam:2, cb:1 June 26, 2018

25 From Conditional Pattern-bases to Conditional FP-trees
For each pattern-base Accumulate the count for each item in the base Construct the FP-tree for the frequent items of the pattern base m-conditional pattern base: fca:2, fcab:1 {} Header Table Item frequency head f 4 c 4 a 3 b 3 m 3 p 3 f:4 c:1 All frequent patterns relate to m m, fm, cm, am, fcm, fam, cam, fcam {} f:3 c:3 a:3 m-conditional FP-tree c:3 b:1 b:1 a:3 p:1 m:2 b:1 p:2 m:1 June 26, 2018

26 Recursion: Mining Each Conditional FP-tree
{} f:3 c:3 am-conditional FP-tree Cond. pattern base of “am”: (fc:3) {} f:3 c:3 a:3 m-conditional FP-tree {} Cond. pattern base of “cm”: (f:3) f:3 cm-conditional FP-tree {} Cond. pattern base of “cam”: (f:3) f:3 cam-conditional FP-tree June 26, 2018

27 A Special Case: Single Prefix Path in FP-tree
Suppose a (conditional) FP-tree T has a shared single prefix-path P Mining can be decomposed into two parts Reduction of the single prefix path into one node Concatenation of the mining results of the two parts a2:n2 a3:n3 a1:n1 {} b1:m1 C1:k1 C2:k2 C3:k3 b1:m1 C1:k1 C2:k2 C3:k3 r1 a2:n2 a3:n3 a1:n1 {} r1 = + June 26, 2018

28 Mining Frequent Patterns With FP-trees
Idea: Frequent pattern growth Recursively grow frequent patterns by pattern and database partition Method For each frequent item, construct its conditional pattern-base, and then its conditional FP-tree Repeat the process on each newly created conditional FP-tree Until the resulting FP-tree is empty, or it contains only one path—single path will generate all the combinations of its sub-paths, each of which is a frequent pattern June 26, 2018

29 FP-Growth vs. Apriori: Scalability With the Support Threshold
Data set T25I20D10K June 26, 2018

30 Mining by Exploring Vertical Data Format
Vertical format: t(AB) = {T11, T25, …} tid-list: list of trans.-ids containing an itemset Deriving closed patterns based on vertical intersections t(X) = t(Y): X and Y always happen together t(X)  t(Y): transaction having X always has Y Using diffset to accelerate mining Only keep track of differences of tids t(X) = {T1, T2, T3}, t(XY) = {T1, T3} Diffset (XY, X) = {T2} June 26, 2018

31 Parallel And Distributed Algorithms
Data Parallelism: parallelize the data Reduced communication cost when compared to task parallelism Initial candidates and local counts Memory at each processor should be large enough to hold all candidates at each scan Task Parallelism: parallelize the candidates Candidates are partitioned and counted separately, therefore can fit in the memory Pass not only candidates but also local set of transactions June 26, 2018

32 Parallel And Distributed Algorithms
Data Parallelism: Count Distribution Algorithm (CDA) Pass k = 1: 1. Generate local C1 from local data Di 2. Merge local candidate sets to find C1 Pass k > 1: 1. Generate complete Ck from Lk-1 2. Count local data Di to find support for Ck 3. Exchange local counts for Ck to find global Ck 4. Compute Lk from Ck June 26, 2018

33 Parallel And Distributed Algorithms
Task Parallelism: Data Distribution Algorithm (DDA) Partition data + candidate sets 1. Generate Ck from Lk-1; retain |Ck| / P locally. 2. Count local Ck using both local and remote data 3. Calculate local Lk using local Ck and synchronize June 26, 2018

34 Sequential Associations
Sequence: ordered list of itemsets I = {i1, i2, …, im} set of items S = <s1, s2,…,sn>, where s1 I e.g: <{A,B},{B,C},{C}> Subsequence of a given sequence is one that can be obtained by removing some items and any resulting empty itemsets from the original sequence. e.g: <{A},{C}> is a subsequence of <{A,B},{B,C},{C}> {A,C},{B} is NOT subsequence of the above sequence June 26, 2018

35 Sequential Associations
Support(s): percentage of sessions (customers) whose session-sequence contains the sequence s Confidence(s → t): ratio of the number of sessions that contain both sequences s and t to the number of ones that contain s. Example: s = <{A},{C}> Support(s) = 1/3 u = <{B,C},{D}> Support(u) = 2/3 Customer Time Itemset C1 10 AB 20 BC 30 D C2 15 ABC C3 ACD June 26, 2018

36 Sequential Associations
Basic Algorithms: AprioriAll Basic algorithm for finding sequential associations GSP More efficient than AprioriAll Includes extensions Window Concept Hierarchy Spade Improves performance by using vertical table structure Equivalence classes Episode mining Events are ordered with occurance time Uses window June 26, 2018

37 Sequential Associations
Support for a sequence: Fraction of total customers that support a sequence. Large Sequence: Sequence that meets min_sup. Maximal Sequence: A sequence that is not contained in any other sequence. June 26, 2018

38 Sequential Associations - Example
Customer ID Customer Sequence 1 { (30) (90) } 2 { (10 20) (30) ( ) } 3 { ( ) } 4 { (30) (40 70) (90) } 5 { (90) } Customer ID Transaction Time Items Bought 1 June 25 '93 30 June 30 '93 90 2 June 10 '93 10,20 June 15 '93 June 20 '93 40,60,70 3 30,50,70 4 40,70 July 25 '93 5 June 12 '93 Maximal sequences with support > 25% { (30) (90) } { (30) (40 70) } Min-sup:25% { (10 20) (30) } Does not satisfy min_sup (Only supported by Cust. 2) { (30) }, { (70) }, { (30) (40) } … are not maximal.

39 Frequent Sequence Generation
L1 1-Seq Support <1> 4 <2> 2 <3> <4> <5> L2 2-Seq Support <1 2> 2 <1 3> 4 <1 4> 3 <1 5> <2 3> <2 4> <3 4> <3 5> <4 5> L3 3-Seq Support <1 2 3> 2 <1 2 4> <1 3 4> 3 <1 3 5> <2 3 4> Customer Sequences <{1 5} {2} {3} {4} > <{1} {3} {4} {3 5}> <{1} {2} {3} {4}> <{1} {3} {5}> <{4} {5}> <3 4 5> 1 L4 4-Seq Support < > 2 <2 5> Maximal: < >, <1 3 5>, <4 5> Minisup = 25%

40 Sequential Associations
Candidate Generation Initially find 1-element frequent sequence At subsequent passes, each pass starts the frequent sequences found in the previous pass. Each candidate sequence has one more item than a previously found frequent sequence; so all the candidate sequences in the same pass will have the same number of items Candidate generation Join: s1 joins with s2 if the subsequence obtained by dropping the first item of s1 is the same as the subsequence obtained by dropping the last element of s2. s1 is extended with the last item in s2. Prune: delete candidate if it has an infrequent contiguous subsequence

41 Sequential Associations
Initial candidates: <a>, <b>, <c>, <d>, <e>, <f>, <g>, <h> Scan database once, count support for candidates Cand Sup <a> 3 <b> 5 <c> 4 <d> <e> <f> 2 <g> 1 <h> <a(bd)bcb(ade)> 50 <(be)(ce)d> 40 <(ah)(bf)abf> 30 <(bf)(ce)b(fg)> 20 <(bd)cb(ac)> 10 Sequence Seq. ID min_sup =2

42 Sequential Associations
51 length-2 Candidates <a> <b> <c> <d> <e> <f> <aa> <ab> <ac> <ad> <ae> <af> <ba> <bb> <bc> <bd> <be> <bf> <ca> <cb> <cc> <cd> <ce> <cf> <da> <db> <dc> <dd> <de> <df> <ea> <eb> <ec> <ed> <ee> <ef> <fa> <fb> <fc> <fd> <fe> <ff> <a> <b> <c> <d> <e> <f> <(ab)> <(ac)> <(ad)> <(ae)> <(af)> <(bc)> <(bd)> <(be)> <(bf)> <(cd)> <(ce)> <(cf)> <(de)> <(df)> <(ef)> Without Apriori property, 8*8+8*7/2=92 candidates Apriori prunes 44.57% candidates

43 Sequential Associations
GSP: <a> <b> <c> <d> <e> <f> <g> <h> <aa> <ab> … <af> <ba> <bb> … <ff> <(ab)> … <(ef)> <abb> <aab> <aba> <baa> <bab> … <abba> <(bd)bc> … <(bd)cba> 1st scan: 8 cand. 6 length-1 seq. pat. 2nd scan: 51 cand. 19 length-2 seq. pat. 10 cand. not in DB at all 3rd scan: 46 cand. 19 length-3 seq. pat. 20 cand. not in DB at all 4th scan: 8 cand. 6 length-4 seq. pat. 5th scan: 1 cand. 1 length-5 seq. pat. Cand. cannot pass sup. threshold Cand. not in DB at all <a(bd)bcb(ade)> 50 <(be)(ce)d> 40 <(ah)(bf)abf> 30 <(bf)(ce)b(fg)> 20 <(bd)cb(ac)> 10 Sequence Seq. ID min_sup =2

44 Sequential Associations
Example: L3 Freq. 3-sequences C4 (candidate 4-seq) After join After pruning < (1,2) (3)> < (1,2) (4)> < (1) (3,4)> < (1,3) (5)> < (2) (3,4)> < (2) (3) (5)> < (1,2) (3,4)> <(1,2) (3) (5) > <(1,2) (3,4)>

45 Sequential Associations
Extended features Concept hierarchies (taxonomies) Sliding window Time constraints

46 Sequential Associations
Spy Le Carre Perfect Spy Smiley’s People Example: min-sup count =2 <(Ringworld) (Ringworld Engineers)> sliding window =7 days <(Foundation, Ringworld) (Ringworld Engineers)> max-gap= 30 days no patterns taxonomy <(Foundation) (Asimov)> Science Fiction Asimov Niven Foundation Foundation and Empire Second Ringworld Ringworld Engineers Sequence-Id Transaction Time Items C1 1 Ringworld 2 Foundation 15 Ringworld Engineers, Second Foundation C2 Foundation, Ringworld 20 Foundation and Empire 50 Ringworld Engineers

47 Sequential Associations
SPADE (Sequential PAttern Discovery using Equivalent Class) developed by Zaki 2001 A vertical format sequential pattern mining method A sequence database is mapped to a large set of Item: <SID, EID> Sequential pattern mining is performed by growing the subsequences (patterns) one item at a time by Apriori candidate generation

48 Sequential Associations - SPADE

49 Mining Multiple-Level Association Rules
Items often form hierarchies Flexible support settings Items at the lower level are expected to have lower support Exploration of shared multi-level mining uniform support Milk [support = 10%] 2% Milk [support = 6%] Skim Milk [support = 4%] Level 1 min_sup = 5% Level 2 min_sup = 3% reduced support June 26, 2018

50 Mining Multi-Dimensional Association
Single-dimensional rules: buys(X, “milk”)  buys(X, “bread”) Multi-dimensional rules:  2 dimensions or predicates Inter-dimension assoc. rules (no repeated predicates) age(X,”19-25”)  occupation(X,“student”)  buys(X, “coke”) hybrid-dimension assoc. rules (repeated predicates) age(X,”19-25”)  buys(X, “popcorn”)  buys(X, “coke”) June 26, 2018

51 Interestingness Measure: Correlation (Lift)
play basketball  eat cereal [40%, 66.7%] is misleading The overall % of students eating cereal is 75% > 66.7%. play basketball  not eat cereal [20%, 33.3%] is more accurate, although with lower support and confidence Measure of dependent/correlated events: lift (also called correlation, interest) Basketball Not basketball Sum (row) Cereal 2000 1750 3750 Not cereal 1000 250 1250 Sum(col.) 3000 5000 June 26, 2018

52 Interestingness Measure: Conviction
Measures the independence of negation of implication Conviction (A → B) = P(A) P(-B) / P(A, -B) 1: not related ∞: always hold Example: Conviction (basketball → cereal) = (3000/5000) * (1250/5000) / (1000/5000) = 0.75 Conviction (basketball → not cereal) = (3000/5000) * (3750/5000) / (2000/5000) = 1.12 Conviction (not basketball → cereal) = (2000/5000) * (1250/5000) / (250/5000) = 2 Basketball Not basketball Sum (row) Cereal 2000 1750 3750 Not cereal 1000 250 1250 Sum(col.) 3000 5000 June 26, 2018

53 Interestingness Measure: Conviction
Conviction (A → B) = P(A) P(-B) / P(A, -B) Example: t1: {Bread, Jelly, Peanutbutter} t2: {Bread, Peanutbutter} t3: {Bread, Milk, Peanutbutter} t4: {Beer, Bread} t5: {Beer, Milk} conviction (Peanutbutter → Bread) = (3/5 * 1/5) / 0 = ∞ conviction (Bread → Peanutbutter) = (4/5 * 2/5) / (1/5) = 8/5 > 1 June 26, 2018

54 Interestingness Measure: f-measure
f-measure = (( 1+ α2) * support * confidence ) / (α * s ) + c α takes a value in [0,1], when 0 support has min effect Another representation F-metric = ((B2 +1 ) * confidence * support ) / (( B * confidence) + support)) June 26, 2018

55 Which Measures Should Be Used?
all-conf or coherence could be good measures Both all-conf and coherence have the downward closure property June 26, 2018


Download ppt "Association Rule Mining"

Similar presentations


Ads by Google