1. Data Mining: Concepts and Techniques
Mining Frequent
Patterns & Association Rules
Dr. Maher Abuhamdeh

2. What Is Frequent Pattern Analysis?
Frequent pattern: a pattern (a set of items, subsequences, substructures, etc.) that occurs frequently in a data set
First proposed by Agrawal, Imielinski, and Swami [AIS93] in the context of frequent itemsets and association rule mining
Motivation: Finding inherent regularities in data
What products were often purchased together?— Tea and Milk?!
What are the subsequent purchases after buying a PC?
What kinds of DNA are sensitive to this new drug?
Can we automatically classify web documents?
Applications
Basket data analysis, cross-marketing, catalog design, sale campaign analysis,.

3. Applications
Market Basket Analysis: given a database of customer transactions, where each transaction is a set of items the goal is to find groups of items which are frequently purchased together.
Telecommunication (each customer is a transaction containing the set of phone calls)
Credit Cards/ Banking Services (each card/account is a transaction containing the set of customer’s payments)
Medical Treatments (each patient is represented as a transaction containing the ordered set of diseases)
Basketball-Game Analysis (each game is represented as a transaction containing the ordered set of ball passes)

4. Basic Concepts: Frequent Patterns
Tid
Items bought 10
Tea, Nuts, Diaper 20
Tea, Coffee, Diaper 30
Tea, Diaper, Eggs 40
Nuts, Eggs, Milk 50
Nuts, Coffee, Diaper, Eggs, Milk
itemset: A set of one or more items
k-itemset X = {x1, …, xk}
(absolute) support, or, support count of X: Frequency or occurrence of an itemset X
(relative) support, s, is the fraction of transactions that contains X (i.e., the probability that a transaction contains X)
An itemset X is frequent if X’s support is no less than a minsup threshold
Customer
buys diaper
buys both
buys Tea

5. Basic Concepts: Association Rules
Tid
Items bought
Find all the rules X  Y with minimum support and confidence
support, s, probability that a transaction contains X  Y
confidence, c, conditional probability that a transaction having X also contains Y
Let minsup = 50%, minconf = 50%
Freq. Pat.: Tea:3, Nuts:3, Diaper:4, Eggs:3, {Tea, Diaper}:3 10
Tea, Nuts, Diaper 20
Tea, Coffee, Diaper 30
Tea, Diaper, Eggs 40
Nuts, Eggs, Milk 50
Nuts, Coffee, Diaper, Eggs, Milk
Customer
buys both
Customer
buys diaper
Customer
buys Tea

6. Rule Measures: Support & Confidence
Simple Formulas:
Confidence (AB) = #tuples containing both A & B / #tuples containing A = P(B|A) = P(A U B ) / P (A)
Support (AB) = #tuples containing both A & B/ total number of tuples = P(A U B)
What do they actually mean ?
Find all the rules X & Y  Z with minimum confidence and support
support, s, probability that a transaction contains {X, Y, Z}
confidence, c, conditional probability that a transaction having {X, Y} also contains Z

9. Support & Confidence : An Example
Let minimum support 50%, and minimum confidence 50%, then we have,
A  C (50%, 66.6%)
C  A (50%, 100%)

10. FP Growth (Han, Pei, Yin 2000)
One problematic aspect of the Apriori is the candidate generation
Source of exponential growth
Another approach is to use a divide and conquer strategy
Idea: Compress the database into a frequent pattern tree representing frequent items

11. FP Growth (Tree construction)
Initially, scan database for frequent 1-itemsets
Place resulting set in a list L in descending order by frequency (support)
Construct an FP-tree
Create a root node labeled null
Scan database
Process the items in each transaction in L order
From the root, add nodes in the order in which items appear in the transactions
Link nodes representing items along different branches

12. FP-Tree Algorithm -- Simple Case
Transaction_id
Time
Items_bought
101
6:35
Milk, bread, cookies, juice
792
7:38
Milk, juice
1130
8:05
Milk, eggs
1735
8:40
Bread, cookies, coffee

13. FP Growth – Another Example
Minimum support of 20% (frequency of 2)
Frequent 1-itemsets
I1,I2,I3,I4,I5
Construct list
L = {(I2,7),(I1,6),(I3,6),(I4,2),(I5,2)}
TID
Items 1
I1,I2,I5 2
I2,I4 3
I2,I3,I6 4
I1,I2,I4 5
I1,I3 6
I2,I3 7 8
I1,I2,I3,I5 9
I1,I2,I3

14. Build FP-Tree Create root node Scan database Transaction1: I1, I2, I5
null
Scan database
Transaction1: I1, I2, I5
Order: I2, I1, I5 1 I5 I4 I3 I1 I2
Maintain header table
(I2,1)
(I1,1)
(I5,1)
Process transaction
Add nodes in item order
Label with items, count
Header table has item, current count, and links to first node of a particular item type.

15. Build FP-Tree null (I2,2) (I4,1) (I1,1) (I5,1) TID Items 1 I1,I2,I5 23
I2,I3,I6 4
I1,I2,I4 5
I1,I3 6
I2,I3 7 8
I1,I2,I3,I5 9
I1,I2,I3
null
(I2,2)
(I1,1)
(I5,1) 1 I5 I4 I3 I1 2 I2
(I4,1)
Exercise: Build the rest of the tree.

16. Minining the FP-tree Start at the last item in the table
Find all paths containing item
Follow the node-links
Identify conditional patterns
Patterns in paths with required frequency
Build conditional FP-tree C
Append item to all paths in C, generating frequent patterns
Mine C recursively (appending item)
Remove item from table and tree

17. Mining the FP-Tree Prefix Paths (I2 I1,1) (I2 I1 I3, 1)
Conditional Path
(I2 I1, 2)
Conditional FP-tree
(I2 I1 I5, 2)
null
(I2,7)
(I1,4)
(I5,1) 2 I5 I4 6 I3 I1 7 I2
(I4,1)
(I3,2)
(I1,2)
null
Exercise: What other frequent patterns come from the CP tree?
Exercise: Mine all frequent patterns.
(I2,2)
(I1,2)

18. FP-Tree Example Continued
Item
Conditional pattern base
Conditional
FP-Tree
Frequent pattern generated I5
{(I2 I1: 1),(I2 I1 I3: 1)}
<I2:2 , I1:2>
I2 I5:2, I1 I5:2, I2 I1 I5: 2 I4
{(I2 I1: 1),(I2: 1)}
<I2: 2>
I2 I4: 2 I3
{(I2 I1: 2),(I2: 2), (I1: 2)}
<I2: 4, I1: 2>,<I1:2>
I2 I3:4, I1 I3: 4 , I2 I1 I3: 2 I1
{(I2: 4)}
<I2: 4>
I2 I1: 4

19. Frequent 1-itemsets Minimum support of 20% (frequency of 2)
I1,I2,I3,I4,I5
Construct list
L = {(I2,7),(I1,6),(I3,6),(I4,2),(I5,2)}
TID
Items 1
I1,I2,I5 2
I2,I4 3
I2,I3,I6 4
I1,I2,I4 5
I1,I3 6
I2,I3 7 8
I1,I2,I3,I5 9
I1,I2,I3

20. Build FP-Tree Create root node Scan database Transaction1: I1, I2, I5
null
Scan database
Transaction1: I1, I2, I5
Order: I2, I1, I5 1 I5 I4 I3 I1 I2
Maintain header table
(I2,1)
Process transaction
Add nodes in item order
Label with items, count
(I1,1)
(I5,1)
Header table has item, current count, and links to first node of a particular item type.

21. Build FP-Tree null (I2,2) (I4,1) (I1,1) (I5,1) TID Items 1 I1,I2,I5 23
I2,I3,I6 4
I1,I2,I4 5
I1,I3 6
I2,I3 7 8
I1,I2,I3,I5 9
I1,I2,I3
null 1 I5 I4 I3 I1 2 I2
(I2,2)
(I1,1)
(I4,1)
(I5,1)
Exercise: Build the rest of the tree.

22. Mining the FP-tree Start at the last item in the table
Find all paths containing item
Follow the node-links
Identify conditional patterns
Patterns in paths with required frequency
Build conditional FP-tree C
Append item to all paths in C, generating frequent patterns
Mine C recursively (appending item)
Remove item from table and tree

23. Mining the FP-Tree Prefix Paths (I2 I1,1) (I2 I1 I3, 1)
Conditional Path
(I2 I1, 2)
Conditional FP-tree
(I2 I1 I5, 2)
null 2 I5 I4 6 I3 I1 7 I2
(I1,2)
(I2,7)
(I3,2)
null
(I4,1)
(I1,4)
(I5,1)
(I3,2)
Exercise: What other frequent patterns come from the CP tree?
Exercise: Mine all frequent patterns.
(I4,1)
(I2,2)
(I3,2)
(I1,2)
(I5,1)

24. FP-tree construction null After reading TID=1: B:1 A:1
Minimum support of 20% (frequency of 2)

25. FP-Tree Construction B 8 A 7 C D 5 E 3 Transaction Database B:8 A:5
null
C:3
D:1
A:2
C:1
E:1
Header table B 8 A 7 C D 5 E 3
Chain pointers help in quickly finding all the paths of the tree containing some given item.
There is a pointer chain for each item.
I have shown pointer chains only for E and D.

26. Transactions containing E
B:8
A:5
null
C:3
D:1
A:2
C:1
E:1
B:1
null
C:1
A:2
D:1
E:1

27. Suffix E A 2 C D B:1 null C:1 A:2 D:1 E:1 (New) Header table
Conditional FP-Tree for suffix E A 2 C D
null
B doesn’t survive because it has support 1, which is lower than min support of 2.
C:1
A:2
C:1
D:1
The set of paths ending in E.
Insert each path (after truncating E) into a new tree.
D:1
We continue recursively.
Base of recursion: When the tree has a single path only.
FI: E

28. Suffix DE A 2 (New) Header table null
The conditional FP-Tree for suffix DE A 2
A:2
C:1
D:1
null
D:1
A:2
The set of paths, from the E-conditional FP-Tree, ending in D.
Insert each path (after truncating D) into a new tree.
We have reached the base of recursion.
FI: DE, ADE

29. Suffix CE (New) Header table null
The conditional FP-Tree for suffix CE
C:1
A:1
C:1
null
D:1
The set of paths, from the E-conditional FP-Tree, ending in C.
Insert each path (after truncating C) into a new tree.
We have reached the base of recursion.
FI: CE

30. Suffix AE (New) Header table The conditional FP-Tree for suffix AE
null
null
A:2
The set of paths, from the E-conditional FP-Tree, ending in A.
Insert each path (after truncating A) into a new tree.
We have reached the base of recursion.
FI: AE

31. Suffix D A 4 B 3 C B:3 A:2 null C:1 D:1 (New) Header table
Conditional FP-Tree for suffix D A 4 B 3 C
null
A:4
B:1
B:2
C:1
C:1
The set of paths containing D.
Insert each path (after truncating D) into a new tree.
C:1
We continue recursively.
Base of recursion: When the tree has a single path only.
FI: D

32. Suffix CD A 2 B (New) Header table null
Conditional FP-Tree for suffix CD A 2 B
A:4
B:1
B:2
C:1
C:1
null
C:1
A:2
B:1
B:1
The set of paths, from
the D-conditional FP-Tree, ending in C.
Insert each path (after truncating C) into a new tree.
We continue recursively.
Base of recursion: When the tree has a single path only.
FI: CD

33. Suffix BCD (New) Header table Conditional FP-Tree for suffix CDB null
The set of paths from
the CD-conditional FP-Tree, ending in B.
Insert each path (after truncating B) into a new tree.
We have reached the base of recursion.
FI: CDB

34. Suffix ACD (New) Header table Conditional FP-Tree for suffix CDA null
The set of paths from
the CD-conditional FP-Tree, ending in A.
Insert each path (after truncating B) into a new tree.
We have reached the base of recursion.
FI: CDA

35. Suffix C B 6 A 4 null (New) Header table
Conditional FP-Tree for suffix C B 6 A 4
B:6
A:1
A:3
C:3
C:1
null
C:3
B:6
A:1
A:3
The set of paths ending in C.
Insert each path (after truncating C) into a new tree.
We continue recursively.
Base of recursion: When the tree has a single path only.
FI: C

36. Suffix AC B 3 (New) Header table null
Conditional FP-Tree for suffix AC B 3
B:6
A:1
A:3
null
B:3
The set of paths from
the C-conditional FP-Tree, ending in A.
Insert each path (after truncating A) into a new tree.
We have reached the base of recursion.
FI: AC, BAC

37. Suffix BC B 6 (New) Header table null
Conditional FP-Tree for suffix BC B 6
B:6
null
The set of paths from
the C-conditional FP-Tree, ending in B.
Insert each path (after truncating B) into a new tree.
We have reached the base of recursion.
FI: BC

38. Suffix A B 5 null (New) Header table Conditional FP-Tree for suffix A
The set of paths ending in A.
Insert each path (after truncating A) into a new tree.
We have reached the base of recursion.
FI: A, BA

39. Suffix B (New) Header table Conditional FP-Tree for suffix B null null
The set of paths ending in B.
Insert each path (after truncating B) into a new tree.
We have reached the base of recursion.
FI: B

40. Apriori Advantages and disadvantages
Generate all candidates item set and test them with minimum support are expensive in both space and time.
Apriori is so slow comparing with FP growth
Apriori is easy to implement

41. FP Growth Advantages and disadvantages
No candidate generation, no candidate test
No repeated scan of entire database
Basic ops: counting local frequent items and building sub FP-tree, no pattern search and matching
Faster than Apriori
FP tree is expensive to build
FP tree not fit in memory

42. ECLAT Algorithm by Example
Equivalence CLASS Transformation
Transform the horizontally formatted data to the vertical format by scanning the database once
The support count of an itemset is simply the length of the TID_set of the itemset
TID
List of item IDS
T100
I1,I2,I5
T200
I2,I4
T300
I2,I3
T400
I1,I2,I4
T500
I1,I3
T600
T700
T800
I1,I2,I3,I5
T900
I1,I2,I3
itemset
TID_set I1
{T100,T400,T500,T700,T800,T900} I2
{T100,T200,T300,T400,T600,T800,T900} I3
{T300,T500,T600,T700,T800,T900} I4
{T200,T400} I5
{T100,T800}

43. ECLAT Algorithm by Example
Frequent 1-itemsets in vertical format
itemset
TID_set I1
{T100,T400,T500,T700,T800,T900} I2
{T100,T200,T300,T400,T600,T800,T900} I3
{T300,T500,T600,T700,T800,T900} I4
{T200,T400} I5
{T100,T800}
min_sup=2
The frequent k-itemsets can be used to construct the candidate (k+1)-itemsets based on the Apriori property
Frequent 2-itemsets in vertical format
itemset
TID_set
{I1,I2}
{T100,T400,T800,T900}
{I1,I3}
{T500,T700,T800,T900}
{I1,I4}
{T400}
{I1,I5}
{T100,T800}
{I2,I3}
{T300,T600,T800,T900}
{I2,I4}
{T200,T400}
{I2,I5}
{I3,I5}
{T800}

44. ECLAT Algorithm by Example
Frequent 3-itemsets in vertical format
itemset
TID_set
{I1,I2,I3}
{T800,T900}
{I1,I2,I5}
{T100,T800}
min_sup=2
This process repeats, with k incremented by 1 each time, until no frequent items or no candidate itemsets can be found
Properties of mining with vertical data format
Take the advantage of the Apriori property in the generation of candidate (k+1)-itemset from k-itemsets
No need to scan the database to find the support of (k+1) itemsets, for k>=1
The TID_set of each k-itemset carries the complete information required for counting such support
The TID-sets can be quite long, hence expensive to manipulate
Use diffset technique to optimize the support count computation