1. Frequent Structure Mining Prajwal Shrestha Department of Computer Science The University of Vermont Spring 2015

2. Original Authors This presentation is based on the paper Zaki MJ (2002). Efficiently mining frequent trees in a forest. Proceedings of the 8th ACM SIGKDD International Conference. The author’s original presentation was used to make this one. I further adapted this from Ahmed R. Nabhan’s and Robert Howe modifications. 2

3. Graph Review #Review graph in python From igraph import * g=Graph() g.add_vertices(“A”,”B”,”C”,”D”) g.add_edges((“A”,”B”),(“B”,C”),(“B”,”D)) g.bfs(“A”) g.pagerank() g.subgraph(“A”,”B”,”C”) #How to generate all subgraphs?

4. Outline Graph Mining Overview Mining Complex Structures - Introduction Motivation and Contributions of author Problem Definition and Case Examples Main Ingredients for Efficient Pattern Extraction Experimental Results Conclusions 4

5. Outline Graph Mining Overview Mining Complex Structures - Introduction Motivation and Contributions of author Problem Definition and Case Examples Main Ingredients for Efficient Pattern Extraction Experimental Results Conclusions 5

6. Why Graph Mining? Graphs are convenient structures that can represent many complex relationships. We are drowning in graph data: Social Networks Biological Networks World Wide Web 6

7. 7 High School UVM BU Facebook Data (Source: Wolfram|Alpha Facebook Report)

8. 8 Facebook Data (Source: Wolfram|Alpha Facebook Report)

9. 9 Biological Data (Source: KEGG (Kyoto Encyclopedia of Genes and Genomes) Pathways Database)

10. Some Graph Mining Problems Pattern Discovery Graph Clustering Graph Classification and Label Propagation Structure and Dynamics of Evolving Graphs 10

11. Graph Mining Framework 11 Mining graph patterns is a fundamental problem in data mining. Graph Data Mine Exponential Pattern Space Select Relevant Patterns Exploratory Task Clustering Classification Structure Indices

12. Basic Concepts Graph – A graph G is a 3-tuple G = (V, E, L) where V is the finite set of nodes. E ⊆ V × V is the set of edges L is a labeling function for edges and nodes. Subgraph – A graph G 1 = (V 1, E 1, L 1 ) is a subgraph of G 2 = (V 2, E 2, L 2 ) iff: V 1 ⊆ V 2 E 1 ⊆ E 2 L 1 (v) = L 2 (v) for all v ∈ V 1. L 1 (e) = L 2 (e) for all e ∈ E 1. 12 A CD BA C B

13. Basic Concepts Graph Isomorphism – “A bijection between the vertex sets of G 1 and G 2 such that any two vertices u and v which are adjacent in G 1 are also adjacent in G 2.” (Wikipedia) 13 A CD B E 1 4 3 2 5 Subgraph Isomorphism is even harder (NP- Complete!)

14. Basic Concepts Graph Isomorphism – Let G 1 = (V 1, E 1, L 1 ) and G 2 = (V 2, E 2, L 2 ). A graph isomorphism is a bijective function f: V 1 → V 2 satisfying L 1 (u) = L 1 ( f (u)) for all u ∈ V 1. For each edge e 1 = (u,v) ∈ E 1, there exists e 2 = ( f(u), f(v)) ∈ E 2 such that L 1 (e 1 ) = L 2 (e 2 ). For each edge e 2 = (u,v) ∈ E 2, there exists e 1 = ( f –1 (u), f –1 (v)) ∈ E 1 such that L 1 (e 1 ) = L 2 (e 2 ). 14

15. Discovering Subgraphs TreeMiner and gSpan both employ subgraph or substructure pattern mining. Graph or subgraph isomorphism can be used as an equivalence relation between two structures. There is an exponential number of subgraph patterns inside a larger graph (as there are 2 n node subsets in each graph and then there are edges.) Finding frequent subgraphs (or subtrees) tends to be useful in data mining. 15

16. Outline Graph Mining Overview Mining Complex Structures - Introduction Motivation and Contributions of author Problem Definition and Case Examples Main Ingredients for Efficient Pattern Extraction Experimental Results Conclusions 16

17. Mining Complex Structures Frequent structure mining tasks Item sets – Transactional, unordered data. Sequences – Temporal/positional, text, biological sequences. Tree Patterns – Semi-structured data, web mining, bioinformatics, etc. Graph Patterns – Bioinformatics, Web Data “Frequent” is a broad term Maximal or closed patterns in dense data Correlation and other statistical metrics Interesting, rare, non-redundant patterns. 17

18. Anti-Monotonicity 18 (Source: SIGMOD ’08) A monotonic function is a consistently increasing or decreasing function*. The author refers to a monotonically decreasing function as anti-monotonic. The frequency of a super- graph cannot be greater than the frequency of a subgraph (similar to Apriori). * Very Informal Definition The black line is always decreasing

19. Outline Graph Mining Overview Mining Complex Structures - Introduction Motivation and Contributions of author Problem Definition and Case Examples Main Ingredients for Efficient Pattern Extraction Experimental Results Conclusions 19

20. Tree Mining – Motivation Capture intricate (subspace) patterns Can be used (as features) to build global models (classification, clustering, etc.) Ideally suited for categorical, high- dimensional, complex, and massive data. Interesting Applications Semi-structured Data – Mine structure and content Web usage mining – Log mining (user sessions as trees) Bioinformatics – RNA secondary structures, Phylogenetic trees 20 (Source: University of Washington)

21. Classification Example Subgraph patterns can be used as features for classification. 21 # of sides 2345678 Amount0100100 Off-the-shelf classifiers (like neural networks or genetic algorithms) can be trained using these vectors. Feature selection is very useful too. “Hexagons are a commonly occurring subgraph in organic compounds.”

22. Contributions Systematic subtree enumeration. Extensions for mining unlabeled or unordered subtrees or sub-forests. Optimizations Representing trees as strings. Scope-lists for subtree occurrences. 22

23. Outline Graph Mining Overview Mining Complex Structures - Introduction Motivation and Contributions of author Problem Definition and Case Examples Main Ingredients for Efficient Pattern Extraction Experimental Results Conclusions 23

24. How does searching for patterns work? Start with graphs with small sizes. Extend k -size graphs by one node to generate k + 1 candidate patterns. Use a scoring function to evaluate each candidate. A popular scoring function is one that defines the minimum support. Only graphs with frequency greater than minisup are kept. 24

25. How does searching for patterns work? “The generation of size k + 1 subgraph candidates from size k frequent subgraphs is more complicated and more costly than that of itemsets” – Yan and Han (2002), on gSpan Where do we add a new edge? It is possible to add a new edge to a pattern and then find that doesn’t exist in the database. The main story of this presentation is on good candidate generation strategies. 25

26. TreeMiner - I TreeMiner uses a technique for numbering tree nodes based on DFS. This numbering is used to encode trees as vectors. Subtrees sharing a common prefix (e.g. the first k numbers in vectors) form an equivalence class. Generate new candidate (k + 1) -subtrees from equivalence classes of k -subtrees (e.g. Apriori) 26

27. TreeMiner - II This is important because candidate subtrees are generated only once! (Remember the subgraph isomorphism problem that makes it likely to generate the same pattern over and over) 27

28. Definitions Tree – An undirected graph where there is exactly one path between any two vertices. Rooted Tree – Tree with a special node called root. 28 This tree has no root node. It is an unrooted tree. This tree has a root node. It is a rooted tree.

29. Definitions Ordered Tree – The ordering of a node’s children matters. Example: XML Documents Exercise – Prove that ordered trees must be rooted. 29 ≠ v3v3 v2v2 v1v1 v3v3 v1v1 v2v2

30. Definitions Labeled Tree – Nodes have labels. Rooted trees also have some special terminology. Parent – The node one closer to the root. Ancestor – The node n edges closer to the root, for any n. Siblings – Two nodes with the same parent. 30 parent sibling ancestor ancestor(X,Y) :- parent(X,Y). ancestor(X,Y) :- parent(Z,Y), ancestor(X,Z). sibling(X,Y) :- parent(Z,X), parent(Z,Y). embedded sibling

31. Definitions Embedded Siblings – Two nodes sharing a common ancestor. Numbering – The node’s position in a traversal (normally DFS) of the tree. A node has a number n i and a label L(n i ). Scope – The scope of a node n l is [l, r], where n r is the rightmost leaf under n l (again, DFS numbering). 31

32. Definitions Embedded Subtrees – S = (N s, B s ) is an embedded subtree of T = (N, B) if and only if the following conditions are met: N s ⊆ N (the nodes have to be a subset). b = (n x, n y ) ∊ B s iff n x is an ancestor of n y. For each subset of nodes N s there is one embedded subtree or subforest. 32 v3v3 v2v2 v1v1 v0v0 v6v6 v7v7 v8v8 v5v5 v4v4 v1v1 v5v5 v4v4 (Colors are only on this graph to show corresponding nodes) subtree

33. Definitions Match Label – The node numbers (DFS numbers) in T of the nodes in S with matching labels. A match label uniquely identifies a subtree. This is useful because a labeling function must be surjective but will not necessarily be bijective. {v 1, v 4, v 5 } or {1, 4, 5} 33 (Colors are only on this graph to show corresponding nodes) v3v3 v2v2 v1v1 v0v0 v6v6 v7v7 v8v8 v5v5 v4v4 v1v1 v5v5 v4v4 subtree

34. Definitions Subforest – A disconnected pattern generated in the same way as an embedded subtree. 34 v3v3 v2v2 v1v1 v0v0 v6v6 v7v7 v8v8 v5v5 v4v4 v1v1 v4v4 (Colors are only on this graph to show corresponding nodes) subforest v7v7 v8v8

35. Problem Definition Given a database (forest) D of trees, find all frequent embedded subtrees. Frequent – Occurring a minimum number of times (use user-defined minisup). Support( S ) – The number of trees in D that contain at least one occurrence of S. Weighted-Support( S ) – The number of occurrences of S across all trees in D. 35

36. Exercise Generate an embedded subtree or subforest for the set of nodes N s = {v 1, v 2, v 5 }. Is this an embedded subtree or subforest, and why? Assume a labeling function L(x) = x. 36 v3v3 v2v2 v1v1 v0v0 v6v6 v7v7 v8v8 v5v5 v4v4 v1v1 v5v5 v2v2 This is an embedded subtree because all of the nodes are connected.

37. Outline Graph Mining Overview Mining Complex Structures - Introduction Motivation and Contributions of author Problem Definition and Case Examples Main Ingredients for Efficient Pattern Extraction Experimental Results Conclusions 37

38. Main Ingredients Pattern Representation Trees as strings Candidate Generation No duplicates. Pattern Counting Scope-based List (TreeMiner) Pattern-based Matching (PatternMatcher) 38

39. String Representation With N nodes, M branches, and a max fanout of F : An adjacency matrix takes (N)(F + 1) space. An adjacency list requires 4N – 2 space. A tree of (node, child, sibling) requires 3N space. String representation requires 2N – 1 space. 39

40. String Representation String representation is labels with a backtrack operator, –1. 40 0131–12 2 2 0 1 3 12 2 2

41. Exercise Based on given tree T, calculate support, weighted support, match labels and string for each S1, S2 and S3 subtrees 41

42. Candidate Generation Equivalence Classes – Two subtrees are in the same equivalence class iff they share a common prefix string P up to the (k – 1) -th node. This gives us simple equivalence testing of a fixed-size array. Fast and parallel – Can be run on a GPU. Caveat – The order of the tree matters. 42

43. Candidate Generation Generate new candidate (k + 1) -subtrees from equivalence classes of k -subtrees. Consider each pair of elements in a class, including self- extensions. Up to two new candidates for each pair of joined elements. All possible candidate subtrees are enumerated. Each subtree is generated only once! 43

44. Candidate Generation Each class is represented in memory by a prefix string and a set of ordered pairs indicating nodes that exist in that class. A class is extended by applying a join operator ⊗ on all ordered pairs in the class. 44

45. Candidate Generation 45 3 1 2 4 1 2 Equivalence Class Prefix String 12 This generates two elements: (3, v 1 ) and (4, v 0 ) The element notation can be confusing because the first item is a label and the second item is a DFS node number.

46. Candidate Generation Theorem 1. Define a join operator ⊗ on two elements as (x, i) ⊗ (y, j). Then apply one of the following cases: (1)If i = j and P is not empty, add (y, j) and (y, j + 1) to class [P x ]. If P is empty, only add (y, j + 1) to [P x ]. (2)If i > j, add (y, j) to [P x ]. (3)If i < j, no new candidate is possible. 46

47. Candidate Generation Consider the prefix class from the previous example: P = (1, 2) which contains two elements, (3, v 1 ) and (4, v 0 ). 1.Join (3, v 1 ) ⊗ (3, v 1 ) – Case (1) applies, producing (3, v 1 ) and (3, v 2 ) for the new class P 3 = (1, 2, 3). 2.Join (3, v 1 ) ⊗ (4, v 0 ) – Case (2) applies. (Don’t worry, there’s an illustration on the next slide.) 47

48. Candidate Generation 48 3 1 2 3 1 2 ⊗ A class with prefix {1,2} contains a node with label 3. This is written as (3, v 1 ), meaning a node labeled ‘3’ is added at position 1 in DFS order of nodes. = 3 1 2 3 1 2 3 3 Prefix = (1, 2, 3) New nodes = (3, v 2 ), (3, v 1 )

49. Candidate Generation 49 3 1 2 4 1 2 ⊗ = Prefix = (1, 2, 3) New nodes = (3, v 2 ), (3, v 1 ), (4, v 0 ) 3 1 2 3 3 1 2 3 3 1 2 4

50. TreeMiner Algorithm DFS T REE M INER ( D, minisup ): F 1 = { frequent 1-subtrees} F 2 = { classes [P] 1 of frequent 2-subtrees } for all [P] 1 ∈ E do Enumerate-Frequent-Subtrees( [P] 1 ) E NUMERATE -F REQUENT -S UBTREES ( [P] ): for each element (x, i) ∈ [P] do [P x ] = ∅ for each element (y, j) ∈ [P] do R = { (x, i) ⊗ (y, j) } L(R) = { L(x) ∩ ⊗ L(y) } if for any R ∈ R, R is frequent, then [P x ] = [P x ] ∪ {R} Enumerate-Frequent-Subtrees( [P x ] ) 50

51. Scope-Lists

52. ScopeList Join 52 Recall that the scope is the interval between the lowest numbered child (or self) node and the highest numbered child node, using DFS numbering. This can be used to calculate support. v3v3 v2v2 v1v1 v0v0 v6v6 v7v7 v8v8 v5v5 v4v4 [0, 8] [1, 5] [2, 2] [3, 5] [4, 4][5, 5] [7, 8] [8, 8]

53. ScopeList Join 53 ScopeLists are used to calculate support. Let x and y be nodes with scopes s x = [l x, u x ], s y = [l y, u y ]. s x contains s y iff l x ≤ l y and u x ≥ u y. A scope list represents the entire forest.

54. ScopeList Join 54 A ScopeList is a list of (t, m, s) 3-tuples. t is the tree ID. m is the match label of the (k – 1) -length prefix of x k. s is the scope of the last item, x k. The use of scope lists allows constant time computations of whether y is a descendent or embedded sibling of x.

55. Scope-list Joins example

56. Outline Graph Mining Overview Mining Complex Structures - Introduction Motivation and Contributions of author Problem Definition and Case Examples Main Ingredients for Efficient Pattern Extraction Experimental Results Conclusions 56

57. Experimental Results Machine: 500Mhz PentiumII, 512MB memory, 9GB disk, RHEL 6.0 Synthetic Data: Web browsing Parameters: N = #Labels, M = #Nodes, F = Max Fanout, D = Max Depth, T = #Trees Create master website tree W For each node in W, generate #children (0 to F) Assign probabilities of following each child or to backtrack; adding up to 1 Recursively continue until D is reached Generate a database of T subtrees of W Start at root. Recursively at each node generate a random number (0 – 1) to decide which child to follow or to backtrack. Default parameters: N=100, M=10,000, D=10, F=10, T=100,000 Three Datasets: D10 (all default values), F5 (F=5), T1M (T=106) Real Data: CSLOGS – 1 month web log files at RPI CS 57 Over 13361 pages accessed (#labels) Obtained 59,691 user browsing trees (#number of trees) Average string length of 23.3 per tree

58. Distribution of Frequent Trees 58 SparseDense Take-Home Point: Many large, frequent trees can be discovered. F5: Max-Fanout = 5 T1M: 106 Trees

59. Experiments (Sparse) 59 SparseDense Take-Home Point: Both algorithms are able to cope with relatively short patterns in sparse data.

60. Experiments (Dense) 60 Sparse (Artificial Dataset) Dense (Real-World Dataset) Take-Home Point: Long patterns at low-support (length=20); the level-wise approach suffers. The authors use the artificial dataset to justify TreeMiner as 20 times faster than pattern matcher.

61. Outline Graph Mining Overview Mining Complex Structures - Introduction Motivation and Contributions of author Problem Definition and Case Examples Main Ingredients for Efficient Pattern Extraction Experimental Results Conclusions 61

62. Conclusions 62 T REE M INER : A novel tree mining approach. Non-duplicate candidate generation. Scope-List joins for frequency comparison. Framework for tree-mining tasks Frequent subtrees in a forest of rooted, labeled, ordered trees. Frequent subtrees in a single tree. There are extensions for unlabeled and unordered trees.

63. Caveats 63 Frequent does not always mean significant! Exhaustive enumeration is a problem even though the candidate generation in TreeMiner is efficient. Low min_sup values increases true positives at the cost of increasing false positives. State-of-the-art graph miners make use of the structure of the search space (e.g. the LEAP search algorithm) to extract only significant structures. Candidate structures can be generated by tree miners and evaluated by some other mean.

64. Question One Generate an embedded subtree or subforest for the set of nodes N s = {v 1, v 2, v 5 }. Is this an embedded subtree or subforest, and why? Assume a labeling function L(x) = x. 64 v3v3 v2v2 v1v1 v0v0 v6v6 v7v7 v8v8 v5v5 v4v4 v1v1 v5v5 v2v2 This is an embedded subtree because all of the nodes are connected.

65. Question Two Based on given tree T, calculate support, weighted support, match labels and string for each S1, S2 and S3 sub trees 65

66. Question Three How is a string representation of a tree useful in graph mining? What requirements does it place on the graph? 66 Answer. A string representation of a tree is useful because string comparisons are worst-case O(n) and can be easily optimized. However, it requires that a tree be rooted and ordered, because otherwise the string comparison operator would not be valid.