Mining Association Rules of Simple Conjunctive Queries Bart Goethals Wim Le Page Heikki Mannila SIAM /8/261

Outline. Motivation Preliminaries Conqueror : algorithm ◦ Selection loop ◦ Projection loop ◦ Constants loop ◦ Eliminating redundancies Experiments Conclusion 2015/8/262

Motivation. First query ask for the actors have starred in a movie of the genre ‘drama’. Second query ask for ‘drama’ and ‘comedy’. Now suppose the answer to the first query consists of 1000 actors, and the answer to the second query consists of 900 actors. 2015/8/263

Motivation (cont) It reveals the potentially interesting pattern that actors starring in ‘drama’ movies typically (with a probability of 90%) also star in a ‘comedy’ movie. In general, we are looking for pairs of queries Q 1,Q 2, such that Q 1 asks for a set of tuples satisfying a certain condition and Q2 asks for those tuples satisfying a more specific condition. 2015/8/264

Preliminaries. Relational database: R(R 1,…,R n ) Definition 1: ◦ Simple Conjunctive Query π X σ F (R 1 ×···× R n ) ◦ F : R i.A = R j.B or R k.A=“c” ◦ x : attributes from R example : Q1: π A,B R or Q2 : π A,B σ A=B R 2015/8/265

Preliminaries (cont) Definition 2 : Containment ◦ Two conjunctive queries Q 1 and Q 2 over R we write Q 1 ⊆ Q 2 if for every possible instance I of R, Q 1 (I) ⊆ Q 2 (I) Definition 3 : Diagonally contained ◦ Q 1 is diagonally contained in Q 2 if Q 1 is contained in a projection of Q 2 (Q 1 ⊆ π X Q 2 ) write Q 1 ⊆ Δ Q /8/266

Preliminaries (cont) Definition 4 : Association Rule ◦ An association rule is of the form Q 1 ⇒ Q 2, such that Q 1 and Q 2 are both simple conjunctive queries and Q 2 ⊆ Δ Q /8/267

Preliminaries (cont) Definition 5 : Support ◦ The support of a conjunctive query Q in an instance I is the number of distinct tuples in the answer of Q on I. ◦ A query is said to be frequent in I if its support exceeds a given minimal support threshold. ◦ The support of an association rule Q 1 ⇒ Q 2 in I is the support of Q 2 in I, an association rule is called frequent in I if Q 2 is frequent in I. 2015/8/268

Preliminaries (cont) Definition 6 : Confidence ◦ An association rule Q 1 ⇒ Q 2 is said to be confident if the support of Q 2 divided by the support of Q 1 exceeds a given minimal confidence threshold. 2015/8/269

Conqueror : Algorithm Divided into two phases. ◦ In a first phase, all frequent simple conjunctive queries are generated. ◦ Then, in a second phase, all confident association rules over these frequent queries are generated. 2015/8/2610

Algorithm (cont) Property 1 : ◦ Let Q 1 and Q 2 be two simple conjunctive queries. If Q 2 ⊆ Δ Q 1, then support(Q 1 ) ≥ support(Q 2 ). 2015/8/2611

Algorithm (cont) Selection loop: ◦ Generate all instantiations of F, without constants, in a breadth-first manner. Projection loop: ◦ For each generated selection, generate all instantiations of X in a breadth-first manner, and test their frequency. Constants loop: ◦ For each generated query in the projection loop, add constant assignments to F in a breadth-first manner. 2015/8/2612

Algorithm (cont) Selection loop ◦ We will use the so called restricted growth string for generating all partitions. ◦ A Restricted Growth string is an array a[1...m] where m is the total number of attributes occurring in the database. ◦ Restricted growth string satisfies the following growth inequality (for i =1, 2,...,n − 1, and with a[1] = 1): a[i +1] ≤ 1+max a[1],a[2],...,a[i]. 2015/8/2613

Algorithm (cont) a[1] = 1 i=1, a[1+1] ≤ 1 + max{a[1]} = 2 i=2, a[2+1] ≤ 1 + max{a[1], a[2]} = 3. EXAMPLE 4. ◦ Let A 1,A 2,A 3,A 4 be the set of all attributes occurring in the database. Then, the restricted growth string 1221 represents the conjunction of equalities A 1 = A 4, A 2 = A /8/2614

Algorithm (cont) 2015/8/2615

Algorithm (cont) Before generating possible projections for a given selection, we first determine whether the selection represents a cartesian product. 2015/8/2616

Algorithm (cont) What is cartesian product ◦ To determine whether a selection represents a cartesian product, we interpret each simple conjunctive query as an undirected graph, such that each relation or constant is a node, and each equality in the selection of the query is an edge between the nodes occurring in that equality. 2015/8/2617

Algorithm (cont) Projection loop ◦ For every generated projection, we first check whether all more general queries are known to be frequent, and if so, the resulting query is evaluated against the database 2015/8/2618

Algorithm (cont) Constant loop ◦ Every block of attribute equalities of the selection can also be set equal to a constant. 2015/8/2619

Algorithm (cont) Candidate evaluation ◦ Evaluated against the database by translating each query to SQL. ◦ The result of such a query is then stored in a temporary table ( τ ). SELECT A, COUNT(*) AS sup FROM τ GROUP BY A ◦ The result of these queries is stored in a new temporary table ( τ A ) holding the constant values together with their support. 2015/8/2620

Algorithm (cont) Let τ A and τ B be the temporary tables holding the constant values for the attributes A and B together with their support. We can now generate the table τ A,B. This is the generated query for getting the values for τ A,B,C using the temporary tables τ, τ A,B, τ A,C, τ B,C. 2015/8/2621

Algorithm (cont) Association rule generation ◦ For all queries Q1 the algorithm finds all queries Q2 such that Q2 ⊆ Δ Q1, it computes the confidence of the rule Q1 ⇒ Q2 and tests whether it is confident. 2015/8/2622

Algorithm (cont) Eliminating redundancies ◦ Consider the following association rules, each based on a vertical containment:  π R.A,R.B,S.E σ R.C=S.F(R × S) ⇒ π R.A,S.E σ R.C=S.F(R × S)  π R.A,S.E σ R.C=S.F(R × S) ⇒ π R.A σ R.C=S.F(R × S)  π R.A,R.B,S.E σ R.C=S.F(R × S) ⇒ π R.A σ R.C=S.F(R × S) ◦ Now suppose the first association rule has a confidence of 100%. Then, the confidence of the second and third association rule must be equal. 2015/8/2623

Algorithm (cont) LEMMA. An association rule Q1 ⇒ Q2 is redundant if ◦ 1. There exists an association rule Q 3 ⇒ Q 1 with confidence 100% ◦ 2. There exists an association rule Q 4 ⇒ Q 2 with confidence 100%, and Q 4 ⊆ Δ Q /8/2624

Experiments. The IMDB snapshot consist of three tables ACTORS (A), MOVIES (M) and GENRES (G),and two tables that represent the connections between them namely ACTORMOVIES (AM) and GENREMOVIES (GM). We can conclude that every movie has a genre because of the following association rule with 100% Confidence π M.MID (M) ⇒ π M.MID σ GM.MID=M.MID (M × GM) 2015/8/2625

Experiments (cont) In our database, not every movie has to have an actor associated with it as the following rule only has 75.91% confidence. π M.MID (M) ⇒ π M.MID σ AM.MID=M.MID (M × AM) We can find ‘frequent’ genres in which actors play. It has 40.44% confidence, so 40.44% of the actors play in a ‘Documentary’ (genre id 3) while the same rule for ‘Drama’ has 49.85% confidence. 2015/8/2626

Experiments (cont) 81.60% of the actors in genre ‘Music’ (genre id 16) only play in one movie. But the same rule for genre ‘Crime’ has only 49.87% confidence. 2015/8/2627

Conclusion. 2015/8/2628