E FFICIENT T OP - K Q UERY E VALUATION ON P ROBABILISTIC D ATA P APER B Y C HRISTOPHER R´ E N ILESH D ALVI D AN S UCIU Presented By Chandrashekar Vijayarenu Anirban Maiti

A GENDA Overview DNF Evaluation Query Evaluation Problem definition Monte Carlo (MC) Simulation Multi simulation(MS) Experiments

P ROBABILISTIC D ATABASES Probabilistic database is an uncertain database in which the possible worlds have associated probabilities. The simplistic definition is that every tuple belongs to the database with some probability (between 0 - 1) Example:

S ELECT Q UERY ON P ROBABILISTIC D ATABASE Find directors with a highly rated Drama and low rated comedy SELECT DISTINCT d.dirName AS Director FROM AMZNReviews a, AMZNReviews b, TitleMatch ax, TitleMatch by, IMDBMovie x, IMDBMovie y, IMDBDirector d WHERE a.asin=ax.asin and b.asin=by.asin and ax.mid=x.mid and by.mid=y.mid and x.did=y.did and y.did=d.did and x.genre=’comedy’ and y.genre=’drama’ and abs(x.year - y.year) <= 5 and a.rating 4 Major challenge in probabilistic database is query evaluation. Dalvi and Suciu have shown that most SQL queries are #P Complete. This paper propose a new approach to query evaluation on probabilistic databases, by combining top-k style queries. User specifies a SQL Query and a number k, and the system returns the highest ranked k answers.

Q UERY P ROCESSING C HALLENGES Compute exact output probabilities is computationally hard. Meaning, any algorithm computing the probabilities need to iterate through all possible subsets of TitleMatch. Potential answers for which we need to calculate the probability is large. User is likely to end up inspecting just the first few of them. This paper introduces the multisimulation algorithm which enables effective processing of probabilistic queries with some error guarantees.

O VERVIEW

P OSSIBLE W ORLDS A possible world is thus any subset of the tuples in the database and its probability can be computed as a product of the probabilities of the tuples in it, and the respective probabilities of the tuples that are not in that world. Consider the following probabilistic database containing two relations S and T: A probabilistic database over schema S is a pair (W,P) where W = {W1,...,Wn} is a set of database instances over S, and P : W ->[0, 1] is a probability distribution (i.e. P j=1,n P(Wj) = 1). Each instance Wj for which P(Wj) > 0 is called a possible world. Let Jp be a database instance over schema Sp. Then Mod(Jp) is the probabilistic database (W,P) over the schema S obtained as described below

DNF F ORMULA In boolean logic, a disjunctive normal form (DNF) is a standardization of a logical formula which is a disjunction of conjunctive clauses. In our Example E = (t 1 Λ t 5 ) V t 2 = true in the possible worlds W 3,W 7,W 10,W 11, and its probability is thus P(E) = P(W 3 ) + P(W 7 ) + P(W 10 ) + P(W 11 ).

Q UERY E VALUATION  DNF E VALUATION qe = SELECT * FROM AMZNReviews a, AMZNReviews b, TitleMatch ax, TitleMatch by, IMDBMovie x, IMDBMovie y, IMDBDirector d WHERE... Each answer returned by qe will have 7 tuple variables defined in where clause: ( a, b, ax p, by p, x, y, d ) ax p and by p are probabilistic tuples. (From TupleMatch table) Thus, every row returned by qe defines a boolean expression t.E = ax p Λ by p.

I NTRODUCE G ROUP B Y Next we group the rows by their directors, and for each group G = {(ax p 1, by p 1 ),..., (ax p m, by p m )} DNF formula: G.E = (ax p 1 Λ by p 1 ) V... V (ax p m Λ bx p m ) The director’s probability give by P(G.E). How to calculate the director’s probability? Brute Force Approach: Choose every possible world and calculate the truth value of the boolean expression. p = P(G.E) is the frequency with which G.E = true #P Hard problem Monte Carlo Simulation : Choose the possible world at random and calculate the truth value.

P ROBLEM D EFINITION G = {G1,...,Gn} of n objects, with unknown Probabilities p 1,..., p n, and a number k <= n. Our goal is to find a set of k objects with the highest probabilities, denoted Top-K Subset of G. Solution: The idea in our algorithm is to run in parallel several Monte-Carlo simulations, one for each candidate answer, and approximate each probability only to the extent needed to compute correctly the top-k answers

M ONTE C ARLO (MC) S IMULATION An MC algorithm repeatedly chooses at random a possible world, and computes the truth value of the Boolean expression G.E (Eq.(3)); the probability p = P(G.E) is approximated by the frequency ˜p with which G.E was true. Luby and Karp have described the variant shown in Algorithm fix an order on the disjuncts: t1, t2,..., tm C := 0 repeat Choose a random disjunct ti Є G Choose a random truth assignment s.t. ti.E = true if forall j < i tj.E = false then C := C + 1 until N times return ˜p = C/N

M ULTISIMULATION (MS) Assumptions: Intervals:

M ULTISIMULATION (MS) Critical Region: The critical region, top objects, and bottom objects are: (c, d) = (topk(a1,..., an), topk+1(b1,..., bn))……Eq (5) T = {Gi | d <= ai} B = {Gi | bi <= c} Algorithm: MS TopK(G, k) : /* G = {G1,...,Gn} */ Let [a1, b1] =... = [an, bn] = [0, 1], (c, d) = (0, 1) while c <= d do Case 1: choose a double crosser to simulate Case 2: choose upper and lower crosser to simulate Case 3: choose a maximal crosser to simulate Update (c, d) using Eq.(5) end while return TopK = T = {Gi | d <= ai}

E XAMPLE : L ET US SELECT T OP G1 G2 G3 G4 G5 cd

E XPERIMENTS

E XPERIMENTS RESULTS

T HANK YOU