Queries with Difference on Probabilistic Databases Sanjeev Khanna Sudeepa Roy Val Tannen University of Pennsylvania

Probabilistic Databases To model and query uncertain data (sensor networks, information extraction…) Possible worlds model Each possible world W is a standard database instance, has a probability P[W] Compact representation D assuming independence S T R a1 a2 a3 b1 b2 b3 0.1 0.5 0.2 b1 b2 b3 0.7 0.8 0.4 a1 a2 a3 0.3 0.4 0.6 D

Query Semantics Query Semantics on probabilistic databases: Apply the query q on each possible world W Add up the probabilities of the worlds that give the same query answer A P[q(D) = A] = ∑W : q(W) = A P[W] Goal: Efficiently evaluate P[q(D) = A] Data complexity; want time polynomial in n = |D| Can we always efficiently compute P[q(D)]? NO, in general it is #P-hard

Query Answering in Two Steps Introduce event variables for tuples (P[w1] = 0.3, …) Step 1: Boolean provenance for q(D) [FR ’97, Z ’97] f = w1 v1 u1 + w2 v2 u1 + w3 v3 u2 + w3 v4 u3 Step 2: Compute P[q(D)] = P[f] given P[w1] = 0.3, P[v1] = 0.4, … easy hard Event variables D S R a1 a2 a3 b1 b2 b3 v1 v2 v3 v4 0.1 0.5 0.2 a1 a2 a3 b1 b2 b3 0.1 0.5 0.2 T Probability b1 b2 b3 u1 u2 u3 0.7 0.8 0.4 b1 b2 b3 0.7 0.8 0.4 a1 a2 a3 w1 w2 w3 0.3 0.4 0.6 a1 a2 a3 0.3 0.4 0.6 Boolean query q():-R(x),S(x, y),T(y)

Probability Computation for Positive Queries Dichotomy Result [DS ’04, ’07; DSS ’10] Given q as input, we can efficiently decide if q is Safe: Safe plans run in poly-time on all instances, or, Unsafe: #P-hard, e.g. q() :- R(x) S(x, y) T(y) Instance-by-instance approach [SDG ’10, RPT ’11] Both q and D are given as input Poly-time algorithm to compute P[q(D)] for special cases even if q is unsafe What about queries with difference?

Boolean Provenances for Difference T b1 b2 c1 c2 c3 u1 u2 u3 c1 c2 c3 a1 a2 a3 v1 v2 v3 v4 a1 a2 a3 w1 w2 w3 q1(x):- R(x, y), S(y, z) q2(x):- R(x, y), S(y, z), T(z) b1 b2 u1(v1 + v2) + u3v4 u2v3 b1 b2 u1v1w1 + u1v2w2 + u3v4w2 u2v3w3 q = q1 – q2 b1 b2 (u1(v1 + v2) + u3v4) . (u1v1w1 + u1v2w2 + u3v4w2) (u2v3) . (u2v3w3)

Previous Work on Difference FOR ’11 Framework for exact and approximate probability computation But, no guarantee of polynomial running time In fact, we show in this paper that with difference, in some cases no approximation exists (unless NP = RP) How far can we go with difference in poly-time?

A Quick Comparison Without difference With difference DNF of boolean provenance is poly-size (n|q|) P[q(D)] is always approximable (FPRAS) With difference DNF of boolean provenance may be exponential in n P[q(D)] may not be approximable FPRAS: Fully Polynomial Randomized Approx. Scheme Compute with prob. ≥ ¾ in time polynomial in n, 1/ε p  [(1-ε) P[q(D)], (1+ε) P[q(D)]

Our Results We study queries of the form q1 – q2 and their generalization FPRAS: If q1 is any UCQ, q2 is any safe CQ- #P-hardness: Even if both q1 and q2 are safe CQ- Inapproximability: Even if q1 is the trivial TRUE query and q2 is a UCQ Our FPRAS result extends to a larger class of queries of which q1 – q2 is a special case [CQ- : Conjunctive queries without self-joins]

Difference Rank Define difference rank (q) of query q recursively (q1 - q2) = (q1) + (q2) + 1 R – S : rank 1 (q1 ⋈ q2) = (q1) + (q2) (R – S1) ⋈ (R - S2) : rank 2 (R - T1) ⋈ T2 : rank 1 (q1  q2) = max ((q1), (q2)) (R – S1) ⋈ (R - S2)  (R - T1) ⋈ T2 : rank 2 Select, project: rank remains the same

FPRAS for queries q with (q) = 1 given some conditions hold (inapproximable for (q) = 1 in general)

Steps in FPRAS Step 1: Compute boolean provenance of q[D] for any query q with (q) = 1 Step 2: Write the boolean provenance in a “Probability Friendly Form” (if possible) Step 3: FPRAS inspired by Karp-Luby framework

Boolean Provenance for Queries q s.t. (q) = 1 Lemma: For any q with (q) = 1, on any D, the provenance f of q(D) has form f is poly-size in n = |D|, poly-time computable

Probability Friendly Form (PFF) f is in PFF, if the negated DNF-s can be written in poly-size d-DNNFs (next slide) If f is in PFF, we can approximate P[f] using Karp-Luby Framework

d-DNNF + Darwiche ’01, ’02, DM ’02 deterministic - Decomposable Negation Normal Form At most one child of a +-node is satisfiable Children of a .-node do not share variables No internal node can have negation + In general, can be a DAG Probability can be computed in linear time

Karp-Luby Framework [KL ’83] Given boolean expression DAGs F1, …, Fm f = F1 + F2 + ... + Fm P[f] can be computed in poly-time (in m, n) if in poly-time,  i (1) P[Fi] can be computed (2) it can be checked if a given assignment satisfies Fi (3) a random satisfying assignment of Fi can be sampled Well-studied special case: DNF counting, where F1, …, Fm are DNF minterms: f = xyz + xyw + wuv

Conditions (1) and (2) hold for PFF Product of minterm and d-DNNF is another d-DNNF + w2=1, z1=1 +

Condition (3) also holds Lemma: Generating a random satisfying assignment on a d-DNNF can be done in poly-time At random + Process in reverse topological order Generate a random satisfying assignment bottom up v1 = 1, v2 = 0 v1 = 0, v2 = 0 v2 = 0 v1 = 1 v1 = 0 v2 = 1 v2 = 0

Expressibility in PFF So, if f is in PFF, we can approximate P[q(D)] But, can we decide in poly-time if some sub-expressions of a boolean expression have poly-size d-DNNFs?  Not known  But, there are natural sufficient conditions that can be verified in poly-time If certain sub-queries are safe and hence generate read-once expressions [OH ’08] If sub-queries generate poly-size OBDDs [JS ’11] Extends to instance-by-instance approach (both q, D given)

#P-hardness for q1 - q2 both q1, q2 are safe CQ-

#P-hardness: Steps in the proof “Hard” query q = q1 – q2 q1() := R1(x, y1) R2(x, y2) R3(x, y3) R4(x, y4) q2() := R1(x1, y) R2(x2, y) R3(x3, y) R4(x4, y) Counting edge covers in bipartite graphs of degree ≤ 4, where the edge set can be partitioned into 4 disjoint matchings Counting independent sets in 3-regular bipartite graphs (XZ ’06)

Other Related Work Semantics of probabilistic query answering Fuhr-Rollecke ’97, Zimanyi ‘97 Dichotomy of CQ- ,CQ and UCQ queries Dalvi-Suciu ’04, ’07, Dalvi-Schnaitter-Suciu ’10 Knowledge compilation techniques Olteanu-Huang ’08, Jha-Olteanu-Suciu ’10, Jha-Suciu ’11, Fink-Olteanu ’11 Instance-by-instance approach Sen-Deshpande-Getoor ’10, Roy-Perduca-Tannen ’11

Conclusions and Future work A step towards understanding complexity of exact and approximate computation for queries with difference operations Future work Dichotomy results that classify syntactically difference queries (similar to positive UCQ)? Extending FPRAS to queries with difference rank > 1? Experimental evaluation of our algorithms

Thank you Questions?