The Communication Complexity of Distributed Set-Joins Dirk Van Gucht, Ryan Williams, David Woodruff, Qin Zhang Indiana University, Stanford, IBM Almaden, Indiana University

Set-Joins Given sets A1, …, Am from [n] = {1, 2, …, n} Given sets B1, …, Bm from [n] Join = {(i,j) for which P(Ai, Bj)}, where P is a predicate P might be: set-intersection |Ai ∩ Bj| > 0 set non-intersection |Ai ∩ Bj| = 0 set equality Ai = Bj set-intersection threshold join |Ai ∩ Bj | > T?

Communication Complexity A1, …, Am B1, …, Bm Alice and Bob want to minimize communication so that they together can output the join The protocol can fail with probability 1/100 (over its randomness)

Communication Complexity of Joins Set-Intersection, Set Non-Intersection, Set-Intersection Threshold Join all require Ω(mn) communication Reduction from set-disjointness on a universe of size m*n Matched by trivial upper bound {x1, x2, …, xmn} {y1, y2, …, ymn} B1 = {y1, y2, …, yn} A1 = {x1, x2, …, xn} B2 = {yn+1, y2, …, y2n} A2 = {xn+1, xn+2, …, x2n} … … Am = {x(m-1)n+1, …, xmn} Bm = {y(m-1)n, …, ymn}

Communication Complexity of Joins Set-Equality has communication Θ(m) Ω(m) bits transferred to the player outputting the join O(m) upper bound due to a set-intersection protocol of [BCKWY] A1, …, Am B1, …, Bm Randomly hash sets to [m2] x1, …, xm in [m2] y1, …, ym in [m2]

Communication Complexity of Joins Can we achieve better communication for joins in practice? Sparsity Output: typically the output size |Join(A1, …, Am, B1, …, Bm)| is at most k << m2 Input: typically |A1|, …, |Am|, |B1|, …, |Bm| each at most s << n Can get better bounds in terms of k, s! Focus on set-intersection join (will generalize to natural join)

Our Main Results O(k1/2n) communication protocol for set-intersection join More efficient than O(mn) for small k! Improves O(kn) communication protocol of [Williams, Yu, SODA ’14] Matching Ω(k1/2n) lower bound For input sparsity s, we show Θ(min(ms, (skn)1/2) + n) communication bound Highlights Use 2 rounds of interaction (1 round is impossible!) For each (i,j) in the join, returns all k for which k in Ai and k in Bj (natural join) New algorithms for input/output sparse matrix multiplication

Talk Outline Our O(k1/2n) communication protocol Extension to solve the natural join problem New algorithm for matrix multiplication input/output Application to Sparse Transitive-Closure An Ω(k1/2n) communication lower bound Estimating sizes of set joins

A Protocol for Set-Intersection Join A1, …, Am B1, …, Bm [ B1 B2 …Bm [ A1 A2 … Am B = A = Set-Intersection Join = {(i,j) for which Ci,j > 0, where C = A*B}

A Protocol for Set-Intersection Join Set-Intersection Join = {(i,j) for which Ci,j > 0, where C = A*B} m x n Matrix A n x m Matrix B Alice computes S*A for a random k x m matrix S, sends S*A to Bob [Compressed Sensing] S has the property that if x is a vector containing at most k non-zero entries, then given S*x, one can recover x w.h.p.

A Protocol for Set-Intersection Join m x n Matrix A m x n Matrix B Alice computes S*A for a random k x m matrix S, sends S*A to Bob Bob computes (S*A)*B. Each column of A*B has at most k non-zeros From S*A*B, Bob can recover all non-zero entries of A*B S can be a random Gaussian matrix with O(log n) bits of precision per entry

A Protocol for Set-Intersection Join m x n Matrix A m x n Matrix B Problem: sending S*A requires k n log n bits of communication Idea: instead let S be a k1/2 x m random matrix! From S*A*B, can recover each column of A*B with at most k1/2 non-zeros Number of columns of A*B with more than k1/2 non-zeros is at most k1/2

Overall Protocol for Set-Intersection Join Alice sends Bob S*A where S has k1/2 rows Alice also sends Bob T*A, where T has O(log n) rows [KNW] Bob computes T*A*B and finds which columns of A*B have > k1/2 non-zeros Set H of such columns has size at least k1/2 Bob computes S*A*B and recovers (A*B)j for each j not in H Bob sends Alice columns Bj for each j in H Alice directly computes A*Bj for each j in H

Extension to Solve the Natural Join Not enough to just figure out Ci,j > 0, need to find all witnesses A witness is an index k for which Ai,k = Bk,j = 1 Can be solved iteratively Bob deletes entries of B one-at-a-time and re-computes S*A*B Alice just sends S*A once Since Alice has Bj for columns j in H, she directly computes all witnesses

A New Algorithm for Matrix Multiplication Suppose we choose S to be a CountSketch matrix [ 0 0 1 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 -1 1 0 -1 0 0-1 0 0 0 0 0 1 The same algorithm works to recover C = A*B! Time nnz(A) to compute S*A nnz(B)k1/2 to compute S*A*B and recover A*Bj for j not in H nnz(A)k1/2 to compute A*Bj for j in H

Matrix Multiplication Application Gives O((nnz(A) + nnz(B))k1/2) time for matrix multiplication [Sparse Transitive Closure Application] Given: a directed graph G on n nodes Transitive-closure is pairs (u,v) of nodes for which v is reachable from u Promise: transitive-closure has O(n) edges Goal: find all edges in the transitive-closure Solve by computing A, A2, A4, …, An , where A is adjacency matrix We obtain O~(n1.5) time algorithm. Previous best is n1.844 time [BCH]

Matching Communication Lower Bound Reduction from set-disjointness on a universe of size k1/2n {y1, y2, …, ysqrt{k}n} {x1, x2, …, xsqrt{k}n} B1 = {y1, y2, …, yn} A1 = {x1, x2, …, xn} B2 = {yn+1, y2, …, y2n} A2 = {xn+1, xn+2, …, x2n} … … Bm = {y(sqrt{k}-1)n, …, ysqrt{k}n} Am = {x(sqrt{k}-1)n+1, …, xsqrt{k}n} A*B is k1/2 x k1/2 matrix so it has at most k non-zeros A non-zero on the diagonal can decide disjointness

Our Other Results Approximate size |Join(A1, …, Am, B1, …, Bm)| Query strategy planning, measuring similarity, etc. Output X = (1±ε) |Join(A1, …, Am, B1, …, Bm)| Communication of Approximate Set-Intersection Join [1-way Communication] Upper and lower bounds of n/ε2 [2-way Communication] Lower bound of n/ ε2/3 Results for set-intersection threshold join as well (tight for 1-way)

Conclusion Summary Separation of communication complexity of different join types Optimal communication protocol for Set-Intersection and Natural Joins in terms of sparsity New algorithms for sparse input and output matrix multiplication Upper and lower communication bounds for approximating join sizes Open Questions Communication bounds not tight for approximating join sizes for most joins For T-threshold join, off by a (large) factor of sqrt{n/T} Multi-player versions and more expressive joins Other applications of our matrix multiplication algorithm