1 Map-Reduce and Datalog Implementation Distributed File Systems Map-Reduce Join Implementations

2 Humongous Data Problems uWe are seeing new applications for very large data operations. wWeb operations, e.g., PageRank. wSocial network data. wCollaborative filtering of commercial data. uResult: new infrastructure. wDistributed file systems. wMap-reduce/Hadoop/Hive/Pig,…

3 Role of Datalog uMany operations are remarkably simple. uExample: suggest new friends in a social network by looking for violations of transitivity: suggest(X,Y) :- friend(X,Z) & friend(Z,Y) & NOT friend(X,Y)

4 Scale of the Problem uFaceBook has 250 million subscribers, each with about 300 friends. uSelf join of friends with itself could have 22.5 trillion tuples. wBut because of “locality,” the size would be less by a factor of perhaps 10–100.

5 Distributed File Systems uTo deal with computations of this size, companies use large collections of commodity servers. wBoth for storage and for computing. Often the same servers. uFiles are stored in chunks, typically 64MB. uChunks are replicated, typically 3 times.

6 Cluster Computing uRacks of compute nodes, interconnected, e.g., by gigabit Ethernet. uNew element: computations involve so much work, that a node failure is common. uMap-reduce (Hadoop) is a framework for dealing effectively with node failure, as well as simplifying certain calculations.

7 Map-Reduce uYou write two functions, Map and Reduce. uSeveral Map tasks and Reduce tasks implement these functions. uEach Map task gets one or more chunks of input data from a distributed file system.

8 Map-Reduce – (2) uMap tasks turn input into a list of key- value pairs. wBut “keys” are not unique. uA master controller assigns each key, and all output from Map tasks with that key, to one of the Reduce tasks. uReduce tasks apply some operation to the values associated with one key.

9 Graph of Map and Reduce Tasks Map Input Reduce.... Output

10 Example: Join by Map-Reduce Answer(X,Y) :- r(X,Z) & s(Z,Y) uMap takes each tuple from r, say r(x,z), and produces the key-value pair [z, (r,x)]. uFrom tuple s(z,y), Map produces key- value pair [z, (s,y)]. uThus, all tuples r(x,z) and s(z,y) go to the same Reduce task.

11 Join by Map-Reduce – (2) uThe Reduce tasks perform a standard join on all the r- and s-tuples they receive. uOutput is the union of the results of all Reduce tasks.

12 Coping With Failures uBecause wEvery Map and Reduce task receives all its input at the beginning, wEvery Map and Reduce task finishes by handing its complete output to the master controller. Any task at a failed node can be restarted without affecting any other task.

13 Multiway Join Via Map-Reduce uFrom Afrati/Ullman in EDBT uUseful for Datalog evaluation because: wBodies often have more than two subgoals. wSeminaive evaluation can involve a complex expression with many relations and their increments (next talk). uNormal procedure is to take a cascade of two-way joins.

14 Multiway Join – (2) uSometimes, it is more efficient to replicate tuples to several Reduce tasks. 1.When relations have large fan-out. uExamples: “friends” or links on the Web. 2.Star joins. uJoin of a large fact table with smaller dimension tables. uIntuition: wins when intermediate joins would be large.

15 Multiway Join – (3) uAssume k Reduce tasks. uCertain variables get shares of a hash function that maps to k buckets. wProduct of the shares = k. uIf variable X has share x, then each X- value is hashed to one of x hash keys. uHash key of a Reduce task = vector of hash values for each variable with a share.

16 Example: Multiway Join Answer(W,X,Y,Z) :- r(W,X) & s(X,Y) & t(Y,Z) uOnly X and Y get a share, say xy = k. wTheorem: never give a share to a dominated variable. = variable that appears only where some other variable also appears. uTuple s(a,b) goes only to Reduce task [h(a), h’(b)].

17 Example: Multiway Join – (2) Answer(W,X,Y,Z) :- r(W,X) & s(X,Y) & t(Y,Z) uHowever, tuple r(a,b) must go to all Reduce tasks [h(b), n] where n is any of y different hash values. uSimilarly, tuple t(a,b) must go to all Reduce tasks [m, h’(a)], where m is any of x different hash values.

18 Example: Multiway Join – (3) Answer(W,X,Y,Z) :- r(W,X) & s(X,Y) & t(Y,Z) uTo minimize the number of tuples transmitted, pick: ux =  k|r|/|t| uy =  k|t|/|r| uIntuition: costs distributing tuples of r and t are the same.

19 Summary of Afrati/Ullman EDBT-2010 uIt is possible to find the optimum shares for variables for any join. uUsually, the process is a straightforward Lagrangean analysis. uIn pathological cases, an exponential search appears necessary. uConstraining to positive integers and adjusting the product add complexity.