Map-Reduce-Merge: Simplified Relational Data Processing on Large Clusters H.Yang, A. Dasdan (Yahoo!), R. Hsiao, D.S.Parker (UCLA) Shimin Chen Big Data Reading Group Presentation

Motivation Map-Reduce framework Compared to relational DBMS “simplified” for data processing in search engines Problem: join multiple heterogeneous datasets Not quite fit into map-reduce Ad-hoc solutions: map-reduce on one data set while reading data from the other dataset on the fly

Contribution Goal: support relational algebra primitives without sacrificing existing generality and simplicity Proposal: map-reduce-merge

Outline Introduction Map-Reduce Map-Reduce-Merge Applications to Relational Data Processing Optimizations and Enhancements Case Studies Conclusion

Let’s Refresh Our Memory Functional programming model

Comments Low-cost unreliable commodity hardware Failure often occurs during each map/reduce task Coordinator re-run mapper or reducer Homogenization: for equi-join Transform each dataset into (join key, payload) Then apply map-reduce to merge entries from different datasets Problem: only equi-joins may take lots of extra disk space, incur excessive communications

Outline Introduction Map-Reduce Map-Reduce-Merge Applications to Relational Data Processing Optimizations and Enhancements Case Studies Conclusion

Map-Reduce-Merge Primitives key join

Focusing on Merge Two sets of inputs generated by multiple reducers: Which α reducers and β reducers match? How to get the next key-value pair? Customized preprocessing for inputs? Merging algorithm? All of these are customizable

Focusing on Merge Two sets of inputs generated by multiple reducers: Partition Selector: Which α reducers and β reducers match? Iterator: How to get the next key-value pair? Processor: Customized preprocessing for inputs? Merger: Merging algorithm? All of these are customizable

Example: Emp & Dept EmployeeDepartment

Partition Selector LHS: reduce key:dept-id, emp-idpartition key: dept-id RHS: reduce key:dept-id, partition key: dept-id Assuming #reducer is the same, LHS reducer K matches RHS reducer K

Processor Pre-processing for each input E.g. building hash table for hash join This example is sort-merge Processor is empty

Iterator for sort-merge

Merger

Other Iterators Nested-loop: For each (k,v) of the first input, get all the second input Then rewind the second input and process the next (k,v) of the first input Hash join: Read all of one input, then read all of the other input

Outline Introduction Map-Reduce Map-Reduce-Merge Applications to Relational Data Processing Optimizations and Enhancements Case Studies Conclusion

Relation Relation R with an attribute set A A is broken down into a key part K, and a value part V

Relational Operators Generalized selection: choosing a subset of records Filtering can be done in mapper/reducer/merger Projection: choosing a subset of attributes User-defined mapper (k,v)  (k’,v’) Aggregation Group-by is performed before reduce Easy to implement aggregation in reducer Joins (set union, intersection, difference, cartesian product) Sort-merge, hash join, nested-loop Rename

Outline Introduction Map-Reduce Map-Reduce-Merge Applications to Relational Data Processing Optimizations and Enhancements Case Studies Conclusion

Partition Selector In general: LHS has R1 reducers, RHS has R2 reducers, performing cartesian-product like operator Suppose R1  R2, use R1 merger, where merger j selects: Input from LHS reducer j Input from RHS all reducers Remote reads: R1*(1+R2) = R1 + R1*R2 Natural equi-join case: Let R1==R2==R, use R merger, where merger j selects: LHS reducer j and RHS reducer j Remote reads: 2*R

Combining Phases Entire workflow consists of multiple map-reduce-merge To avoid remote copying: ReduceMap, MergeMap: co-locate next mapper with previous reducer or merger ReduceMerger: co-locate merger with one of the reducer ReduceMergeMap

Map-Reduce-Merge Library Put common merge implementations into a library Joins Common iterators etc.

Configuration API for building a Customized Workflow Map/ reduce Map/reduce/mergeMultiple Map/reduce/merge

Outline Introduction Map-Reduce Map-Reduce-Merge Applications to Relational Data Processing Optimizations and Enhancements Case Studies Conclusion

Webgraphs Each row: (URL, in-links, out-links) Potentially large number of links Only a few are needed for many operations Store each column of the table in a separate file Reconstruct the table by join E.g. compute the intersection of in-links and out- links

TPC-H Query 2

After Combining Phases

Conclusion Extend map-reduce Support relational operators However, the merge step seems quite complicated