FlumeJava Easy, Efficient Data-Parallel Pipelines Mosharaf Chowdhury

Problem Efficient data-parallel pipelines – Chain of MapReduce programs – Iterative jobs –…–… Exposes a limited set of parallel operations on immutable parallel collections

Goals Expressiveness Abstractions – Data representation – Implementation strategy Performance – Lazy evaluation – Dynamic optimization Usability & deployability – Implemented as a Java library – Inspired by the failure of Lumberjack

FlumeJava Workflow Write a Java program using the FlumeJava library FlumeJava.run(); Optimize Execute PCollection words = lines.parallelDo(new DoFn () { void process(String line, EmitFn emitFn) { for (String word : splitIntoWords(line)) { emitFn.emit(word); } }, collectionOf(strings())); PCollection words = lines.parallelDo(new DoFn () { void process(String line, EmitFn emitFn) { for (String word : splitIntoWords(line)) { emitFn.emit(word); } }, collectionOf(strings()));

Core Abstractions Parallel Collections 1.PCollection 2.PTable Data-parallel Operations Primitives 1.parallelDo() 2.groupByKey() 3.combineValues() 4.flatten() Derived operations 1.count() 2.join() 3.top()

MapShuffleCombineReduce (MSCR) Transform combinations of the four primitives into single MapReduce Generalizes MapReduce – Multiple reducers/combiners – Multiple output per reducer – Pass-through outputs

Optimization Optimizer Strategy 1.Sink flattens 2.Lift CombineValues 3.Insert fusion blocks 4.Fuse parallelDos 5.Fuse MSCRs Optimizer Output 1.MSCR 2.Flatten 3.Operate

Hit or Miss? Sizable reduction in SLOC – Except for Sawzall 5x reduction in average number of stages Faster than other approaches – Except for Hand-optimized MapReduce chains 319 users over a year period