MapReduce: Simplified Data Processing on Large Cluster Authors: Jeffrey Dean and Sanjay Ghemawat Presented by: Yang Liu, University of Michigan EECS 582 – W161

About the Authors EECS 582 – W162 Jeff DeanSanjay Ghemawat

Motivation Challenge at google Input data too large -> distributed computing Most computations are straightforward(log processing, inverted index) -> boring work Complexity of distributed computing Machine failure Scheduling EECS 582 – W163

Solution: MapReduce MapReduce as the distributed programing infrastructure Simple Programming interface: Map + Reduce Distributed implementation that hides all the messy details Fault tolerance I/O scheduling parallelization EECS 582 – W164

Programming Model EECS 582 – W165 Inspired by map and reduce functions in Lisp and other functional programing languages Lisp: Map #‘length’ (() (a) (ab) (abc)) Reduce #‘+’ ( )

Programing Model Programmer only need to specify two functions: Map Function map (in_key, in_value) -> list(out_key, intermediate_value) Process input key/value pair Produce set of output key/intermediate value pairs Reduce Function reduce (out_key, intermediate_value) -> list(out_value) Process intermediate key/value pairs Combines intermediate values per unique key Produce a set of merged output values(usually just one) EECS 582 – W166

Programming Model EECS 582 – W167 [input (key, value)] [Intermediate (key, value)] [Unique key, output value list] Map Function Shuffle (merge sort by key)Reduce function

Example: WordCount EECS 582 – W168 the small brown fox a fox speaks to another fox brown cow cross the road Input Shuffle Split the small brown fox a fox speaks to another fox brown cow cross the road Map Reduce Output a, 1 another 1 brown, 2 cross, 1 cow, 1 fox, 3 road, 1 small, 1 speaks, 1 the, 2 to, 1

More Programs based on MR EECS 582 – W169 Inverted Index Distributed Sort Distributed Grep URL Frequency MapReduce Program in Google source tree

System Implementation: Overview Cluster Characteristic 100s/1000s of 2-CPU x86 machines, 2-4 GB of memory Limited bisection bandwidth Storage is on local IDE disks Infrastructure GFS: distributed file system manages data (SOSP'03) Job scheduling system: jobs made up of tasks, scheduler assigns tasks to machines (Borg?) EECS 582 – W1610

GFS Scheduler submit allocate Control Flow and data flow EECS 582 – W1611 User Program Worker Master Worker fork assign map assign reduce read remote read, sort Output File 0 Output File 1 write Split 0 Split 1 Split 2 Input Data MapReduce Architecture local write Notify location of local write

Execution EECS 582 – W1612

Parallel Execution EECS 582 – W1613

Coordinate EECS 582 – W1614 Master data structures Task status: (idle, in-progress, completed) Idle tasks get scheduled as workers become available When a map task completes, it sends the master the location and sizes of its R intermediate files, one for each reducer Master incrementally pushes this info to reducers Master pings workers periodically to detect failures

Fault Tolerance EECS 582 – W1615 Map worker failure Completed or in-progress tasks are reset to idle Reduce worker failure Only in-progress tasks are reset to idle Why? Master failure MapReduce Task is aborted and client is notified Reset tasks are rescheduled on another machine

Task Granularity EECS 582 – W1616 Many more map tasks than machines Minimizes time for fault recovery Can pipeline shuffling with map Better dynamic Load balancing Execution Often use 200,000 map/5000 reduce tasks w/ 2000 machines

Backup Tasks EECS 582 – W1617 Slow worker delay completion time Processor’s cache being disabled Bad disk with soft errors Start back-up tasks, for those in-progress as the job nears completion First task to complete is considered Disk

Disk Locality EECS 582 – W1618 Map tasks are scheduled close to data on nodes that have input data if not, on nodes that are nearer to input data Ex. Same network switch Conserves network bandwidth Leverage the Google File System

Combiner Function EECS 582 – W1619 Local reducer at the map worker Can save network time by pre-aggregating at mapper Works only if reduce function is commutative and associative combine(k1, list(v1)) Map

WordCount: No combine EECS 582 – W1620 the small brown fox a fox speaks to another fox brown cow cross the road Input Shuffle Split the small brown fox a fox speaks to another fox brown cow cross the road Map Reduce Output a, 1 another 1 brown, 2 cross, 1 cow, 1 fox, 3 road, 1 small, 1 speaks, 1 the, 2 to, 1

WordCount: Combine EECS 582 – W1621 the small brown fox a fox speaks to another fox brown cow cross the road Input Shuffle Split the small brown fox a fox speaks to another fox brown cow cross the road Map Reduce Output a, 1 another 1 brown, 2 cross, 1 cow, 1 fox, 3 road, 1 small, 1 speaks, 1 the, 2 to, 1

Partitioning Function EECS 582 – W1622 Records with the same intermediate key end up at the same reducer Default partition function e.g., hash(key) mod R Sometimes useful to override E.g., hash(hostname(URL)) mod R ensures URLs from a host end up in the same output file

More Features EECS 582 – W1623 Skipping Bad Records Input and Output Types Local Execution Status Information Counters etc...

Performance EECS 582 – W1624 Tests run on cluster of 1800 machines: 4 GB of memory 2 GHz Intel Xeon Dual-processor Dual 160 GB IDE disks Two benchmarks: MR_GrepScan byte records to extract records matching a rare pattern MR_SortSort byte records

Performance – MR_Grep EECS 582 – W1625 Locality optimization helps: 1800 machines read 1 TB of data at peak of ~31GB/s Without this, rack switches would limit to 10 GB/s Startup overhead is significant for short jobs

Performance – MR_Sort EECS 582 – W1626 Backup tasks reduce job completion time significantly System deals well with failures

Conclusion EECS 582 – W1627 Inexpensive commodity machines can be the basis of a large scale reliable system MapReduce hides all the messy details of distributed computing MapReduce provides a simple parallel programming interface

Lessons learn EECS 582 – W1628 General Design General Abstraction -> Solve many problems -> Success Simple Interface -> Fast Adaption -> Success Distributed System design Network is a scarce resource Locality matters Pre-aggregate whenever possible Master-Worker architecture is simple yet powerful

Influence EECS 582 – W1629 MapReduce is one of the MOST cited system paper : as for 03/08/ Together with Google File System, Bigtable, it inspires the Big Data Era What happen after MapReduce?

In Open source world: Hadoop EECS 582 – W : Doug Cutting and Michael J. Cafarella developed Hadoop to support distribution for the Nutch search engine project(invert index) Now:

In Google EECS 582 – W1631

Barrier Problem with MapReduce: I/O Barrier Any MR algorithm can be simulated on BSP and vice versa EECS 582 – W1632 Map Reduce GFSGFS GFSGFS

Post MapReduce System EECS 582 – W1633 Hadoop MapReduce Giraph... MapReduce MillWheel DataFlow Model(Apache Beam) TensorFlow Pregel Google Open Source MapReduce DAG Computing Graph Computing Stream Processing Machine Learning General Model Tez Spark FlumeJava MLib GraphX Spark Streaming Spark Net Storm

Questions? EECS 582 – W1634

References MapReduce Architecture: MapReduce Presentation: MapReduce Presentation: Operating system support for warehouse-scale computing: Apache Ecosystem Pic: MapReduce: osdi04.pdf/ osdi04.pdf/ FlumeJava: CloudPapers/FlumeJava.pdf/ CloudPapers/FlumeJava.pdf/ MillWheel: Pregel: EECS 582 – W1635

References Giraph: Spark: Tez: DataFlow: Tensorflow: Apache Beam: SparkNet: Caffe on Spark: distributed-deep-learning-on-hadoophttp://yahoohadoop.tumblr.com/post/ /large-scale- distributed-deep-learning-on-hadoop EECS 582 – W1636