MapReduce: Simplied Data Processing on Large Clusters Written By: Jeffrey Dean and Sanjay Ghemawat Presented By: Manoher Shatha & Naveen Kumar Ratkal

Overview Motivation Introduction Programming Model Implementation Refinements Performance Conclusion

Motivation Issues in processing large amount of data over thousands of CPU’s. How to perform parallelization, data distribution etc. How to handle CPU failures ? MapReduce Provides: Automatic parallelization and Distribution. Fault tolerance. Monitoring and Status updates.

Introduction MapReduce: is a programming model and an associated implementation for processing and generating large data sets. Users specify Map and Reduce functions. Map is applied to get intermediate key/value pair. Reduce function combines all the intermediate values that shares the same key. These functions parallelize large computations easily. Definition taken from the paper

Programming Model MapReduce model takes set of key/value pair as the input and produces a new set of key/value pair as the output. MapReduce is a two fold process. Map function library groups together all intermediate values associated with same intermediate key ‘I’ and passes them to Reduce function. Reduce function library accepts intermediate key I and set of values for that key and then it combines all the similar keys.

Example of MapReduce The problem of counting the number of occurrences of each word in a large collection of documents. Input consists of (url, contents) pairs map(key=url, val=contents): For each word w in contents, emit (w, “1”) reduce(key=word, values=uniq_counts): Sum all “1”s in values list Emit result “(word, sum)” Map output: (shatha, 1) (ratkal, 1) (alice, 1) (shatha, 1) (bob, 1) (shatha, 1) (ratkal, 1) Reduce output: (alice, 1 ) (bob, 1 ) (ratkal, 2 ) (shatha, 3) Input document: shatha ratkal alice shatha bob shatha ratkal Code taken from the paper

Example of MapReduce Count of URL access frequency Map Function goes through all the log files (web page requests) and outputs (URL, 1). Reduce combines all values for the same URL and outputs (URL, total count). Reverse Web Link Graph Map: It generates (target, source) pairs for each link found in a web page. Reduce: It combines all the list of all source URLs linked with a given target URL and outputs (target, list(source) ).

Example of MapReduce Contd… Inverted Index Map : This function parses each document, and emits a sequence of (word, document ID ) pairs. Reduce : It accepts all pairs for a given word, sorts the corresponding document ID’s and emits a (word, list(document ID)). The set of all output form a simple inverted index.

Implementation Many different implementations are possible, best choice depends on the environment. Computing environment Dual Processor x86 Processors running Linux. With 2-4 GB memory per machine. N/W bandwidth is 1 gigabit/sec. Clusters consists of thousands of machines. Uses inexpensive IDE’s. Jobs are submitted to scheduling system.

Execution Overview Input Key/Value pair are split into blocks. Map function is performed on those blocks in parallel. Input splits can be processed in parallel by different machines. Sort the key/value pair. Reduce function is run on these intermediate key/value pair. If any of the function fails then those blocks are executed on other machines.

Execution Overview Contd… Figure taken from the paper Split 1 Split 0 Split 2 Split 3 Split 4 User Program Master Worker Output File 0 Output File 1 (1) fork (2) assign map (2) assign Reduce (3) read (4) Local write (5) Remote read (6) write Input Files Map Phase Intermediate files (on local disks) Reduce Phaseoutput Files

Master Data Structures Data structures are maintained by master. Master keeps the state of the worker machine for each map/reduce tasks. Intermediate files are passed from map to reduce function by master.

Fault Tolerance Worker Failure: The master pings each and every worker. Master waits for specified time. If no response then the worker machine is marked as failed. Work completed by failure worker machines are re-executed. All completed reduced tasks are not re-executed.

Fault Tolerance Contd… Master Failure: Master writes periodic check points to the master data structure. If master task dies, a new copy can be started from the last check point.

Backup Tasks What lengthens the total time for MapReduce operation? Straggler Straggler is a machine which takes longtime for assigned Map or Reduce tasks. How it is solved? When MapReduce operation is about to complete, the master schedules a backup executions for all those on going tasks. Backup is marked complete if it executes first else the primary.

Refinements User specify number of reduce tasks/output files that they need. Data get partitioned using the partitioning function on the intermediate key. Default partitioning function is provided. Example : hash(key) mod R Special partitioning function is : Example : hash(hostname(urlkey)) mod R Partitioning Function Ordering Guarantees The intermediate key/value pair are processed in increasing key order. This helps in generating the sorted output file per partition.

Refinements Contd… Some times there are significant repetitions in the intermediate keys generated by each map task. Example: Users are allowed to specify the combiner function that does partial merging. Partial combining considerably speeds up certain classes of map reduce operations. Combiner Function Input/Output Types Map Reduce library supports reading input data in several format. Example: In text file Key is offset in the file and value is the content of the line. One can add support for new input type by providing implementation of a simple reader interface.

Refinements Contd… Due to bugs in the Map Reduce function system crashes on certain records. Worker process installs a signal handler. If user code generates a signal then the sequence number is sent to master. If master notices enough failures then it skips that particular record. Skipping Bad Records

Refinements Contd… Status Information Master generates a set of status pages by running the HTTP server. These pages contains the information about the progress of computation, processing rate, etc. It even shows which worker failed and which Map Reduce task they were processing.

Performance Tests run on cluster of 1800 machines: 4 GB of memory Dual-processor 2 GHz Xeons Dual 160 GB IDE disks Gigabit Ethernet per machine Bench Marks Grep: The Grep program scans through 10 exp byte records, searching for relatively rare three character pattern. Sort : The Sort program sorts 10 exp byte records.

Performance Contd… Graph shows the progress of the computation overtime. Y-axis shows the rate at which the input data is scanned. The rate gradually peaks up as more machines assigned to this computation. As the map tasks finish, the rate starts dropping and hits zero about 80 sec in computation. Map Reduce Grep

Performance Contd… Map Reduce Sort Backup tasks reduce job completion time a lot. System deals well with failures. Normal No backups 200 processed killed

Conclusion MapReduce is a frame work for running large-scale parallel computations developed at Google. Easy to use. Google stores petabytes of data, MapReduce helps in processing and generating this data over thousands of commodity servers. Greatly simplifies large scale computations.