1. MapReduce: Simplified Data Processing on Large Clusters Authors: Jeffrey Dean and Sanjay Ghemawat Presenter: Guangdong Liu Jan 28th, 2011

2. Presentation Outline qMotivation qGoal qProgramming Model qImplementation qRefinement

3. Motivation qLarge-scale data processing l Many data-intensive applications involve processing huge amounts of data and then producing lots of other data l Certain common themes are shared when executing such applications 4 Hundreds or thousands of machines are used 4 Two categories of basic operation on the input data: 1) Map():process a key/value pair to generate a set of intermediate key/value pairs 2) Reduce(): merge all intermediate values with the same key

4. Goal qMapReduce: an abstraction that allows users to perform simple computations across large data set which is distributed on large clusters of commodity PCs while hiding the details of parallelization, data distribution, load balancing and fault toleration l User-defined functions l Automatic parallelization and distribution l Fault tolerance l I/O scheduling l Status monitoring

5. Programming Model qInspired by Lisp primitives map and reduce qMap(key, val) l Written by a user l Process a key/value pair to generate intermediate key/value pairs l The MapReduce library groups all intermediate values associated with the same key together and passes them to the reduce function qReduce(key,vals) l Also written by a user l Merge all intermediate values associated with the same key

6. Programming Model

7. qCount words in docs l Input consists of (doc_url, doc_contents) pairs l Map(key=doc_url, val=doc_contents), for each word w in contents, emit(w, “1”) l Reduce(key=word, values=counts_list), sum all “1”s in value list and emit result “(word, sum)”

8. Programming Model Hello World, Bye World! Hello MapReduce, Goodbye to MapReduce. Welcome to UNL, Goodbye to UNL. Reduce Phase DFS Map Phase Intermediate Result DFS M1M2M3 (Hello, 1) (Bye, 1) (World, 1) (Welcome, 1) (to, 1) (Goodbye, 1) (UNL, 1) (Hello, 1) (to, 1) (Goodbye, 1) (MapReduce, 1) R1 R2 (Hello, 2) (Bye, 1) (Welcome, 1) (to, 3) (World, 2) (UNL, 2) (Goodbye, 2) (MapReduce, 2)

9. Implementation qUser to do list l Indicate input and output files l M: number of map tasks l R: number of reduce tasks l W: number of machines l Write map and reduce functions l Submit jobs l This requires no knowledge of parallel/distributed systems!!!

10. Implementation … Reduce Phase DFS … Map Phase Master M2 R1 Input P1... … Pr B2 … Bn B1 M1 Local Write Read from DFS P1 … Pr P1 … Pr Assign MapTask Assign ReduceTas k Remote Read Output 1 Output r Write to DFS … Intermediate Result DFS Rr ReducerMapper Mn

11. Implementation 1.Input files split (M splits) l Each block is typically 16~64MB l Start up many copies of user program on a cluster of machines 2.Master & Workers l One special instance becomes the master l Workers are assigned tasks by the master l There are M map tasks and R reduce tasks to assign l Master finds idle workers and assigns map or reduce tasks to them

12. Implementation 3. Map tasks l Map workers read contents of corresponding input partition l Perform user-defined map computation to create intermediate pairs l The intermediate pairs produced by the map function are buffered in memory 4.Writing intermediate data to disk (R regions) l Buffered output pairs written to local disk periodically l Partitioned into R regions by a partitioning function l Location of these buffered pairs on the local disk are passed back to the master

13. Implementation 5.Read & Sorting l Use remote procedure calls to read the buffered data from the local disks of map workers l Sort intermediate data by the intermediate keys 6.Reduce tasks l Reduce worker iterates over ordered intermediate data l Each unique key encountered – key & values are passed to user's reduce function l Output of user's reduce function is written to output file on a global file system 7.When all tasks have completed, the master wakes up user program

14. Implementation qFault tolerance-in a word, redo l Workers are periodically pinged by master l No response = failed worker l Reschedule failed tasks l Note: completed map task by the failed worker need to be re-executed because the output is stored on the local disk

15. Implementation qLocality l Input data is managed by GFS and has several replicas l Schedule a task on a machine containing a local replica or near a replica qTask Granularity l M map tasks and R reduce tasks l Make M and R much larger than number of worker machines

16. Implementation qBackup tasks l Straggler: a machine that takes an unusually long time to complete one of the last few map or reduce tasks in the computation. l Cause: bad disk, competition for CPU … l Resolution: schedule backup executions of in-progress tasks when a MapReduce operation is close to completion

17. Source qThe example is quoted from: l Wei Wei; Juan Du; Ting Yu; Xiaohui Gu;, "SecureMR: A Service Integrity Assurance Framework for MapReduce," Computer Security Applications Conference, 2009. ACSAC '09. Annual, vol., no., pp.73-82, 7-11 Dec. 2009

18. Making Cluster Application Energy-Aware Authors: Nedeljko Vaasic, Martin Braistits and Vincent Salzgerber Jan 28th, 2011

19. Outline qIntroduction qCase Study qApproach

20. Introduction qPower consumption l A critical issue in large scale clusters l Data centers consume as much energy as a city l 7.4 billion dollars per year qCurrent techniques for efficiency l Consolidate workload into fewer machines l Minimize the energy consumption while keeping the same overall performance level qProblems l Cannot operate at multiple power levels l Cannot deal with energy consumption limits

21. Case Study qGoogle’s Server Utilization and Energy Consumption

22. Case Study qHadoop Distributed File System (HDFS)

23. Case Study qHadoop Distributed File System (HDFS)

24. Case Study qMapReduce

25. Case Study qConclusion l It is a wise decision to aggregate load on a fewer number of machines for saving energy l Distributed applications must actively participate in the power management in order to avoid poor performance

26. Approach

27. On the Energy (In)efficiency of Hadoop Clusters Authors: Jacob Leverich, Christ Kozyrakis Jan 28th, 2011

28. Introduction qImprovement of energy efficiency of a cluster l Place some nodes into low-power standby modes l Avoid energy waste on oversized components for each node qProblems

29. Approach qHadoop data layout overview l Distribute replicas across different nodes in order to improve performance and reliability l The user specifies a block replication factor n to ensure n identical copies of any data-block are stored across a cluster (typically n=3) l The largest number of nodes that can be disabled without impacting data availability is n-1

30. Approach qCovering subset l At least one replica of a data-block must be stored in a subset of nodes called covering subset l Make sure that a large number of nodes can be gracefully removed from a cluster without affecting the availability of data or interrupting the normal operation of a cluster