1. Introduction to MapReduce ECE7610

2. The Age of Big-Data  Big-data age  Facebook collects 500 terabytes a day(2011)  Google collects 20000PB a day (2011)  Data is an important asset to any organization  Finance company; insurance company; internet company  We need new  Algorithms/data structures/programming model 2

3. What to do ? (Word Count)  Consider a large data collection and count the occurrences of the different words 3 Data collection web2 weed1 green2 sun1 moon1 land1 part1 {web, weed, green, sun, moon, land, part, web, green,…}

4. What to do ?(Word Count) 4 Data collection web2 weed1 green2 sun1 moon1 land1 part1 Multi-thread Lock on shared data

5. What to do?(Word Count) 5 Data collection  Single machine cannot serve all the data: you need a distributed special (file) system  Large number of commodity hardware disks: say, 1000 disks 1TB each  Critical aspects: fault tolerance + replication + load balancing, monitoring  Exploit parallelism afforded by splitting parsing and counting  Provision and locate computing at data locations

6. What to do? (Word Count) 6 KEYwebweedgreensunmoonlandpartwebgreen……. VALUE web2 weed1 green2 sun1 moon1 land1 part1 Data collection Separate counters Separate data Data collection Data collection Data collection Data collection

7. It is not easy to parallel…. 7 Fundamental issues Scheduling, data distribution, synchronization, inter- process communication, robustness, fault tolerance, … Different programming models Message Passing Shared Memory Architectural issues Flynn’s taxonomy (SIMD, MIMD, etc.), network topology, bisection bandwidth, cache coherence, … Common problems Livelock, deadlock, data starvation, priority inversion, …dining philosophers, sleeping barbers, cigarette smokers, … Different programming constructs Mutexes, conditional variables, barriers, … masters/slaves, producers/consumers, work queues,. … Actually, Programmer’s Nightmare….

8. MapReduce: Automate for you  Important distributed parallel programming paradigm for large-scale applications.  Becomes one of the core technologies powering big IT companies, like Google, IBM, Yahoo and Facebook.  The framework runs on a cluster of machines and automatically partitions jobs into number of small tasks and processes them in parallel.  Features: fairness, task data locality, fault-tolerance. 8

9. MapReduce 9 MAP: Input data  pair Data Collection: split1 web1 weed1 green1 sun1 moon1 land1 part1 web1 green1 …1 KEYVALUE Split the data to Supply multiple processors Data Collection: split 2 Data Collection: split n Map …… Map …

10. MapReduce 10 Reduce MAP: Input data  pair REDUCE: pair  Data Collection: split1 Split the data to Supply multiple processors Data Collection: split 2 Data Collection: split n Map …… Map …

11. C. Xu @ Wayne State11 Count Large scale data splits Parse- hash Map Reducers (say, Count)

12. MapReduce 12

13. How to store the data ? 13 Compute Nodes What’s the problem here?

14. Distributed File System  Don’t move data to workers… Move workers to the data!  Store data on the local disks for nodes in the cluster  Start up the workers on the node that has the data local  Why?  Not enough RAM to hold all the data in memory  Network is the bottleneck, disk throughput is good  A distributed file system is the answer  GFS (Google File System)  HDFS for Hadoop 14

15. GFS/HDFS Design  Commodity hardware over “exotic” hardware  High component failure rates  Files stored as chunks  Fixed size (64MB)  Reliability through replication  Each chunk replicated across 3+ chunkservers  Single master to coordinate access, keep metadata  Simple centralized management  No data caching  Little benefit due to large data sets, streaming reads  Simplify the API  Push some of the issues onto the client 15

16. GFS/HDFS 16

17. MapReduce Data Locality  Master scheduling policy  Asks HDFS for locations of replicas of input file blocks  Map tasks typically split into 64MB (== GFS block size)  Locality levels: node locality/rack locality/off-rack  Map tasks scheduled as close to its input data as possible  Effect  Thousands of machines read input at local disk speed. Without this, rack switches limit read rate and network bandwidth becomes the bottleneck. 17

18. MapReduce Fault-tolerance  Reactive way  Worker failure Heartbeat, Workers are periodically pinged by master –NO response = failed worker If the processor of a worker fails, the tasks of that worker are reassigned to another worker.  Master failure Master writes periodic checkpoints Another master can be started from the last checkpointed state If eventually the master dies, the job will be aborted 18

19. MapReduce Fault-tolerance  Proactive way (Speculative Execution)  The problem of “ stragglers ” (slow workers) Other jobs consuming resources on machine Bad disks with soft errors transfer data very slowly Weird things: processor caches disabled (!!)  When computation almost done, reschedule in- progress tasks  Whenever either the primary or the backup executions finishes, mark it as completed 19

20. MapReduce Scheduling  Fair Sharing  conducts fair scheduling using greedy method to maintain data locality  Delay  uses delay scheduling algorithm to achieve good data locality by slightly compromising fairness restriction  LATE(Longest Approximate Time to End)  improves MapReduce applications' performance in heterogenous environment, like virtualized environment, through accurate speculative execution  Capacity  introduced by Yahoo, supports multiple queues for shared users and guarantees each queue a fraction of the capacity of the cluster 20

21. MapReduce Cloud Service Providing MapReduce frameworks as a service in clouds becomes an attractive usage model for enterprises. A MapReduce cloud service allows users to cost-effectively access a large amount of computing resources with creating own cluster. Users are able to adjust the scale of MapReduce clusters in response to the change of the resource demand of applications. 21

22. Amazon Elastic MR You 1. Scp data to cluster 2. Move data into HDFS 3. Develop code locally 4. Submit MapReduce job 4a. Go back to Step 3 5. Move data out of HDFS 6. Scp data from cluster 0. Allocate Hadoop cluster EC2 Your Hadoop Cluster 7. Clean up!

23. New Challenges  Interference between co-hosted VMs  Slow down the job 1.5-7 times  Locality preserving policy no long effective  Lose more than 20% locality (depends)  Need specifically designed scheduler for virtual MapReduce cluster  Interference-aware  Locality-aware 23

24. MapReduce Programming  Hadoop implementation of MR in Java (version 1.0.4)  WordCount example: hadoop- 1.0.4/src/examples/org/apache/hadoop/examples/WordCount.java 24

25. MapReduce Programming 25

26. Map  Implement your own map class extending the Mapper class 26

27. Reduce  Implement your own reducer class extending the reducer class 27

28. Main() 28

29. Demo 29