1. Lecture 16 (Intro to MapReduce and Hadoop)
CSE 482: Big Data Analysis
Lecture 16 (Intro to MapReduce and Hadoop)

2. Outline of Today’s Lecture
What is Hadoop and MapReduce?
Why use Hadoop/MapReduce?
What is Hadoop distributed file system (HDFS)?
How to use HDFS?
How to run a MapReduce job on Hadoop cluster?

3. MapReduce versus Hadoop
MapReduce is a distributed programming model for writing and executing applications that require processing massive amounts of data
Originally developed at Google
Hadoop is its open-source implementation, whose development was led by Yahoo Research (and is now part of the Apache project)
This class will focus on the Hadoop implementation

4. Why use Hadoop/MapReduce?
Big data applications require huge computational resources beyond those provided by a single machine
However, writing programs that run on multiple machines can be daunting
Hadoop/MapReduce provides a computing environment that simplifies the process of writing distributed programs

5. Advantages of Hadoop/MapReduce
Accessible: runs on large clusters of commodity (PC) machines (instead of expensive parallel machines)
Robust: can gracefully handle hardware failures
Scalable: scales linearly to handle larger data by adding more nodes to the cluster
Simple: allows users to quickly write efficient parallel code (that can be executed on a single machine or a cluster of machines)

6. Key Concepts in MapReduce/Hadoop
Hadoop architecture includes both distributed storage and distributed computation
Distributed storage is managed by the Hadoop Distributed File System (HDFS)
Data is split into smaller blocks that are distributed and replicated across multiple machines in the cluster
Data are stored as “key-value” pairs instead of relational tables

7. Key Concepts in MapReduce/Hadoop
“Scale-out” rather than “Scale-up” architecture
Instead of using expensive high-end servers to process the massive data, Hadoop is designed to run on a cluster of commodity PC machines
Move code to data (instead of data to code)
Offline batch processing instead of online transaction processing (OLTP)
although, there has been some recent development for stream processing – Apache Storm

8. Hadoop Architecture
Hadoop employs a master/slave architecture for its distributed storage and computation
When you launch Hadoop, the following “daemons” (resident programs) will run in the background
For distributed storage
Master node has a NameNode daemon
Each slave node has a DataNode daemon
For distributed computation
Master node launches a Jobtracker daemon
Each slave node launches a Tasktracker daemon

9. NameNode and DataNodes
The distributed storage system is called Hadoop File System (HDFS)
Each file is divided into blocks (the default size of each block is 64MB), stored and replicated on multiple machines
NameNode is the master of HDFS that directs the slave DataNode to perform low-level I/O tasks
When your client program needs to read/write a HDFS file, the NameNode will tell your client which DataNode each block resides; your client then communicates directly with the DataNode (this is done automatically without the knowledge of the programmer)

10. NameNode and DataNodes

11. JobTracker and TaskTracker
JobTracker serves as a liason between your client program and Hadoop
When you submit a job, the JobTracker determines the execution plan (which files to process, how to assign nodes to different tasks, and monitors for completion of the tasks)
If a task fail, JobTracker will automatically relaunch the task, possibly on a different node, up to a predefined limit of retries
TaskTracker manages the execution of individual tasks on each slave node

12. JobTracker and TaskTracker

13. Hadoop Distributed File System (HDFS)
Important: Before you run any Hadoop jobs, you must first upload your data set to HDFS
After the job has completed, you can copy the results from HDFS to your local filesystem
Thus, when executing a Hadoop program, you must be able to distinguish between the local filesystem and the Hadoop filesystem (HDFS)

14. Basic HDFS commands Syntax: To list files in current directory:
To view the subdirectories:
To make a directory:
To copy a file from HDFS to local filesystem
To copy a file from local filesystem to HDFS

15. Basic HDFS Commands To view a file
If file is too large, can pipe it to a Linux command
To display the last kilobyte of the file
To delete a file
To remove a directory

16. Hadoop Programs
Next, we will describe the basics of Hadoop / MapReduce programs
How do the Hadoop/MapReduce programs differ from standard programs?
An example of running a hadoop job will also be illustrated

17. Motivating Example
Consider the problem of finding “hot items” in a data stream of tokens
Example: find the most frequent words that appear in twitter messages
Imagine you write a program to do this on a single machine
Keep a counter for each unique term
Increment the counter associated with each term that appears in the data stream

18. WordCount Example
Pseudocode (serial algorithm)
NOT SCALABLE

19. WordCount Example
If you have millions of documents, one possibility is to distribute the work over several machines
Pseudocode for a distributed approach
Phase 1:
Phase 2:

20. Distributed WordCount Example
Phase 1
Phase 2

21. Challenges for Distributed WordCount
Bottleneck if all the data files are stored on a single server
Solution is to store the files over many machines
Phase 2 is performed by only one machine
Can we do phase 2 in a distributed fashion?
E.g., Machine 1 processes all words starting from a – i, machine 2 processes words starting from j-r, machine 3 processes words starting from s-z
Need to partition the wordCount and shuffle each partition to different machines so that each machine handles only one partition in phase two

22. Hadoop
Provides an abstraction that hides many of the system-level implementation details so that programmers do not have to worry about
How to partition a large job into smaller tasks that can be assigned to different machines to be executed in parallel during phase 1 and phase 2?
How to divide the data among the machines?
How to coordinate synchronization among the different machines that are working in parallel?
How to collect and combine the partial results from the parallel machines?
How to accomplish all of the above in face of software errors and hardware failures?

23. Hadoop
Hadoop provides built-in Java libraries to support implementation of the distributed computing tasks
Libraries include Java classes for mappers (phase 1) and reducers (phase 2)
Other classes include partitioners and combiners
Programming in Hadoop requires you to decompose a computational problem into mapper and reducer tasks
Once it has been decomposed, scaling the application to use hundreds or thousands of machines is simple (can be done without requiring you to change the code) 23

24. Mappers and Reducers Mappers Reducers
Performs local computation on each machine
Reducers
“Combines” the local results obtained from different mappers to obtain the “global” solution
Input data to mappers and reducers are in the form of <key, value> pairs

25. Hadoop Example

26. Running a Hadoop Job
Read the lecture16b supplementary material on how to create an account on Amazon Web Service and setup your private-public keys
You should apply for AWS Education credit ($35) to run your hadoop code on AWS
After signing in and successfully setting up your account and private-public keys
Launch an AWS EMR cluster
EMR (Elastic MapReduce) cluster already has software needed for this class (hadoop, hive, spark, pig)
Connect to the cluster using SSH

27. Running a Hadoop Job

28. Running a Hadoop Job
Directory that contains hadoop installation on AWS

29. WordCount Example Part of hadoop-examples.jar code
Example: count frequency of terms that appear in tweets from CDCgov

30. WordCount Example
Every line in the tweet.txt file corresponds to a separate tweet message

31. WordCount Example First, you need to locate hadoop-examples.jar file.
Next, you will run the hadoop wordcount program on the tweet.txt file

32. WordCount Example
After the program has been successfully completed, you can visualize the solution in output/part-r-00000

33. Summary
We have introduced the Hadoop/MapReduce distributed computed framework
We have introduced the Hadoop distributed file server (HDFS)
We have illustrated a simple word count example and showed how to run the code on Hadoop cluster on Amazon Web Service (AWS)