1. COMP9313: Big Data Management Lecturer: Xin Cao Course web site: http://www.cse.unsw.edu.au/~cs9313/

2. Chapter 4: HDFS and Hadoop I/O

3. Part 1: HDFS Introduction

4. File System
A filesystem is the methods and data structures that an operating system uses to keep track of files on a disk or partition; that is, the way the files are organized on the disk.

5. How to Move Data to Workers?
In many traditional cluster architectures, storage is viewed as a distinct and separate component from computation.
NAS
SAN
Compute Nodes
What’s the problem here?
As dataset sizes increase, the link between the compute nodes and the storage becomes a bottleneck!

6. Distributed File System
Don’t move data to workers… move workers to the data!
Store data on the local disks of nodes in the cluster
Start up the workers on the node that has the data local
Why?
Not enough RAM to hold all the data in memory
Disk access is slow (low-latency), but disk throughput is reasonable (high throughput)
A distributed file system is the answer
A distributed file system is a client/server-based application that allows clients to access and process data stored on the server as if it were on their own computer
GFS (Google File System) for Google’s MapReduce
HDFS (Hadoop Distributed File System) for Hadoop

7. Latency and Throughput
Latency is the time required to perform some action or to produce some result.
Measured in units of time -- hours, minutes, seconds, nanoseconds or clock periods.
I/O latency: the time that it takes to complete a single I/O.
Throughput is the number of such actions executed or results produced per unit of time.
Measured in units of whatever is being produced (e.g., data) per unit of time.
Disk throughput: the maximum rate of sequential data transfer, measured by Mb/sec etc.

8. GFS: Assumptions Commodity hardware over “exotic” hardware
Scale “out”, not “up”
High component failure rates
Inexpensive commodity components fail all the time
“Modest” number of huge files
Multi-gigabyte files are common, if not encouraged
Files are write-once, mostly appended to
Perhaps concurrently
Large streaming reads over random access
High sustained throughput over low latency

9. GFS: Design Decisions HDFS = GFS clone (same basic ideas)
Files stored as chunks
Fixed size (64MB)
Reliability through replication
Each chunk replicated across 3+ chunkservers
Single master to coordinate access, keep metadata
Simple centralized management
No data caching
Little benefit due to large datasets, streaming reads
Simplify the API
Push some of the issues onto the client
HDFS = GFS clone (same basic ideas)

10. From GFS to HDFS Terminology differences: GFS master = Hadoop Namenode
GFS chunkservers = Hadoop Datanodes
Functional differences:
HDFS performance is (likely) slower

11. Assumptions and Goals of HDFS
Very large datasets
10K nodes, 100 million files, 10PB
Streaming data access
Designed more for batch processing rather than interactive use by users
The emphasis is on high throughput of data access rather than low latency of data access.
Simple coherency model
Built around the idea that the most efficient data processing pattern is a write-once read-many-times pattern
A file once created, written, and closed need not be changed except for appends and truncates
“Moving computation is cheaper than moving data”
Data locations exposed so that computations can move to where data resides

12. Assumptions and Goals of HDFS (Cont’)
Assumes Commodity Hardware
Files are replicated to handle hardware failure
Hardware failure is normal rather than exception. Detect failures and recover from them
Portability across heterogeneous hardware and software platforms
designed to be easily portable from one platform to another
HDFS is not suited for:
Low-latency data access (HBase is a better option)
Lots of small files (NameNodes hold metadata in memory)

13. HDFS Features
The Hadoop Distributed File System (HDFS) is a distributed file system designed to run on commodity hardware. 
Basic Features:
Suitable for applications with large data sets
Streaming access to file system data
High throughput
Can be built out of commodity hardware
Highly fault-tolerant

14. HDFS Architecture
HDFS is a block-structured file system: Files broken into blocks of 64MB or 128MB
A file can be made of several blocks, and they are stored across a cluster of one or more machines with data storage capacity.
Each block of a file is replicated across a number of machines, To prevent loss of data.

15. HDFS Architecture HDFS has a master/slave architecture.
There are two types (and a half) of machines in a HDFS cluster
NameNode: the heart of an HDFS filesystem, it maintains and manages the file system metadata. E.g., what blocks make up a file, and on which datanodes those blocks are stored.
Only one in an HDFS cluster
DataNode: where HDFS stores the actual data. Serves read, write requests, performs block creation, deletion, and replication upon instruction from Namenode
A number of DataNodes usually one per node in a cluster.
A file is split into one or more blocks and set of blocks are stored in DataNodes.
Secondary NameNode: NOT a backup of NameNode!!
Checkpoint node. Periodic merge of Transaction log
Help NameNode start up faster next time

16. HDFS Architecture

17. Functions of a NameNode
Managing the file system namespace:
Maintain the namespace tree operations like opening, closing, and renaming files and directories.
Determine the mapping of file blocks to DataNodes (the physical location of file data).
Store file metadata.
Coordinating file operations:
Directs clients to DataNodes for reads and writes
No data is moved through the NameNode
Maintaining overall health:
Collect block reports and heartbeats from DataNodes
Block re-replication and rebalancing
Garbage collection

18. NameNode Metadata
HDFS keeps the entire namespace in RAM, allowing fast access to the metadata.
4GB of local RAM is sufficient
Types of metadata
List of files
List of Blocks for each file
List of DataNodes for each block
File attributes, e.g. creation time, replication factor
A Transaction Log (EditLog)
Records file creations, file deletions etc

19. Functions of DataNodes
Responsible for serving read and write requests from the file system’s clients.
Perform block creation, deletion, and replication upon instruction from the NameNode.
Periodically sends a report of all existing blocks to the NameNode (Blockreport)
Facilitates Pipelining of Data
Forwards data to other specified DataNodes

20. Communication between NameNode and DataDode
Heartbeats
DataNodes send heartbeats to the NameNode to confirm that the DataNode is operating and the block replicas it hosts are available.
Once every 3 seconds
The NameNode marks DataNodes without recent Heartbeats as dead and does not forward any new IO requests to them
Blockreports
A Blockreport contains a list of all blocks on a DataNode
The Namenode receives a Heartbeat and a BlockReport from each DataNode in the cluster periodically

21. Communication between NameNode and DataDode
TCP – every 3 seconds a Heartbeat
Every 10th heartbeat is a Blockreport
Name Node builds metadata from Blockreports
If Name Node is down, HDFS is down

22. Inside NameNode
FsImage - the snapshot of the filesystem when NameNode started
A master copy of the metadata for the file system
EditLogs - the sequence of changes made to the filesystem after NameNode started

23. Inside NameNode
Only in the restart of NameNode, EditLogs are applied to FsImage to get the latest snapshot of the file system.
But NameNode restart are rare in production clusters which means EditLogs can grow very large for the clusters where NameNode runs for a long period of time.
EditLog become very large , which will be challenging to manage it
NameNode restart takes long time because lot of changes has to be merged
In the case of crash, we will lost huge amount of metadata since FsImage is very old
How to overcome this issue?

24. Secondary NameNode
Secondary NameNode helps to overcome the above issues by taking over responsibility of merging EditLogs with FsImage from the NameNode.
It gets the EditLogs from the NameNode periodically and applies to FsImage
Once it has new FsImage, it copies back to NameNode
NameNode will use this FsImage for the next restart, which will reduce the startup time

25. File System Namespace
Hierarchical file system with directories and files
/user/comp9313
Create, remove, move, rename etc.
NameNode maintains the file system
Any meta information changes to the file system recorded by the NameNode (EditLog).
An application can specify the number of replicas of the file needed: replication factor of the file.

26. Data Replication
The NameNode makes all decisions regarding replication of blocks.

27. HDFS Working Flow … … HDFS namenode Application /foo/bar
(file name, block id)
File namespace
HDFS Client
block 3df2
(block id, block location)
HDFS datanode
Linux file system …
instructions to datanode
HDFS datanode
Linux file system …
datanode state
(block id, byte range)
block data

28. File Read Data Flow in HDFS

29. File Write Data Flow in HDFS

30. Data Pipelining
Client retrieves a list of DataNodes on which to place replicas of a block
Client writes block to the first DataNode
The first DataNode forwards the data to the next node in the Pipeline
When all replicas are written, the Client moves on to write the next block in file

31. Replication Engine NameNode detects DataNode failures
Missing Heartbeats signify lost Nodes
NameNode consults metadata, finds affected data
Chooses new DataNodes for new replicas
Balances disk usage
Balances communication traffic to DataNodes

32. Data Correctness Use Checksums to validate data Use CRC32
File Creation
Client computes checksum per 512 byte
DataNode stores the checksum
File access
Client retrieves the data and checksum from DataNode
If Validation fails, Client tries other replicas

33. Cluster Rebalancing Goal: % disk full on DataNodes should be similar
Usually run when new DataNodes are added
Rebalancer is throttled to avoid network congestion
Does not interfere with MapReduce or HDFS
Command line tool

34. Fault tolerance Failure is the norm rather than exception
A HDFS instance may consist of thousands of server machines, each storing part of the file system’s data.
Since we have huge number of components and that each component has non-trivial probability of failure means that there is always some component that is non-functional.
Detection of faults and quick, automatic recovery from them is a core architectural goal of HDFS.

35. Metadata Disk Failure
FsImage and EditLog are central data structures of HDFS. A corruption of these files can cause a HDFS instance to be non-functional.
A NameNode can be configured to maintain multiple copies of the FsImage and EditLog
Multiple copies of the FsImage and EditLog files are updated synchronously

36. HDFS Configuration HDFS Defaults Block Size – 64 MB
Replication Factor – 3
Web UI Port – 50070
HDFS conf file - /etc/hadoop/conf/hdfs-site.xml
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///data1/cloudera/dfs/nn,file:///data2/cloudera/dfs/nn</value>
</property>
<name>dfs.blocksize</name>
<value> </value>
<name>dfs.replication</name>
<value>3</value>
<name>dfs.namenode.http-address</name>
<value>YOURHTTPURL:50070</value>

37. Application Programming Interface
HDFS provides Java API for application to use.
FSDataInputStream
FSDataOutputStream
FileStatus
… …
Python access is also used in many applications.
A C language wrapper for Java API is also available.
A HTTP browser can be used to browse the files of a HDFS instance

38. Unique features of HDFS
HDFS has a bunch of unique features that make it ideal for distributed systems:
Failure tolerant - data is duplicated across multiple DataNodes to protect against machine failures. The default is a replication factor of 3 (every block is stored on three machines).
Scalability - data transfers happen directly with the DataNodes so your read/write capacity scales fairly well with the number of DataNodes
Space - need more disk space? Just add more DataNodes and re-balance
Industry standard - Other distributed applications are built on top of HDFS (HBase, Map-Reduce)
HDFS is designed to process large data sets with write-once-read-many semantics, it is not for low latency access

39. FS Shell, Admin and Browser Interface
HDFS provides a command line interface called the FS shell that lets the user interact with data in the HDFS.
The syntax of the commands is similar to bash and csh.
$ hdfs dfs: runs filesystem commands on the HDFS
$ hdfs fsck: runs a HDFS filesystem checking command
Example: to create a directory /foodir
$ /bin/hdfs dfs –mkdir /foodir
There is also DFSAdmin interface available
$ hdfs dfsadmin: runs HDFS administration commands
Example: report the status
$ hadoop dfsadmin -report
Browser interface is also available to view the namespace.

40. HDFS DFS Command List

41. Part 2: Hadoop I/O

42. Compression Two major benefits of file compression
Reduce the space needed to store files
Speed up data transfer across the network
When dealing with large volumes of data, both of these savings can be significant, so it pays to carefully consider how to use compression in Hadoop

43. Compression Formats Compression formats “Splittable” column
Indicates whether the compression format supports splitting
Whether you can seek to any point in the stream and start reading from some point further on
Splittable compression formats are especially suitable for MapReduce

44. Compression and Input Splits
When considering how to compress data that will be processed by MapReduce, it is important to understand whether the compression format supports splitting
Example of not-splitable compression problem
A file is a gzip-compressed file whose compressed size is 1 GB
Creating a split for each block won’t work since it is impossible to start reading at an arbitrary point in the gzip stream, and therefore impossible for a map task to read its split independently of the others

45. Codecs
A codec is the implementation of a compression-decompression algorithm
CompressionCodec
createOutputStream(OutputStream out): create a CompressionOutputStream to which you write your uncompressed data to have it written in compressed form to the underlying stream
createInputStream(InputStream in): obtain a CompressionInputStream, which allows you to read uncompressed data from the underlying stream

46. Example
The program get the codec from the argument and compress the input from stdin
public class StreamCompressor {
public static void main(String[] args) throws Exception {
String codecClassname = args[0];
Class<?> codecClass = Class.forName(codecClassname);
Configuration conf = new Configuration();
CompressionCodec codec = (CompressionCodec)
ReflectionUtils.newInstance(codecClass, conf);
CompressionOutputStream out = codec.createOutputStream(System.out);
IOUtils.copyBytes(System.in, out, 4096, false);
out.finish(); }
% echo "Text" | hadoop StreamCompressor org.apache.hadoop.io.compress.GzipCodec \
| gunzip -
Text

47. Use Compression in MapReduce
If the input files are compressed, they will be decompressed automatically as they are read by MapReduce, using the filename extension to determine which codec to use.
We do not need to do anything unless to guarantee that the filename extension is correct
In order to compress the output of a MapReduce job, you need to:
Set the mapreduce.output.fileoutputformat.compress property to true
Set the mapreduce.output.fileoutputformat.compress.codec property to the classname of the compression codec
Use the static method of FileOutputFormat to set these properties

48. Use Compression in MapReduce
In WordCount.java, you can do:
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "word count");
job.setJarByClass(WordCount.class);
job.setMapperClass(TokenizerMapper.class);
job.setReducerClass(IntSumReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
FileOutputFormat.setCompressOutput(job, true);
FileOutputFormat.setOutputCompressorClass(job, GzipCodec.class);
System.exit(job.waitForCompletion(true) ? 0 : 1); }

49. Serialization
Process of turning structured objects into a byte stream for transmission over a network or for writing to persistent storage
Deserialization is the reverse process of serialization
Requirements
Compact
To make efficient use of storage space
Fast
The overhead in reading and writing of data is minimal
Extensible
We can transparently read data written in an older format
Interoperable
We can read or write persistent data using different language

50. Writable Interface
Hadoop defines its own “box” classes for strings (Text), integers (IntWritable), etc.
Writable is a serializable object which implements a simple, efficient, serialization protocol
All values are instances of Writable
All keys are instances of WritableComparable
context.write(WritableComparable, Writable)
You cannot use java primitives here!!
public interface Writable {
void write(DataOutput out) throws IOException;
void readFields(DataInput in) throws IOException; }

52. Writable Wrappers for Java Primitives
There are Writable wrappers for all the Java primitive types except shot and char (both of which can be stored in an IntWritable)
get() for retrieving and set() for storing the wrapped value
Variable-length formats
If a value is between -122 and 127, use only a single byte
Otherwise, use first byte to indicate whether the value is positive or negative and how many bytes follow

53. Writable Examples Text Writable for UTF-8 sequences
Can be thought of as the Writable equivalent of java.lang.String
Maximum size is 2GB
Use standard UTF-8
Text is mutable (like all Writable implementations, except NullWritable)
Different from java.lang.String
You can reuse a Text instance by calling one of the set() method
NullWritable
Zero-length serialization
Used as a placeholder
A key or a value can be declared as a NullWritable when you don’t need to use that position

54. Multiple Output Values
If we are to output multiple values for each key
E.g., a pair of String objects, or a pair of int
How do we do that?
WordCount output a single number as the value
Remember, our object containing the values needs to implement the Writable interface
We could use Text
Value is a string of comma separated values
Have to convert the values to strings, build the full string
Have to parse the string on input (not hard) to get the values

55. Implement a Custom Writable
Suppose we wanted to implement a custom class containing a pair of integers. Call it IntPair.
How would we implement this class?
Needs to implement the Writable interface
Instance variables to hold the values
Construct functions
A method to set the values (two integers)
A method to get the values (two integers)
write() method： serialize the member variables (two integers) objects in turn to the output stream
readFields() method: deserialize the member variables (two integers) in turn from the input stream
As in Java: hashCode(), equals(), toString()

56. Implement a Custom Writable
implement the Writable interface
Instance variables to hold the values
Construct functions
set() method
public class IntPair implements Writable {
private int first, second;
public IntPair() { }
public IntPair(int first, int second) {
set(first, second);
public void set(int left, int right) {
first = left;
second = right; }

57. Implement a Custom Writable
get() method
write() method
Write the two integers to the output stream in turn
readFields() method
Read the two integers from the input stream in turn
public int getFirst() {
return first; }
public int getSecond() {
return second;
public void write(DataOutput out) throws IOException {
out.writeInt(first);
out.writeInt(second); }
public void readFields(DataInput in) throws IOException {
first = in.readInt();
second = in.readInt(); }

58. Complex Key If the key is not a single value
E.g., a pair of String objects, or a pair of int
How do we do that?
The co-occurrence matrix problem, a pair of terms as the key
Our object containing the values needs to implement the WritableComparable interface
Why Writable is not competent?
We could use Text again
Value is a string of comma separated values
Have to convert the values to strings, build the full string
Have to parse the string on input (not hard) to get the values
Objects are compared according to the full string!!

59. Implement a Custom WritableComparable
Suppose we wanted to implement a custom class containing a pair of String objects. Call it StringPair.
How would we implement this class?
Needs to implement the WritableComparable interface
Instance variables to hold the values
Construct functions
A method to set the values (two String objects)
A method to get the values (two String objects)
write() method： serialize the member variables (i.e., two String) objects in turn to the output stream
readFields() method: deserialize the member variables (i.e., two String) in turn from the input stream
As in Java: hashCode(), equals(), toString()
compareTo() method: specify how to compare two objects of the self-defind class

60. Implement a Custom WritableComparable
implement the Writable interface
Instance variables to hold the values
Construct functions
set() method
public class StringPair implements WritableComparable<StringPair> {
private String first, second;
public StringPair() { }
public StringPair(String first, String second) {
set(first, second);
public void set(String left, String right) {
first = left;
second = right; }

61. Implement a Custom WritableComparable
get() method
write() method
Utilize WritableUtils.
readFields() method
public String getFirst() {
return first; }
public String getSecond() {
return second;
public void write(DataOutput out) throws IOException {
String[] strings = new String[] { first, second };
WritableUtils.writeStringArray(out, strings); }
public void readFields(DataInput in) throws IOException {
String[] strings = WritableUtils.readStringArray(in);
first = strings[0];
second = strings[1]; }

62. Implement a Custom WritableComparable
compareTo() method:
public int compareTo(StringPair o) {
      int cmp = compare(first, o.getFirst());
      if(cmp != 0){
          return cmp;
      }
      return compare(second, o.getSecond()); }
private int compare(String s1, String s2){
      if (s1 == null && s2 != null) {
          return -1;
      } else if (s1 != null && s2 == null) {
          return 1;
      } else if (s1 == null && s2 == null) {
          return 0;
      } else {
          return s1.compareTo(s2);

63. Implement a Custom WritableComparable
You can also make the member variables as Writable objects
Instance variables to hold the values
Construct functions
set() method
private Text first, second;
public StringPair() {
set(new Text(), new Text()); }
public StringPair(Text first, Text second) {
set(first, second);
public void set(Text left, Text right) {
first = left;
second = right; }

64. Implement a Custom WritableComparable
get() method
write() method
Delegated to Text
readFields() method
public Text getFirst() {
return first; }
public Text getSecond() {
return second;
public void write(DataOutput out) throws IOException {
first.write(out);
second.write(out); }
public void readFields(DataInput in) throws IOException {
first.readFields(in);
second.readFields(in); }

65. Implement a Custom WritableComparable
In some cases such as secondary sort, we also need to override the hashCode() method.
Because we need to make sure that all key-value pairs associated with the first part of the key are sent to the same reducer!
By doing this, partitioner will only use the hashCode of the first part.
You can also write a paritioner to do this job
public int hashCode()
return first.hashCode(); }

66. Implement a Custom Partitioner
A customized partitioner extends the Partitioner class
The key and value are the intermediate key and value produced by the map function
In the co-occurrence matrix problem
It overrides the getPartition function, which has three parameters
The numReduceTasks is the number of reducers used in the MapReduce program and it is specified in the driver program (by default 1)
public static class YourPatitioner extends Partitioner<Key, Value>{
public static class FirstPatitioner extends Partitioner<StringPair, IntWritable>{
public int getPartition(WritableComparable key, Writable value, int numPartitions)
public int getPartition(StringPair key, IntWritable value, int numPartitions){
          return (key.getFirst().hashCode() & Integer.MAX_VALUE) % numPartitions;
      }

67. Number of Maps and Reduces
The number of maps is usually driven by the total size of the inputs, that is, the total number of blocks of the input files.
The right level of parallelism for maps seems to be around maps per-node, although it has been set up to 300 maps for very cpu-light map tasks.
If you expect 10TB of input data and have a blocksize of 128MB, you’ll end up with 82,000 maps, unless Configuration.set(MRJobConfig.NUM_MAPS, int) (which only provides a hint to the framework) is used to set it even higher.
Reduces
The right number of reduces seems to be 0.95 or 1.75 multiplied by (<no. of nodes> * <no. of maximum containers per node>)
With 0.95 all of the reduces can launch immediately and start transferring map outputs as the maps finish. With 1.75 the faster nodes will finish their first round of reduces and launch a second wave of reduces doing a much better job of load balancing.
Use job.setNumReduceTasks(int) to set the number

68. Lifecycle of Mapper/Reducer
Lifecycle: setup -> map -> cleanup
setup(): called once at the beginning of the task
map(): do the map
cleanup(): called once at the end of the task.
We do not invoke these functions

69. About the First Assignment
Problem setting
Ignore terms starting with non-alphabetical characters
Convert all terms to lower case
The length of the term is obtained by the length() function of String
Length of “text234sdf” is 10
Use the tokenizer give in Lab 3
Example input and output will be given
In-mapper combining
Number of mappers and reducers
Submission
Two java files
Two ways
Deadline: 29 Aug 2016, 09:59:59 am

70. References
HDFS design:
Hadoop The Definitive Guide. HDFS and Hadoop I/O chapters.
Understanding Hadoop Clusters and the Network. By Brad Hedlund.

71. End of Chapter 4