1. HDFS Hadoop Distributed File System
柯懷貿
王建鑫
彭偉慶

2. Outline
Introduction
HDFS – How it works
Pros and Cons
Conclusion
柯懷貿

3. Introduction to HDFS Hadoop Distributed File System Cloud Computing
JAVA
Processing PB-Level Data
Distributed Computing Environment
Hadoop MapReduce
HDFS
HBase
Allow files shared via internet
Write-once-read-many
Restricting access
Replication & Fault tolerance
Mapping between logical objects & physical objects
Dung Cutting established
Nutch Project
File System for Hadoop framework
Remote Procedure Call
Master/Slave
Yahoo! has accomplished 10,000-core Hadoop cluster in 2008
柯懷貿

4. MapReduce
柯懷貿

5. HBase
NoSQL
Using several servers to store PB-level data
柯懷貿

6. HDFS Distributed, scalable, and portable File replication(default : 3)
Reading efficacy
柯懷貿

7. 王建鑫

8. HDFS major roles Client(user) – read/write data from/to file system
Name node(masters) – oversee and coordinate the data storage function, receive instructions from Client
Data node(slaves) – store data and run computations, receive instructions from Namenode
王建鑫

9. 王建鑫

10. 王建鑫

11. Rack Awareness
王建鑫

12. 王建鑫

13. 王建鑫

14. 王建鑫

15. 王建鑫

16. 王建鑫

17. 王建鑫

18. HDFS fault tolerance Node failure – data node or nam enode is dead
Communication failure – cannot send and retrieve data
Data corruption – data corrupted while sending over network or corrupted in the hard disks
Write failure – the data node which is ready to be written is dead
Read failure - the data node which is ready to be read is dead
王建鑫

19. 王建鑫

20. Detect the Network failure
Whenever data is sent, an ACK is replied by the receiver
If the ACK is not received(after several retries), the sender assumes that the host is dead, or the network has failed
Also Checksum is sent along with transmitted data→can detect corrupt data when transferring
王建鑫

21. Handling the write/read failure
Client write the block in smaller data units(usually 64KB) called packet
Each data node replies back an ACK for each packet to confirm that they got the packet
If client don’t get the ACKs from some nodes, dead node detected
Client then adjust the pipeline to skip that node(then?)
Handling the read failure：just read another node
王建鑫

22. Handling the write failure cont’d
Name node contains two tables:
List of blocks – blockA in dn1, dn2,dn8； blockB in dn3, dn7, dn9…
List of Data nodes – dn1 has blockA, blockD； dn2 has blockE, blockG…
Name node check list of blocks to see if a block is not properly replicated
If so, ask other data nodes to copy block from data nodes that have the replication.
王建鑫

23. Pros Very large files Streaming data access Commodity hardware
A file size overs xxxMB, GB, TB, PB .…..
Streaming data access
Write-once, read-many.
Efficient on reading whole dataset.
Commodity hardware
High reliability and availability.
Doesn’t require expensive, highly reliable hardware.
彭偉慶

24. Cons
彭偉慶

25. Conclusion HDFS - an Apache Hadoop subproject.
Highly fault-tolerant and is designed to be deployed on low-cost hardware.
High throughput but not low latency.
彭偉慶