1. The IEEE International Conference on Cluster Computing 2010
CDRM: A Cost-effective Dynamic Replication Management Scheme for Cloud Storage Cluster
Qingsong Wei
Data Storage Institute, A-STAR, Singapore
Bharadwaj Veeravalli, Bozhao Gong
National University of Singapore
Lingfang Zeng, Dan Feng
Huazhong University of Science & Technology, China

2. Cost-effective Dynamic Replication Management (CDRM)
Agenda
Outline
Introduction
Problem Statement
Cost-effective Dynamic Replication Management (CDRM)
Evaluation
Conclusion

3. 1, Introduction Outline HDFS Architecture Clients Name Node Network
Meta Data
Name Node
Network
Control
Node
Disk
Disks
Node
Disk
Disks
Node
Disk
Disks
Node
Disk
Disks
Node
Disk
Disks
Node
Disk
Disks
Data Blocks
Data Nodes

4. 1. Introduction
In the HDFS, files are striped into date blocks across multiple data nodes to enable parallel access. B1 B2 … Bm
Data
Striping
However, Block may be unaccessible due to date node unavailable. If one of the blocks is unavailable, so as the whole file.
Failure is normal instead of exception in large scale storage cloud system. Fault tolerance is required in such a system. …
Node1
Node2
Noden
Page 4 of 19

5. 1. Introduction Data nodes Clients 1 1 2 5 2 2 4 3 4 5 4 3 5
Replication is used in HDFS.
When one data node fails, the data is still accessible from the replicas and storage service need not be interrupted.
Besides fault tolerance, replicas among data nodes can be used to balance workload.
Page 5 of 19 5

6. 2. Problem Statement Current replication managements
Treat all data as same: same replica number for all data
Treat all storage nodes as same
Fixed and Static 1 2 3 4 5 5 1 2 3 4 4 5 1 2 3
High cost & Poor load balance
Page 6 of 19 6

7. 2. Problem Statement
Replica number is critical to management cost. More replica, more cost. 1 2 1 5 2 2 4
The block 5 is modified 3 5 4 4 3 5
Update to maintain consistency
Because large number of blocks are stored in system, even a small increase of replica number can result in a significant increase of management cost in the overall system.
Then, how many minimal replica should be kept in the system to satisfy availability requirement?
Page 7 of 19 7

8. 2. Problem Statement
Replica placement influences intra-request parallelism.
Client
File (B1, B2, B3)
B3 B2 B1
Requests
Blocked
If B1 has a replica in Datanode 2 which has free sessions, the access can be immediately serviced. B3 B2 B1 B1
Data Node1
Data Node2
Data Node3
Sessionmax=3
Sessionfree=1
Sessionmax=3
Sessionfree=2
Sessionmax=2
Sessionfree=0
Page 8 of 19

9. 2. Problem Statement
Replica placement also influences inter-request parallelism.
Client1
Client2
B3 B2 B1 B1
Requests
How to place these replicas among Data nodes clusters in a balance way to improve access parallelism? B1 B2 B1 B3
Data Node1
Data Node2
Data Node3
Sessionmax=3
Sessionfree=0
Sessionmax=3
Sessionfree=1
Sessionmax=2
Sessionfree=0
Page 9 of 19

10. 3. Cost-effective Dynamic Replication Management
System Model
pj : popularity
sj : size
rj : replica number
tj : access latency requirement
(p1, s1, r1, t1)
(pj, sj, rj, tj)
(pM, sM, rM, tM) …… …… B1 Bj BM
Total arrival rate: λ
λi : req. arr. rate
τi : average ser. time
fi : failure rate
ci : max sessions
Node1 …
Nodei …
NodeN
Two key issues to be addressed
1, Minimal replica number
2, How to place these replicas in balanced way
(λ1, τ1, f1, c1)
(λi, τi, fi, ci)
(λN, τN, fN, cN)
Data has different attributes
Data nodes are different
Page 10 of 19

11. 3. Cost-effective Dynamic Replication Management
Availability
Suppose file F is striped into m blocks {b1 , b2 ,…, bm}. To retrieve whole file F, we must get all the m blocks.
Availability is modeled as function of replica number.
Suppose the expected availability for file F is Aexpect, which defined by users. To satisfy the availability requirement for a given file, we get
Minimum replicas can be calculated from above Eq. for a given expected availability.
Page 11 of 19

12. 3. Cost-effective Dynamic Replication Management
Blocking Probability
Blocking probability is used as criterion to place replicas among data nodes to improve load balance .
An data node Si is modeled as M/G/ci system with arrival rate λi and service time τi, and accordingly, the blocking probability of data node Si can be given to be
Replica placement policy: replica will be placed into data node with lowest blocking probability to dynamically maintain overall load balancing.
After determining how many replicas the system should maintain at least to satisfy availability requirement, we shall explain how to place these replicas efficiently to maximize performance and load balancing.
Blocking probability describe the probability of data node to reject a request.
Page 12 of 19

13. Framework of cost-effective dynamic replication management in HDFS
Request to create a file with <Availability, Block Number> 1
Client
Name Node Bm … B2 B1
Return replication policy <Bi, Replication factor, DataNode list> 3 4
Flush and replicate blocks to selected Datanodes
Calculate the replication factor and Search the Datanode B+Tree to obtain Datanode list. 2
Data Nodes
Besides default replication management, HDFS provides flexible API for us to extend and implement efficient replication management.
Replication Pipelining
Framework of cost-effective dynamic replication management in HDFS
Page 13 of 19

14. 4. Evaluation
Setup
Our test platform is built on a cluster with one name node and twenty data nodes of commodity computer
The operating system is Red Hat AS4.4 with kernel
Hadoop version is and java version is
AUSPEX file system trace is used
A synthesizer is developed to create workloads with different characteristics, such as data sets of different sizes, varying data rates, and different popularities. These characteristics reflect the differences among various workloads to the cloud storage cluster.
Page 14 of 19

15. 4. Evaluation Cost effective Availability
Initially, one replica per object.
CDRM only maintain minimal replicas to satisfy availability.
Higher failure rate, more replica required.
Dynamic replication with Data node failure rate of 0.1 and 0.2 , Aexpect=0.8
Page 15 of 19

16. Effect of popularity and access arrival rate, 20 data nodes
4. Evaluation
Performance
CDRM vs. HDFS default Replication Management (HDRM) under different popularity and workload intensity.
Performance of CDRM is much better than that of HDRM when popularity is small.
CDRM outperform HDRM under heavy workload.
popularity is a ratio between the size of the most popular data and the size of the total data set. A large popularity indicates the data are more uniformly accessed. A small popularity means accesses concentrate in a small amount of data.
Effect of popularity and access arrival rate, 20 data nodes
Page 16 of 19

17. System utilization among data nodes, popularity=10%, λ=0.6
4. Evaluation
Load Balance
The figure shows the difference of system utilization of each data node comparing to the average system utilization of the cluster.
CDRM can dynamically distribute workload among whole cluster.
System utilization among data nodes, popularity=10%, λ=0.6
Page 17 of 19

18. Current replication management policies
5. Conclusion
Current replication management policies
CDRM
Data is same
Data is different
Storage node are same
Storage nodes are different
Same replica number for all data
Different replica number for different data
Static placement
Dynamic placement
High Cost
Cost effective
Poor load balance
Good balance
Low performance
High performance
Page 18 of 19

19. Thanks & Question
For more questions, please contact Dr. Qingsong Wei by
Page 19 of 19