Design of Cloud Management Layer for High-Performance File Transfer 高效能檔案傳輸之雲端層設計 1

Outline Introduction Background System Design Implementation Experiments 2

Introduction Motivation – File Transfer with load sharing and fault tolerance. 3

Introduction User (Client) Management Layer Cluster Storage Server Upload Files Clients can upload the files to the cloud, and promise your file cannot be lose. Files Download 4

Introduction (Cont.) Surveillance Application Size: 0.2~0.5 GB/1hr Storage:2TB Device x 30 Host 2000G(2T) / 30 = 67G 67G / 0.3G = 223h 223h / 24h = 9.3 day 5

Introduction (Cont.) Used Platforms and API  Hadoop Hadoop Distributed File System (HDFS) – File Storage Hbase – User Authentication  Socket File Data – Transmissions 6

Outline Introduction Background System Design Implementation Experiments 7

Background (Cont.) What is Cloud Computing? Scalable Computing and Storage Resource. Cloud Source Models 8

Background (Cont.) Hadoop DataBase File System PigDataflow language and parallel execution Framework HiveData warehouse infrastructure ZooKeeperDistributed coordination service ChukwaSystem for collecting management data AvroData serialization system The other components of Hadoop Parallel Processing 9

Background (Cont.) What is “Hadoop Distributed File System (HDFS)”? Cluster Storage Server Datanode PC Clients Namenode Transmission 10

Background (Cont.) 11 Hbase have high availability, high performance, and high expansion flexibility. Table: ‘t1’ row-key 'f1''f2''fn'column-family ‘c1’‘c2’‘c3’‘c4’*column-quantifier r1 v1 v2 r2 v3 v4

Outline Introduction Background System Design Implementation Experiments Conclusions Reference 12

System Design(Cont.) Components – Manager Selection – Authentication – Synchronization – Cloud-Based File Transfer 13

System Design(Cont.) Manager Selection 1.Connect Management layer and perform load balance 2.Collect memory load information from each nodes 3.Select the lowest memory load as connection port(Node i ) 4.Connect with Node i 5.Authentication 14

System Design(Cont.) Authentication 1.Key in account / password 2.Authentication(Hbase verification) 3.Synchronization 15

System Design(Cont.) Synchronization 1.HDFS files synchronization in client side 2.Check files existence 3.Cloud-Based files transfer 4.Background Monitoring 16

System Design(Cont.) Synchronization 1.HDFS files synchronization in client side 2.Check files existence 3.Cloud-Based files transfer 4.Background Monitoring 17

System Design(Cont.) Cloud-Based File Transfer 1.Send request using socket 2.Analysis The packet (to extract the command and path) 3.Execute command (Write/Send/Delete) 18

Outline Introduction Background System Design Implementation Experiments 19

Implementation Virtualization software – VMware WorkStation 8.0 OS Environment – Ubuntu Linux 、 Windows Development tools – Ubuntu Linux : C/C++ 、 Java – Windows : Borland C++ builder 20

Implementation(Cont.) Client & Management Layer – C++ Socket API Management Layer – Java Hbase API – C/C++ HDFS API JNI (Java Native Interface) 21

Outline Introduction Background System Design Experiments Conclusions Reference 22

Experiments Manager Selection – 隨機配置 – 負載平衡配置 選擇負載最小之節點： 5 個 更新時間： 10 秒 可同時連線數量： 10 個 Environmental Environment 1 – Servers: 50, 100, 150, 200, 250 – Clients : 2000 Environmental Environment 2 – Servers: 50 – Clients : 1000, 2000, 3000, 4000,

Experiments(Cont.) Experiment 1 – Servers: 50, 100, 150, 200, 250 – Clients :

Experiments(Cont.) Experiment 2 – Servers: 50 – Clients : 1000, 2000, 3000, 4000,