1 An Adaptive File Distribution Algorithm for Wide Area Network Takashi Hoshino, Kenjiro Taura, Takashi Chikayama University of Tokyo

2 Background New environments for parallel and distributed computation  Clusters, cluster of clusters, GRID Offer scalability and good cost performance Setting up computation in such environments is complex, however  Install programs/data

3 Setting up computation in DS Often involves copying large programs/data to many nodes Manually copying large files is troublesome because:  faults occur easily  firewalls block (some) connections  transfers must be scheduled carefully for good performance

4 Contribution NetSync  A file replicator optimized for copying large data to many nodes in parallel (application-level) Features  Automatic load-balancing scalability  Self-stabilizing construction of transfer route fault-tolerant  Adaptive optimization of transfer route  No reliance on physical topology information

5 Outline What are efficient/inefficient transfer routes? Demo Algorithm  Base algorithm  Adaptive optimization Implementation Experiments Related work Summary and future work

6 Inefficient Transfer Routes Many inter-subnet/cluster transfer connections Many branches Node Subnet/cluster Data transfer line

7 What’s Wrong with Branches? Branches  share hardware capability of nodes themselves CPU power Disk performance NIC ability  enlarge possibilities of bottleneck CPU NIC DISK CPU NIC DISK No bottleneckBottleneck One childThree children 100Mbps x133Mbps x3

8 Efficient Transfer Route Minimum inter-subnet/cluster transfer connections No or minimum branches Node Subnet/cluster Data transfer line

9 Demo Playback of our experiment using logs A00 A01 B00A07 A06 A05 A04 A03 A02 B07 B06 B05 B04 B03 B02 B01 CXX Node Data flow (Parent-Child) A00 A01 B00 A07 A06 A05 A04 A03 A02 B07 B06 B05 B04 B03 B02 B01

10 System Overview A.dat(1GB) User Order(A.dat,1GB) A.dat(1GB)

11 Algorithm Simple base algorithm  Fault-tolerance, scalability, self-stabilization Add-on adaptive optimization heuristics  Well-adapted today’s typical network Very easy configuration  Only need information of (some) neighbors  Need no physical topology  Need no performance measurement Pseudo-code is described in our paper

12 Base Algorithm (1) Each node seeks a node to be its parent Pipeline transfer in whole nodes Fault leads to seeking new parent again 100 % 0% 25 % 50 % 25 % 50 % 75 % 50 % 75 % 50 % 75 % 100 % 75 % 100 %

13 Base Algorithm(2) Pseudo code (simplified) while(not has complete data) parent doesn’t exist  seek candidate if found candidate then ask candidate if it can be the parent OK  start to get data from the parent NG  seek candidate again end parent timed out  seek candidate again end

14 Base Algorithm (2) Child (has not its parent) side send ASK to candidate to be its parent recv OK  start getting data recv NG  seeks candidate again Parent (received ask message) side recv ASK from a node  if my offset > node’s offset and # of children < LIMIT_CHILDREN then send OK and start putting data else send NG end

15 Adaptive Optimization Two heuristics  NearParent  Tree2List

16 NearParent Heuristics NearParent: reduce "long" connections  Each node changes its parent to a closer node parent candidate self candidate parent

17 Tree2List Heuristics Tree2List: reduce branches  If the current parent is not closer than one of its siblings X, change its parent to X  A node which has more than one children suggests its children to change their parent to one of their siblings self X parent X self

18 How to measure closeness? Features  Throughput  Latency  Prefix of IP address A B C

19 Property of Heuristics (1) Assuming there is no firewall… 1. Minimum inter-cluster/subnet connections 2. All nodes connect each other as a list subnet/cluster

20 Property of Heuristics (2) If firewall blocks some connections… 1. Minimum inter-cluster/subnet connections 2.  N – 1 branches for N subnets (assume no firewalls inside a subnet) subnet/cluster Firewall

21 Property of Heuristics (2) If there is no firewall  Distribution tree becomes MST  Minimum inter-group connections with any scale  All nodes connect each other as a list subnet cluster

22 Property of Heuristics (3) Firewall subnet cluster If multiple levels of groups exist (subnets, clusters), it optimizes all levels simultaneously  Minimum inter-subnet edges  Minimum inter-cluster edges

23 Property of Heuristics (3) If multiple levels of groups exist (subnets, clusters), it optimizes all levels simultaneously  Minimum inter-subnet edges  Minimum inter-cluster edges subnet cluster

24 Implementation File replicator for a large data and many nodes written in Java Ability of detecting latency: about 1ms Usage:  Install and run NetSync in all nodes  Throw a file information to several nodes  Wait for finishing the replication Very simple usage!!!

25 Experiments Measure performance of our heuristics Distributed a file to many nodes  Compared completion time Environments  A single cluster  Multiple clusters

26 Experiment in a single cluster (1) Distributed 500MB from one node to other 16nodes in the cluster  Only NIC (100Mbps) can be bottleneck Compared two settings  Random Tree Only using base algorithm Limited # of children from 1 to 5.  Tree2List NearParent has no effect

27 Experiment in a single cluster (2) Fewer children, better performance Tree2List is very close to optimal Limit 1 is not scalable (using our base algorithm)

28 Experiment in multiple clusters (1) Distributed 300MB to over 150 nodes in seven clusters Heuristics on, off, and fixed manually optimized tree 1G1G 100M 1G1G 1G1G

29 Experiment in multiple clusters (2) Our heuristics is close to the ideal fixed tree

30 Related Work Application-level Multicast  Overcast[Jannotti], ALMI[Pendarakis], etc.  Aims to optimize bandwidth and latency Content Distribution Network (CDN)  Has roots in HTTP accelerator and HTTP proxy.  Aims to optimize latency and load-balancing. Our approach  Maximize throughput, even if sacrificing latency

31 Summary and Future Work We designed a simple algorithm  for copying large data to many nodes in parallel  with fault-tolerance, scalability, self- organization, and adaptive optimization Evaluations show our implementation is effective in real environment Future Work  Integration with searching for contents, or storage systems for distributed computing