A Stable Broadcast Algorithm Kei Takahashi Hideo Saito Takeshi Shibata Kenjiro Taura (The University of Tokyo, Japan) 1 CCGrid Lyon, France

 To distribute the same, but large data to many nodes Ex: content delivery  Widely used in parallel processing 2 Broadcasting Large Messages Data Kei Takahashi, Hideo Saito, Takeshi Shibata and Kenjiro Taura

 Usually, in a broadcast transfer, the source can deliver much less data than a single transfer from the source 3 Problem of Broadcast S D S D Kei Takahashi, Hideo Saito, Takeshi Shibata and Kenjiro Taura

 Pipeline-manner transfers improve the performance  Even in a pipeline transfer, nodes with small bandwidth (slow nodes) may degrade receiving bandwidth of all other nodes 4 Problem of Slow Nodes  

1.Propose an idea of Stable Broadcast In a stable broadcast: Slow nodes never degrade receiving bandwidth to other nodes All nodes receive the maximum possible amount of data Contributions 5

2.Propose a stable broadcast algorithm for tree topologies Proved to be stable in a theoretical model Improve performances in general graph networks 3.In a real-machine experiment, our algorithm achieved 2.5 times the aggregate bandwidth than the previous algorithm (FPFR) Contributions (cont.) 6

 Introduction  Problem Settings  Related Work  Proposed Algorithm  Evaluation  Conclusion 7 Agenda Kei Takahashi, Hideo Saito, Takeshi Shibata and Kenjiro Taura

1.Target: large message broadcast 2.Only computational nodes handle messages 8 Problem Settings Kei Takahashi, Hideo Saito, Takeshi Shibata and Kenjiro Taura

3.Only bandwidth matters for large messages (Transfer time) = (Latency) + 4.Bandwidth is only limited by link capacities Assume that nodes and switches have enough processing throughput 9 Problem Settings (cont.) (Message Size) (Bandwidth) 50msec 1Gbps 1GB 99%

5.Bandwidth-annotated topologies are given in advance Bandwidth and topologies can be rapidly inferred - Shirai et al. A Fast Topology Inference - A building block for network-aware parallel computing. (HPDC 2007) - Naganuma et al. Improving Efficiency of Network Bandwidth Estimation Using Topology Information (SACSIS 2008, Tsukuba, Japan) 10 Problem Settings (cont.)

 Previous algorithms evaluated broadcast by completion time  However, it cannot evaluate the effect of slowly receiving nodes It is desirable that each node receives as much data as possible  Aggregate Bandwidth is a more reasonable evaluation criterion in many cases Evaluation of Broadcast 11

 All nodes receive maximum possible bandwidth Receiving bandwidth for each node does not lessen by adding other nodes to the broadcast 12 Definition of Stable Broadcast D0D1D2D D2 120 Single Transfer Broadcast

 Maximize aggregate bandwidth  Minimize completion time 13 Properties of Stable broadcast

 Introduction  Problem Settings  Related Work  Proposed Algorithm  Evaluation  Conclusion 14 Agenda Kei Takahashi, Hideo Saito, Takeshi Shibata and Kenjiro Taura

 Flat tree: The outgoing link from the source becomes a bottleneck  Random Pipeline: Some links used many times become bottlenecks  Depth-first Pipeline: Each link is only used once, but fast nodes suffer from slow nodes  Dijkstra: Fast nodes do not suffer from slow nodes, but some link are used many times Single-Tree Algorithms 15 Flat TreeRandom PipelineDijkstraDepth-First (FPFR) Kei Takahashi, Hideo Saito, Takeshi Shibata and Kenjiro Taura

 FPFR (Fast Parallel File Replication) has improved the aggregate bandwidth from algorithms that use only one tree  Idea: (1) Construct multiple spanning trees (2) Use these trees in parallel FPFR Algorithm [†] 16 [†] Izmailov et al. Fast Parallel File Replication in Data Grid. (GGF-10, March 2004.)

 Iteratively construct spanning trees Create a spanning tree ( Tn ) by tracing every destination Set the throughput ( Vn ) to the bottleneck bandwidth in Tn Subtract Vn from the remaining bandwidth of each link Second Tree (T2) 17 Tree constructions in FPFR Bottleneck First Tree (T1) V1 V2

 Each tree sends different fractions of data in parallel The proportion of data sent through each tree may be optimized by linear programming (Balanced Multicasting [†] ) 18 Data transfer with FPFR T1: Sends the former partT2: sends the latter part [†] den Burger et al. Balanced Multicasting: High-throughput Communication for Grid Applications (SC ‘2005) V1 V2

 In FPFR, slow nodes degrade receiving bandwidth to other nodes  For tree topologies, FPFR only outputs one depth-first pipeline, which cannot utilize the potential network performance Problems of FPFR 19 Bottleneck  

 Introduction  Problem Settings  Related Work  Our Algorithm  Evaluation  Conclusion 20 Agenda Kei Takahashi, Hideo Saito, Takeshi Shibata and Kenjiro Taura

 Modify FPFR algorithm Create both spanning trees and partial trees  Stable for tree topologies whose links have the same bandwidth in both directions 21 Our Algorithm V V Kei Takahashi, Hideo Saito, Takeshi Shibata and Kenjiro Taura

22 T3: Third Tree (Partial Tree) SABC T1: First Tree (Spanning) SABC T2: Second Tree (Partial Tree) SABC  Iteratively construct trees Create a tree Tn by tracing every destination Set the throughput Vn to the bottleneck in Tn Subtract Vn from the remaining capacities Tree Constructions V1 V2 V3 Throughput of T1

 Send data proportional to the tree throughput Vn  Example: Stage1: use T1, T2 and T3 Stage2: use T1 and T2 to send data previously sent by T3 Stage3: use T1 to send data previously sent by T2 Data Transfer 23 AB S C T1 T2 T3 AB S C T1 T2 AB S C T1 (V1) (V2) (V3)

1.Our algorithm is Stable for tree topologies (whose links have the same capacities in both directions) Every node receives maximum bandwidth 2.For any topology, it achieves greater aggregate bandwidth than the baseline algorithm (FPFR) Fully utilize link capacity by using partial trees 3.It has small calculation cost to create a broadcast plan Properties of Our Algorithm 24

 Introduction  Problem Settings  Related Work  Proposed Algorithm  Evaluation  Conclusion 25 Agenda Kei Takahashi, Hideo Saito, Takeshi Shibata and Kenjiro Taura

 Simulated 5 broadcast algorithms using a real topology  Compared the aggregate bandwidth of each method Many bandwidth distributions Broadcast to 10, 50, and 100 nodes 10 kinds of conditions (src, dest) (1) Simulations 26 … … …… 110 nodes81 nodes 36 nodes4 nodes Kei Takahashi, Hideo Saito, Takeshi Shibata and Kenjiro Taura

Compared Algorithms 27 Flat Tree Random Dijkstra Depth-First (FPFR) … and OURS

 Mixed two kinds of Links (100 and 1000) Vertical axis: speedup from FlatTree 40 times more than random, 3 times more than depth-first (FPFR) with 100 nodes Result of Simulations

 Tested 8 bandwidth distributions Uniform distribution ( ) Uniform distribution ( ) Mixed 100 and 1000 links Uniform distribution ( ) between switches (for each distribution, tested two conditions that bandwidth of both directions are the same and different)  Our method achieved the largest bandwidth in 7/8 cases Large improvement especially in large bandwidth variance In a uniform distribution ( ) and link bandwidth in two directions are different, Dijkstra achieved 2% more aggregate bandwidth Result of Simulations (cont.) 29

 Performed broadcasts in 4 clusters Number of destinations:10, 47 and 105 nodes Bandwidths of each link: (10M - 1Gbps)  Compared the aggregate bandwidth in 4 algorithms 1.Our algorithm 2.Depth-first (FPFR) 3.Dijkstra 4.Random (Best among 100 trials) (2) Real Machine Experiment 30

Theoretical Maximum Aggregate Bandwidth 31  Also, we calculated the theoretical maximum aggregate bandwidth The total of the receiving bandwidth in a case of separate direct transfer from the source to each destination D0D1D2D

 For 105 nodes broadcast, 2.5 times more bandwidth than the baseline algorithm DepthFirst (FPFR)  However, our algorithm stayed 50-70% the aggregate bandwidth compared to the theoretical maximum Computational nodes cannot fully utilize up/down network Evaluation of Aggregate Bandwidth

 Compared aggregate bandwidth of 9 nodes before/after adding one slow node Unlike DepthFirst(FPFR), existing nodes do not suffer from adding a slow node in our algorithm Achieved 1.6 times bandwidth than Dijkstra Evaluation of Stability 33 Slow

 Introduction  Problem Settings  Related Work  Our Algorithm  Evaluation  Conclusion 34 Agenda Kei Takahashi, Hideo Saito, Takeshi Shibata and Kenjiro Taura

 Introduced the notion of Stable Broadcast Slow nodes never degrade receiving bandwidth of fast nodes  Proposed a stable broadcast algorithm for tree topologies Theoretically proved 2.5 times the aggregate bandwidth in real machine experiments Confirmed speedup in simulations with many different conditions 35 Conclusion Kei Takahashi, Hideo Saito, Takeshi Shibata and Kenjiro Taura

 Algorithm that maximizes aggregate bandwidth in general graph topologies  Algorithm that changes relay schedule by detecting bandwidth fluctuations Future Work 36 Kei Takahashi, Hideo Saito, Takeshi Shibata and Kenjiro Taura

 Algorithm that maximizes aggregate bandwidth in general graph topologies  Algorithm that changes relay schedule by detecting bandwidth fluctuations Future work 37

All the graphs 38

 BitTorrent gradually improves the transfer schedule by adaptively choosing the parent node  Since relaying structure created by BitTorrent has many branches, these links may become bottlenecks 39 Broadcast with BitTorrent [†] [†] Wei et al. Scheduling Independent Tasks Sharing Large Data Distributed with BitTorrent. (In GRID ’05) Transfer tree snapshot Bottleneck Link

 Uniform distribution ( ) between switches Vertical axis: speedup from FlatTree 36 times more than FlatTree, 1.2 times more than DepthFirst (FPFR) for 100-nodes broadcast Simulation ~1000

 Trace all the destinations from the source Some links used by many transfers become bottlenecks 41 Topology-unaware pipeline Bottleneck

 Construct a depth-first pipeline by using topology information Avoid link sharing by using each link only once Minimize the completion time in a tree topology  Slow nodes degrade the performance of other nodes 42 Depth-first Pipeline Slow Node [†] Shirai et al. A Fast Topology Inference - A building block for network-aware parallel computing. (HPDC 2007)

 Construct a relaying structure in a greedy manner Add a node reachable in the maximum bandwidth one by one Effects of slow nodes are small  Some links may be used by many transfers, may become bottlenecks Dijkstra Algorithm [†] 43 [†] Wang et al. A novel data grid coherence protocol using pipeline-based aggressive copy method. (GPC, pages 484–495, 2007) Bottleneck Link