1 Roie Melamed, Technion AT&T Labs Araneola: A Scalable Reliable Multicast System for Dynamic Wide Area Environments Roie Melamed, Idit Keidar Technion

Roie Melamed, TechnionAT&T Labs2 Outline Goals Goals The overlay The overlay –Fault-tolerance –Average cost per join/leave Multicast Multicast –Undisrupted service in the face of churn Exploiting network proximity and bandwidth heterogeneity Exploiting network proximity and bandwidth heterogeneity

Roie Melamed, TechnionAT&T Labs3 Design Goals Scalability Scalability –Low constant load on each node Reliability Reliability –Undisrupted service –Under node and link failures –Despite churn Coping with high churn Coping with high churn –Low constant cost for handling joins and failures Easy deployment without relying on infrastructure Easy deployment without relying on infrastructure

Roie Melamed, TechnionAT&T Labs4 Araneola: Main Features Very cheap maintenance Very cheap maintenance –Good performance with high churn Robust to link & node failures Robust to link & node failures Balanced low degree basic overlay Balanced low degree basic overlay –Fair: no node contributes more than its share –Leaves ample bandwidth for communication with near-by nodes Additional links to near-by nodes according to available bandwidth Additional links to near-by nodes according to available bandwidth

Roie Melamed, TechnionAT&T Labs5 Basic (Random) Overlay For k ≥ 3, approximate k-regular random graph: For k ≥ 3, approximate k-regular random graph: –Each node has either k or k+1 random neighbors –Logarithmic diameter –k- connected –Expander, remains highly connected following random removal of large subsets of edges or nodes –Regularity allows degree to be as low as 3 instead of log(N) in a normal random graph Each join or leave operation incurs sending only a total of about 3k messages Each join or leave operation incurs sending only a total of about 3k messages

Roie Melamed, TechnionAT&T Labs6 Building and Maintaining the Basic Overlay Two tunable parameters Two tunable parameters –L (usually 5) –H (usually L+5) Each node connects to L random nodes Each node connects to L random nodes –A connect request is accepted iff the node’s degree is below H Use gossip-based membership service to choose random nodes Use gossip-based membership service to choose random nodes –Empirically, this is good enough

Roie Melamed, TechnionAT&T Labs7 The Connect Task N1N

Roie Melamed, TechnionAT&T Labs8 The Connect Task: Redirection N1N N3 12

Roie Melamed, TechnionAT&T Labs9 Steady State If there is a time after which no joins, leaves, or failures occur, eventually, each node’s degree is between L and H If there is a time after which no joins, leaves, or failures occur, eventually, each node’s degree is between L and H Reduction rules (next slides) will further reduce node degrees Reduction rules (next slides) will further reduce node degrees

Roie Melamed, TechnionAT&T Labs10 Reduction Rule 1 Removes the connection between neighboring nodes with degrees > L Removes the connection between neighboring nodes with degrees > L –Without reducing any node's degree to be below L N2N

Roie Melamed, TechnionAT&T Labs11 Rule 1: Unapproved Disconnect N2N1 6 N

Roie Melamed, TechnionAT&T Labs12 Steady State: Take II If there is a time after which no joins, leaves, or failures occur, eventually, each node’s degree is between L and H, and at most 50% of the nodes have degrees > L If there is a time after which no joins, leaves, or failures occur, eventually, each node’s degree is between L and H, and at most 50% of the nodes have degrees > L Reduction Rule 2 (next slides) will reduce each node’s degree to be L or L+1 Reduction Rule 2 (next slides) will reduce each node’s degree to be L or L+1

Roie Melamed, TechnionAT&T Labs13 Reduction Rule 2 n hl

Roie Melamed, TechnionAT&T Labs14 Steady State: Take III If there is a time after which no joins, leaves or failures occur, eventually, each node’s degree is either L or L+1, and at most 50% of the nodes have degree L+1 If there is a time after which no joins, leaves or failures occur, eventually, each node’s degree is either L or L+1, and at most 50% of the nodes have degree L+1

Roie Melamed, TechnionAT&T Labs15 Degree Distribution, 8000 Nodes (On Cluster)

Roie Melamed, TechnionAT&T Labs16 Overlay Scalability #Nodes%nodes degree=5 diameteravg distance avg #paths

17 Roie Melamed, Technion AT&T Labs Fault Tolerance and Graceful Degradation Fault Tolerance and Graceful Degradation The expansion properties of k-regular random graphs provide fault tolerance

Roie Melamed, TechnionAT&T Labs18 Resilience to Edge Removals (L=5)

Roie Melamed, TechnionAT&T Labs19 Resilience to Node Removals (L=5)

20 Roie Melamed, Technion AT&T Labs High Churn Experiments High Churn Experiments Araneola achieves low cost per join/leave thanks to amortization of costs

Roie Melamed, TechnionAT&T Labs21 High Churn Experiments: Diminishing Average Cost Per Join/Leave

Roie Melamed, TechnionAT&T Labs22 Multicasting over Araneola Option 1: Flooding Flooding multicast messages Flooding multicast messages Low latency Low latency  Many duplicate messages Flooding message identifiers Flooding message identifiers Each data message is sent exactly N - 1 times (in the absence of failures) Each data message is sent exactly N - 1 times (in the absence of failures)  Many duplicate identifiers  in many packets

Roie Melamed, TechnionAT&T Labs23 Multicasting over Araneola Option 2: Gossip Gossip about recent identifiers to neighbors Gossip about recent identifiers to neighbors Each data message is sent exactly N - 1 times (in the absence of failures) Each data message is sent exactly N - 1 times (in the absence of failures) Bundle several identifiers into a single packet Bundle several identifiers into a single packet  Higher latency (OK for non-real time apps) We now evaluate gossip over Araneola versus normal gossip We now evaluate gossip over Araneola versus normal gossip

Roie Melamed, TechnionAT&T Labs24 Message Propagation Rate, 8000 nodes

Roie Melamed, TechnionAT&T Labs25 Message Propagation Rate, L= 5

Roie Melamed, TechnionAT&T Labs26 Araneola vs. Gossip, 1000 Nodes

Roie Melamed, TechnionAT&T Labs27 Araneola vs. High Fanout Gossip, 1000 Nodes

28 Roie Melamed, Technion AT&T Labs Undisrupted Service The large number of disjoint paths in the overlay allows for undisrupted service under high churn The large number of disjoint paths in the overlay allows for undisrupted service under high churn

Roie Melamed, TechnionAT&T Labs29 Undisrupted Service with Churn

30 Roie Melamed, Technion AT&T Labs Exploiting Available Bandwidth for Application Needs

Roie Melamed, TechnionAT&T Labs31 Observation In order to achieve the important mathematical properties of k-regular random graphs, 3 random neighbors suffice In order to achieve the important mathematical properties of k-regular random graphs, 3 random neighbors suffice Additional neighbors can be chosen according to application needs Additional neighbors can be chosen according to application needs –network proximity, available bandwidth, etc.

Roie Melamed, TechnionAT&T Labs32 Example: Adding Near-By Neighbors Connect near-by task, like basic the connect task, except that – Connect near-by task, like basic the connect task, except that – –L and H are replaced by NB - the number of near-by nodes to connect to –No reductions rules

Roie Melamed, TechnionAT&T Labs33 Hop-Count Statistics with Different Selections of Hop-Count Statistics with Different Selections of % of links on the same machine % of short links avg hop count

Roie Melamed, TechnionAT&T Labs34 Question Rank the following Araneola overlays according to their robustness: Rank the following Araneola overlays according to their robustness: – –

Roie Melamed, TechnionAT&T Labs35 Answer (Edges)

Roie Melamed, TechnionAT&T Labs36 Answer (Nodes)

Roie Melamed, TechnionAT&T Labs37 Conclusions Scalability: Scalability: –The only aspect of Araneola that varies with the number of nodes is message latency, which increases logarithmically with the group size –Araneola's load, reliability, resilience to message loss, resilience to simultaneous node failures, and overhead for handling join and leave events are all independent of the number of nodes High reliability in the presence of sizable message loss rates, simultaneous failures of a certain percentage of the nodes, and high churn High reliability in the presence of sizable message loss rates, simultaneous failures of a certain percentage of the nodes, and high churn Build an expander using the minimal number of random links Build an expander using the minimal number of random links –leaves ample bandwidth for application-specific needs