Opportunistic Multipath Forwarding in Publish/Subscribe Systems Reza Sherafat Kazemzadeh AND Hans-Arno Jacobsen Middleware Systems Research Group University of Toronto

Agenda The content-based publish/subscribe model – Characteristics and challenges Our approach – Overlay neighborhoods – Adaptive multipath forwarding Experimental evaluation results 2Opportunistic Multipath ForwardingMiddleware 2012

Content-Based Pub/Sub Model 3 Pub/Sub S S S S S S S S S S S S S S P P Publish P P P P P P S S Subscribers P P P Publishers Opportunistic Multipath ForwardingMiddleware 2012

Content-Based Pub/Sub Model 4 Pub/Sub Publish S S Subscribers P P P Publishers Opportunistic Multipath ForwardingMiddleware 2012 Many to many communication between a large number of publishers/subscribers Selective delivery based on subscription matching Unicast (one) Multicast (some) Broadcast (all) Variations in traffic patterns makes it difficult to design an optimal overlay network Many to many communication between a large number of publishers/subscribers Selective delivery based on subscription matching Unicast (one) Multicast (some) Broadcast (all) Variations in traffic patterns makes it difficult to design an optimal overlay network

Forwarding paths in the overlay are constructed in fixed end-to-end manner (no/little path diversity) This results in a high number of “pure forwarding” brokers Current Pub/Sub Overlays D C E B A P S ✗ ✗ ✓ ✗ ✓ 5 P P #msgs delivered #msgs sent System Yield = = 1/5 = 20% Opportunistic Multipath ForwardingMiddleware 2012

Summary of Our Goals Construct a highly connected overlay mesh that provides high path diversity between publishers and subscribers Avoid pure forwarders by allowing brokers to make fine- grained forwarding decisions based on individual publications and their matching sets Improve system yield, efficiency, scalability and delivery delay Support dynamic adaptive routing based on live traffic patters while avoiding high costs of full overlay reconfiguration 6Opportunistic Multipath ForwardingMiddleware 2012

Fixed end-to-end (baseline) Total msgs: 6 Forwarding strategy 1 Total msgs: 5 Forwarding strategy 2 Total msgs: 3 ABC * D * * Forwarding Strategies ABC * D * * p * * * * * * p * * * * * * ABC * D * * p * * * * * * 7Opportunistic Multipath ForwardingMiddleware 2012

Routing Tables (Δ-neighborhoods knowledge) Links Management (best links via a gain function) Pub. Forwarding (Path Computations for strategies) Our Approach in a Nutshell A A Δ=1 Δ=2 Δ=3 S S 8Opportunistic Multipath ForwardingMiddleware 2012

AC ** * B D AC ** * B D Path Computation for Forwarding Strategies Strategy 1 Middleware 2012Opportunistic Multipath Forwarding9 ABC * D * * p * * * * * * ABC * D * * p * * * * * * Strategy 2

Experimental Evaluations We have implemented the algorithms and performed large-scale experimental evaluations with up to 500 brokers Datasets – Synthesized based on Zipf distribution – Social networking traces from Facebook We measured performance of the system in terms of: – Overlay mesh connectivity – Delivery delay – Maximum system throughput – System yield – Publication propagation path length – Memory and CPU overhead Middleware 2012Opportunistic Multipath Forwarding10

Number of Available paths Path Diversity in Overlay Mesh Network size:120 Delta=3 Network size:250 Delta=3 Path diversity: 20% of brokers w/ 100 paths Path diversity: 10% of brokers w/ 1000 paths 11Opportunistic Multipath ForwardingMiddleware 2012 Graph is based on a snapshot of the state of links in a running system

Publication Hop Count Experiment setup: 120 Brokers Publish rate is 1,800 msgs/sec and number of deliveries: 73,000 (in 5 min) 12Opportunistic Multipath ForwardingMiddleware 2012 Fixed-end-to-end Strategy 2 115% throughput enhancement

Conclusions Brokers build a highly connected overlay mesh and make fine-grained forwarding decisions for each publication in order to avoid pure forwarding neighbors We used the notion of overlay neighborhoods to enable local traffic profiling and avoid high costs of overlay reconfiguration Our approach enhances system’s efficiency and yield, and ultimately improves its scalability and performance 13Opportunistic Multipath ForwardingMiddleware 2012

Thanks Questions! 14Opportunistic Multipath ForwardingMiddleware 2012

Client Broker Pub/Sub Characteristics Many to many communication between a large number of publishers/subscribers Selective delivery based on subscription matching – Unicast (one) – multicast (some) – broadcast (all) Traffic patterns depends on workload application and may change over time 15Opportunistic Multipath ForwardingMiddleware 2012

Related Work: Overlay Reconfiguration Broker overlay is “reconfigured” by addition and removal of links between brokers Advantages – Forwarding path may be improved bringing some publishers and subscribers closer together Disadvantages – Some forwarding paths between publishers and subscribers may indeed suffer – Resulting overlay still relies on fixed end-to-end paths – Reconfiguration is costly and requires full or partial re-propagation of subscriptions [1] Virgillito, A., Beraldi, R., Baldoni, R.: On event routing in content-based publish/subscribe through dynamic networks, FTDCS ‘03 [2] Virgillito, A., Beraldi, R., Baldoni, R.: On event routing in content-based publish/subscribe through dynamic networks. In: FTDCS. (2003)   Re- configure 16Opportunistic Multipath ForwardingMiddleware 2012

Links types – Primary links – Soft links – shortcut links – Candidate links – expeditionary links Traffic profiling – In intervals of T sec, brokers count the number of pubs sent over each links – Gain function gain(A,B,T) = pub_traffic_during_T * dist(A,B) Soft link selection – Profiling input – Link state measured – Broker load information exchanged Links ranking A B C D E F G E F … high → low Links Management Overlay Network AB C D B B Primary link Candidate link Soft link 17Opportunistic Multipath ForwardingMiddleware 2012

Master v. Working Routing Data Structures Overlay map captured by brokers’ Δ-neighborhoods are relatively static  Master overlay Map (MOM) Brokers link connectivity change dynamically, brokers need an efficient way to compute forwarding paths over the changing set of links  Working Overlay Map (WOM) MOM is a concise representation of the primary overlay that only contains brokers with a direct link Master Routing Tables Working Routing Tables construct 18Opportunistic Multipath ForwardingMiddleware 2012

Master v. Working Subscription Tables Similar to WOM, brokers adapt their subscription tables based on the current set of available links:  Working Subscription Table (WST) Between Brokers Sets Beyond Brokers Sets Behind Brokers Sets Master Subscription Table Working Subscription Table Working Overlay Map 19Opportunistic Multipath ForwardingMiddleware 2012

Delivered publications StrategyNumber of pure Pure Forwarders System Yield 284,000 (Dense Workload) Baseline 195,00059% Strategy 1 104,00073% Strategy 2 69,00080% Delivered publications Strategy Number of pure Pure Forwarders System Yield 73,000 (Sparse Workload) Fixed end-to-end 91,00044% Strategy 1 42,00063% Strategy 2 29,00071% System Yield (measure of efficiency) 20Opportunistic Multipath ForwardingMiddleware 2012

Publication Hop Count Experiment setup: 120 Brokers Sparse and dense workloads Publish rate of 1,800 msgs/sec  Deliveries: 73,000 in 5 min 21Opportunistic Multipath ForwardingMiddleware 2012

Publication Hop Count Experiment setup: 120 Brokers Sparse publication/subscription workload Publish rate of 1,800 msgs/sec  Deliveries: 73,000 in 5 min 22Opportunistic Multipath ForwardingMiddleware 2012

Publication Hop Count Sparse Matching WorkloadDense Matching Workload Multi-path forwarding is more effective in sparse workloads 23Opportunistic Multipath ForwardingMiddleware 2012

Impact of Broker Fanout on Subscription Covering Experiment setup: 500 brokers Fanout of Opportunistic Multipath ForwardingMiddleware 2012