Cognitive Publish/Subscribe for Heterogeneous Clouds Šarūnas Girdzijauskas, Swedish Institute of Computer Science (SICS) Joint work with: Fatemeh Rahimian (SICS)

Based on decentralized architecture Abundance of networked collection of connected devices forming micro-clouds Decentralized Publish/Subscribe service – Content distribution – IP TV Streaming – Online gaming – Collaborative editing – Etc.. Adapting to the topology and network dynamics of microclouds Adapting to different usage patterns Future Clouds? Šarūnas Girdzijauskas, Cloud Futures, Redmond, April 2010

Pub/Sub Systems: Our Focus Šarūnas Girdzijauskas, Cloud Futures, Redmond, April 2010 Scalable pub/sub service – Very large number of nodes – Very large number of “topics” – Heterogeneous environments Arbitrary geographical distribution Arbitrary subscription and dissemination patterns – Central solutions will not scale

 Tradeoffs: Node degree Number of uninterested (relay) nodes involved Dissemination delay Dissemination cost  Cognitive pub/sub: Fixed node degree Account for the underlying topology (bandwidth & cost) Minimize the number of relay nodes by exploiting user subscription correlation & event publication rates Šarūnas Girdzijauskas, Cloud Futures, Redmond, April 2010 Pub/Sub Systems: Our Focus (2)

Conceptual Architecture Šarūnas Girdzijauskas, Cloud Futures, Redmond, April 2010 Physical Network Cognitive Overlay Pub/sub Dissemination structures

Gossip (epidemic) overlays – A lot of research (e.g., Cyclon, T-man) – Lightweight, scalable and robust mechanism – Cyclic/Periodic, pair-wise interaction between peers (bounded amount of information) Gossip based pub/sub Šarūnas Girdzijauskas, Cloud Futures, Redmond, April 2010

Gossiping enables us to find and cluster peers with similar interests connected by cheap and fast links – A node starts with a local fixed size view in a random network – Performs a bidirectional exchange of the view with a random node  2 views – Keeps the only the preferred (ranking function) nodes in the view  1 view – Repeat Towards Cognitive Structure Šarūnas Girdzijauskas, Cloud Futures, Redmond, April 2010

Gossiping Building Cognitive Structure Šarūnas Girdzijauskas, Cloud Futures, Redmond, April 2010 $ Making clusters by utilizing ranking function which prefers neighbors with similar interests Peer interest similarity metric – Node subscriptions s1, s2 ⊆ T – sim(s1, s2) =|s1 ∩ s2|/|s1 ∪ s2| Weighted by link cost (bandwidth and $) Weighted by Topic publication rates – Number of neighbors is limited! Decided locally on each peer

Problem: How to publish? Clustering peers of similar interests into bandwidth and cost effective clusters – Clusters might (will) be disjoint – Event publishing requires connected components for each topic Šarūnas Girdzijauskas, Cloud Futures, Redmond, April 2010

Navigable Small-World Network Inter-Cluster Connectivity Structure is added: Navigable Small-World topology – Purely by using gossiping Šarūnas Girdzijauskas, Cloud Futures, Redmond, April 2010

Building Navigable Structure Šarūnas Girdzijauskas, Cloud Futures, Redmond, April Every peer decides on random ID Updating ranking function for choice of neighbors: – Ring Link(s) – Long-Range link (Small- World style) for polylogarithmic routing performance

Navigable Small-World Network Inter-Cluster Connectivity Structure is added (Navigable Small-World made by gossiping) – Ring Links – Long-Range (finger) link(s) – Clustering (friend) links Clusters are connected by greedy routes – Rendezvous node for each topic – All links are used! All topics become connected – For publishing “flood the topic”, or – Choose a rendezvous node to publish Šarūnas Girdzijauskas, Cloud Futures, Redmond, April 2010

Ongoing work Synthetic data sets for user subscription correlation Twitter data set Skype churn data Our experiments show: – Up to 10 fold reduction of relay traffic as compared to existing approaches (e.g., Scribe, Bayeux) Šarūnas Girdzijauskas, Cloud Futures, Redmond, April 2010

Gossip based pub/sub (recap) Large scale pub/sub for heterogeneous environments Dissemination structures are self-organizing – Forming clusters of similar nodes – Converging into least expensive dissemination paths on the underlying physical network – Continuously adapting to the environment conditions – Fast convergence, robustness to churn and failures. Šarūnas Girdzijauskas, Cloud Futures, Redmond, April 2010

Thank you! Questions? Šarūnas Girdzijauskas, Cloud Futures, Redmond, April 2010