SCRIBE A large-scale and decentralized application-level multicast infrastructure

Overview  Pastry  PAST  distributed file system layered on top of Pastry  SCRIBE  decentralized publish/subscribe system

Pastry – Quick Review  Chord like routing  Consistent hashing  Prefix routing  Leaf set

Pastry – locality properties  Short routes  Total distance traveled  Average dist 1.59 to 2.2 times actual dist  Route convergence  Dist Traveled by 2 messages sent to same key  Equal to dist between to nodes before routes converge

Pastry API  nodeID = pastryInit(Credentials)  Causes node to join pastry network  route(msg,key)  send(msg,IP-addr)  Applications must export:  deliver(msg,key)  forward(msg,key,nextID)  newLeafs(leafSet)

SCRIBE  Built on top of Pastry  Support large number of groups  Handle a high rate of membership turnover  SCRIBE nodes can:  Create groups  Join groups  Multicast messages to groups

SCRIBE API  create(credentials, groupID)  join(credentials, groupID)  leave(credentials, groupID)  multicast(credentials, groupID, message)

SCRIBE – Creating a Group  Pastry route(msg, key)  SCRIBE route(CREATE, groupID)  groupID => hash textual name cat creator name  Message delivered to closest key which become rendez-vous point for the group (root of multicast tree for group)  Adds to local list of groups  Stores credentials  Alternative use itself as root  good choice if creator sends to group often

SCRIBE – Joining a Group  Pastry route(msg, key)  SCRIBE route(JOIN, groupID)  routed to rendez-vous point  along the way multicast tree formed

SCRIBE – Leaving a Group  Remove from local group children list  If list becomes empty forward to parent  Part of the multicast tree may be removed

SCRIBE – Sending a multicast message  route(MULTICAST, groupID)  ask for rendez-vous IP address  If rendez-vous fails re-request rendez-vous point  Pastry handles node duplication  All messages are sent through the rendez-vous point

SCRIBE – Repairing the Multicast Tree  Messages are delivered only in best-effort  may be out of order delivery  Periodic heartbeat message sent to all children  Child rejoins the tree through sending a new JOIN message if suspects parent has failed  Can repair rendez-vous point  Pastry handles node duplication in leaf nodes  Children nodes JOIN new root when missing heartbeat is detected

SCRIBE – Forming a Multicast Tree  Rendez-vous point (root)  Forwarders  may or may not be members of the group  maintain a children table (IP and nodeID) for group

SCRIBE - Strengths  Pastry handles root duplication  Rendez-vous point does not handle all join requests  Locality properties of Pastry  short routes delay from rendez-vous point to member is short  route convergence load imposed on physical network is small

SCRIBE – Experimental Evalutation  Simulation experimental results  Focus on three metrics:  delay to deliver events to group members  stress on each node  stress on each physical network link

SCRIBE – Simulator Evaluation  5050 routers and 100,000 end nodes  1,500 groups of different sizes  10 different runs using same parameters but different random seeds  Averaged all results  Compared results with IP multicast

SCRIBE – Delay Penalty  RMD – ratio between max delay using SCRIBE & max delay using IP multicast  RAD – ratio between average delay using SCRIBE  & average delay using IP multicast

SCRIBE – Node Stress  Average node is responsible for forwarding a small number of multicast messages

SCRIBE – Link Stress  Total num links = 1,035,295  SCRIBE = 2,489,824 messages (mean 2.4)  IP multicast = 758,853 messages (mean 0.7)

SCRIBE – Bottleneck remover  Bottlenecks  Low capacity nodes  High capacity nodes with extremely high children entries  Drop children if over capacity  Select child to drop and send message with children table  Child chooses new parent node and sends JOIN message  Result  Removes long tail in node stress graph  Increases average link stress

SCRIBE – Scalability Small Groups  50,000 nodes  30,000 groups 11 members each group  SCRIBE performs poorly for large number small groups  SCRIBE collapse  Removes long paths removing nodes that are not members of a group & have only one entry in their children table

SCRIBE – Scalability Small Groups  Average link stress 6.1 to 3.3  Average number of children 21.2 to 8.5

SCRIBE - Conclusion  Fully decentralized  Support large number of groups  Support large group size  Multiple multicast sources per group  QUESTIONS