1. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 1 Principles of Reliable Distributed Systems Lecture 13: SkipNet Spring 2006 Dr. Idit Keidar

2. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 2 Reading Material SkipNet: A Scalable Overlay Network with Practical Locality Properties Harvey, Jones, Saroiu, Theimer, Wolman Microsoft Research

3. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 3 Reminder: DHT Advantages Peer-to-peer: no centralized control or infrastructure Scalability: O(log N) routing, routing tables, join time Load-balancing Overlay robustness

4. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 4 DHT Disadvantages: SkipNet Motivation No control where data is stored In practice, organizations want: –Content Locality – explicitly place data where we want (inside the organization) –Path Locality – guarantee that local traffic (a user in the organization looks for a file of the organization) remains local No prefix search

5. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 5 SkipNet Content Locality Place files at nodes according to names Name ID space (DNS-like) –for files and nodes –node name = reverse DNS name of the host (com.microsoft.host1) –file names have same prefix Problem?

6. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 6 Constrained Load-Balancing Data uniformly distributed in designated subset of nodes –e.g., inside organization

7. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 7 SkipNet’s Two Name Spaces Name ID Space Numerical ID Space com.microsoft.host1 h(com.microsoft.host1) non-uniform uniform

8. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 8 Skip Lists - Reminder In-memory dictionary data structure. –Sorted linked list with a subset of nodes having additional links to skip over many list elements Perfect (deterministic) skip list: –Search: O (log N), N – number of nodes in the list. –Insertion/deletion: expensive/awkward Probabilistic skip list: –Search, Insert, Delete: O (log N) w.h.p.

9. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 9 Skip List: Good for Us? The Good: –Sorted list: path locality for name-based search –O(log N) search with skip pointers –Up to log(N) skip pointers: O(log N) instertion The Bad: –Lookup starts from root only –Unequal load

10. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 10 SkipNet Global View Ring 000 Ring 001 Ring 010 Ring 011 Ring 100 Ring 101 Ring 110 Ring 111 A D M O T Z X V A M T X D O ZV AT M X O Z D V AT M XZ OD V Ring 00Ring 01Ring 10Ring 11 Ring 0 Ring 1 Root Ring Level L = 0 L = 1 L = 2 L = 3 The full SkipNet routing infrastructure for an 8 node system, including the ring labels.

11. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 11 SkipNet Structure Skip Graph = Distributed Skip List –Every node belongs to rings at all levels –Search can start at any node –Use doubly linked lists at each level to account for absence of head and tail nodes. Perfect vs. Probabilistic –Perfect : Pointers at level h point to nodes that are exactly 2 h nodes to the left and right. –Probabilistic : A node in level h probabilistically determines which ring it belongs to.

12. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 12 SkipNet Routing Tables Level: L = 0 L = 1 L = 2 Ring 00 Ring 01 Ring 10Ring 11 Ring 000 Ring 001 Ring 010 Ring 011 Ring 100 Ring 101 Ring 110 Ring 111 A Root Ring D M O T V X Z Ring 0 A M T X Ring 1 D Z V O O Z AT M X D V A T M X D V Z O L = 3 Node A’s Routing Table

13. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 13 An Alternative View Level 2TT 1MX 0DZ SkipNet nodes ordered by name ID. Routing tables of nodes A and V shown. A D M O T Z X V Level 2DD 1ZO 0XT 000001 010 011 100 101110 111

14. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 14 Routing By Name ID Routing in Skip Graph = Search in Skip Lists Simple Rule: –Forward the message to node that is closest to destination, without going too far. Route either clockwise/counterclockwise Terminates when messages arrives at a node whose name ID is closest to destination. Number of hops is O(log N) w.h.p.

15. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 15 Example: Routing from A to V Level: L = 0 L = 1 L = 2 Ring 00 Ring 01 Ring 10Ring 11 Ring 000 Ring 001 Ring 010 Ring 011 Ring 100 Ring 101 Ring 110 Ring 111 A Root Ring D M O T V X Z Ring 0 A M T X Ring 1 D Z V O O Z AT M X D V A T M X D VZ O L = 3

16. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 16 Example: Routing from A to V Level: L = 0 L = 1 L = 2 Ring 00 Ring 01 Ring 10Ring 11 Ring 000 Ring 001 Ring 010 Ring 011 Ring 100 Ring 101 Ring 110 Ring 111 A Root Ring D M O T V X Z Ring 0 A M T X Ring 1 D Z V O O Z AT M X D V A T M X D V Z O L = 3 Node T’s Routing Table

17. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 17 Example: Routing from A to V Level: L = 0 L = 1 L = 2 Ring 00 Ring 01 Ring 10Ring 11 Ring 000 Ring 001 Ring 010 Ring 011 Ring 100 Ring 101 Ring 110 Ring 111 A Root Ring D M O T V X Z Ring 0 A M T X Ring 1 D Z V O O Z AT M X D V A T M X D V Z O L = 3

18. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 18 Example: Routing to Object Level: L = 0 L = 1 L = 2 Route from A to F -> Terminates at E Ring 00 Ring 01 Ring 10Ring 11 Ring 000 Ring 001 Ring 010 Ring 011 Ring 100 Ring 101 Ring 110 Ring 111 A Root Ring D E O V X Z Ring 0 A ETX Ring 1 D Z V O O Z AT E X D V A T E X D VZ O L = 3 T

19. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 19 Name ID Routing Algorithm SendMsg(nameID, msg) { if( LongestPrefix(nameID,localNode.nameID)==0 ) msg.dir = RandomDirection(); else if( nameID<localNode.nameID ) msg.dir = counterClockwise; else msg.dir = clockwise; msg.nameID = nameID; RouteByNameID(msg); } // Invoked at all nodes (including the source and // destination nodes) along the routing path. RouteByNameID(msg) { // Forward along the longest pointer // that is between us and msg.nameID. h = localNode.maxHeight; while (h >= 0) { nbr = localNode.RouteTable[msg.dir][h]; if (LiesBetween(localNode.nameID, nbr.nameID, msg.nameID, msg.dir)) { SendToNode(msg, nbr); return; } h = h - 1; } // h<0 implies we are the closest node. DeliverMessage(msg.msg); } Load Balancing Path Locality

20. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 20 Routing By Numeric ID Numeric id’s are random, no ring is sorted by them –We can’t route top-down! Bottom-up Routing –Routing begins at level 0 ring until a node is found whose numeric ID matches the destination numeric ID in the first digit. –Messages forwarded from ring in level h, R h, to a ring in level h+1, R h+1, such that nodes in R h+1 share h+1 digits with destination numeric ID. –Terminates when message delivered, or none the nodes in R h share h+1 digits with destination numeric ID

21. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 21 Example: Routing by Numeric ID –Hash(“Foo.c”) = 101 Level: L = 0 L = 1 L = 2 Ring 00 Ring 01 Ring 10 Ring 11 Ring000Ring000 Ring 001 Ring 001 Ring 010 Ring 010 Ring 011 Ring 011 Ring 100 Ring 100 Ring 101 Ring 101 Ring 110 Ring 110 Ring 111 Ring 111 Root Ring D M O T V X Z Ring 0 M T X Ring 1 D Z V O O Z AT M X D V A T M X D V Z O L = 3 Foo.c A A

22. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 22 Routing by Numeric ID The same routing tables are used for routing by nameID and numericID The number of message hops is O(log N) whp What sequential data structure does this search resemble?

23. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 23 Routing Algorithm // Invoked at all nodes (including the source and destination nodes) along the routing path. // Initially: msg.ringLvl = -1, msg.startNode = msg.bestNode = null & msg.finalDestination = false RouteByNumericID(msg) { if (msg.numID == localNode.numID || msg.finalDestination) { DeliverMessage(msg.msg); return; } if (localNode == msg.startNode) { // Done traversing current ring. msg.finalDestination = true; SendToNode(msg.bestNode); return; } h = CommonPrefixLen(msg.numID, localNode.numID); if (h > msg.ringLvl) { // Found a higher ring. msg.ringLvl = h; msg.startNode = msg.bestNode = localNode; } else if ( abs(localNode.numID - msg.numID) < abs(msg.bestNode.numID - msg.numID)) { // Found a better candidate for current ring. msg.bestNode = localNode; } // Forward along current ring. nbr = localNode.RouteTable[clockWise][msg.ringLvl]; SendToNode(nbr); }

24. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 24 Routing Summary It all depends on how we look at the routing tables … What is the data structure consisting of all the pointers in the rings that the specific node’s name ID belongs to? –A Skip List! Search is top-down. What is the data structure consisting of all the rings in respect to searching by numeric id? –A Trie! Search is bottom-up. The search in both directions takes O(log N) messages whp. Ready for join/departure procedures?

25. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 25 Node Join Two-stage process: (1) bottom-up + (2) top-down Bottom-up: find the top level ring that matches the node’s numeric ID. Top-down: build the new node’s routing table –Find a neighbor in the top ring using name ID search. –Starting from this neighbor, search for the name ID at the next lower level and thus find neighbors at lower level. –Repeated until the search reaches the root. Update of the existing nodes’ routing tables: –after the new node has joined the root ring.

26. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 26 Node join illustrated Ring P0 Ring P1 Ring P Only a few in expectation Joining node

27. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 27 Node Join - Analysis Key ideas: –Climb to a weakly populated ring. –Search for the node’s neighbors at the lower levels only after finding the neighbors at the higher levels. –The range of traversed nodes at the level = the range of neighbors at the next higher level. Insertion traverses O(log N) hops whp –Expected O(log N) levels, constant number of neighbors at each level.

28. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 28 Node Departure/Failure Graceful (notified) vs crash departure Key issue –routing tables’ update Key idea – separate vital info from optimizations –Routing is correct as long as the root level ring is maintained. –Other levels regarded as optimization hints –Does this remind something? Upper-ring membership maintained through a background repair process.

29. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 29 Leaf Sets Idea = use redundant pointers at level 0: –Protect from independent failures –Improve the search performance Store L/2 pointers in every direction SkipNet uses L=16 –Not an original SkipNet idea – used in Pastry.

30. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 30 Constrained Load Balancing (CLB) Multiple DHTs with differing scopes using a single SkipNet structure –A result of the ability to route in both address spaces Divide data object names into two parts with ! CLB Domain CLB Suffix microsoft.com ! skipnet.html Numeric Routing Name Routing microsoft.com/skipnet.html! – controlled placement !microsoft.com/skipnet.html – Global DHT

31. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 31 CLB Example File ID = “com.microsoft ! skipnet.html” –Route by name ID to com.microsoft –Inside com.microsoft, route by numeric ID to hash(“skipnet.html”) com.sun edu.ucb gov.irs com.microsoft skipnet. html

32. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 32 SkipNet Path Locality Organizations correspond to contiguous SkipNet segments –Internal routing by NameID remains internal Nodes have left / right pointers com.sun edu.ucb gov.irs com.microsoft com.microsoft.research

33. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 33 Course Summary

34. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 34 Main Topics State machine replication for consistency and availability. –Uses Atomic Broadcast. –Uses Consensus. Asynchronous Message-Passing Models –Consensus impossible –Solvable with eventual synchrony, failure detectors <>S,  –In two communication rounds in “fast” case Shared memory –Convenient model –Can be emulated using message-passing –Good for “data-centric” replication

35. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 35 Course Summary (What I Hope You Learned…) Distributed systems are subtle. –It’s very easy to get things wrong –Lesson: don’t design a distributed system without proving the algorithm first! Redundancy is the key to reliability. –Multiple replicas: 2t+1, 3t+1, etc.. Strong consistency is attainable but costly and has scalability limitations.

36. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2006 36 Have a Great Summer!