1. Alex Shraer, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 1 Principles of Reliable Distributed Systems Tutorial 4: SkipNet Spring 2008 Alex Shraer

2. Alex Shraer, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 2 Reading Material SkipNet: A Scalable Overlay Network with Practical Locality Properties Harvey, Jones, Saroiu, Theimer, Wolman Microsoft Research

3. Alex Shraer, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 3 Reminder: DHT Advantages Peer-to-peer: no centralized control or infrastructure Scalability: O(log N) routing, routing tables, join time Load-balancing Overlay robustness

4. Alex Shraer, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 4 DHT Disadvantages: SkipNet Motivation No control where data is stored –Data may be stored far from its users –Data may be stored outside its administrative domain hard to administer privileges invites different security attacks –Local accesses leave local organization In practice, organizations want: –Content Locality – explicitly place data where we want (inside the organization) –Path Locality – guarantee that local traffic (a user in the organization looks for a file of the organization) remains local No prefix search –Search(key) returns file whose name has key as prefix

5. Alex Shraer, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 5 Practical Requirements Data Controllability: –Organizations want control over their own data –Even if local data is globally available Manageability: –Data control allows for data administration, provisioning and manageability

6. Alex Shraer, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 6 Practical Requirements (cont’d) Security: –Content and path locality are key building blocks for dealing with certain external attacks (DoS, Traffic analysis) Data availability –Local data survives network partitions. Performance –Data can be stored near clients that use it

7. Alex Shraer, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 7 SkipNet Content Locality Place files at nodes according to names Name ID space (DNS-like) –for files and nodes –node name = reverse DNS name of the host (com.microsoft.host1) –file names have same prefix Problem?

8. Alex Shraer, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 8 Constrained Load-Balancing Data uniformly distributed in designated subset of nodes –e.g., inside organization How can this be achieved? Numeric ID space! –similar to Chord, Pastry and others –nodes are randomly distributed –Hashes of the node names and content identifiers mapped into the numeric ID. –Content is stored on the node with id closest to content’s hashed name. Key property of SkipNet: two address spaces

9. Alex Shraer, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 9 Skip Lists - Reminder In-memory dictionary data structure. –Sorted linked list with a subset of nodes having additional links to skip over many list elements Perfect (deterministic) skip list: –Pointer at level h skips over 2 h elements –Search: O (log N), N – number of nodes in the list. –Insertion/deletion: expensive/awkward

10. Alex Shraer, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 10 Skip Lists - Reminder Probabilistic skip list: –Node at level h with probability 1/2 h –Search, Insert, Delete: O (log N) w.h.p.

11. Alex Shraer, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 11 Skip List: Good for Us? The Good: –Sorted list: path locality for name-based search –O(log N) search with skip pointers –Up to log(N) skip pointers: O(log N) instertion The Bad: –Lookup starts from root only –Unequal load nodes on the top levels have high chance to be in routing path

12. Alex Shraer, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 12 SkipNet Global View Ring 000 Ring 001 Ring 010 Ring 011 Ring 100 Ring 101 Ring 110 Ring 111 A D M O T Z X V A M T X D O ZV AT M X O Z D V AT M XZ OD V Ring 00Ring 01Ring 10Ring 11 Ring 0 Ring 1 Root Ring Level L = 0 L = 1 L = 2 L = 3 The full SkipNet routing infrastructure for an 8 node system, including the ring labels.

13. Alex Shraer, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 13 SkipNet Structure Skip Graph = Distributed Skip List –Every node belongs to rings at all levels –Search can start at any node –Use doubly linked lists at each level to account for absence of head and tail nodes. Perfect vs. Probabilistic –Perfect : Pointers at level h point to nodes that are exactly 2 h nodes to the left and right. –Probabilistic : A node in level h probabilistically determines which ring it belongs to. All rings are sorted according to Name IDs Ring membership is according to Numeric IDs –All nodes sharing the same prefix of Numeric IDs of length h are members of the same ring at level h

14. Alex Shraer, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 14 SkipNet Routing Tables Level: L = 0 L = 1 L = 2 Ring 00 Ring 01 Ring 10Ring 11 Ring 000 Ring 001 Ring 010 Ring 011 Ring 100 Ring 101 Ring 110 Ring 111 A Root Ring D M O T V X Z Ring 0 A M T X Ring 1 D Z V O O Z AT M X D V A T M X D V Z O L = 3 Node A’s Routing Table

15. Alex Shraer, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 15 An Alternative View Level 2TT 1MX 0DZ SkipNet nodes ordered by name ID. Routing tables of nodes A and V shown. A D M O T Z X V Level 2DD 1ZO 0XT 000001 010 011 100 101110 111

16. Alex Shraer, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 16 Routing By Name ID Routing in Skip Graph = Search in Skip Lists Simple Rule: –Forward the message to node that is closest to destination, without going too far. Route either clockwise/counterclockwise Terminates when messages arrives at a node whose name ID is closest to destination. Number of hops is O(log N) w.h.p.

17. Alex Shraer, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 17 Example: Routing from A to V Level: L = 0 L = 1 L = 2 Ring 00 Ring 01 Ring 10Ring 11 Ring 000 Ring 001 Ring 010 Ring 011 Ring 100 Ring 101 Ring 110 Ring 111 A Root Ring D M O T V X Z Ring 0 A M T X Ring 1 D Z V O O Z AT M X D V A T M X D VZ O L = 3

18. Alex Shraer, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 18 Example: Routing from A to V Level: L = 0 L = 1 L = 2 Ring 00 Ring 01 Ring 10Ring 11 Ring 000 Ring 001 Ring 010 Ring 011 Ring 100 Ring 101 Ring 110 Ring 111 A Root Ring D M O T V X Z Ring 0 A M T X Ring 1 D Z V O O Z AT M X D V A T M X D V Z O L = 3 Node T’s Routing Table

19. Alex Shraer, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 19 Example: Routing from A to V Level: L = 0 L = 1 L = 2 Ring 00 Ring 01 Ring 10Ring 11 Ring 000 Ring 001 Ring 010 Ring 011 Ring 100 Ring 101 Ring 110 Ring 111 A Root Ring D M O T V X Z Ring 0 A M T X Ring 1 D Z V O O Z AT M X D V A T M X D V Z O L = 3

20. Alex Shraer, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 20 Example: Routing to Object Level: L = 0 L = 1 L = 2 Route from A to F -> Terminates at E Ring 00 Ring 01 Ring 10Ring 11 Ring 000 Ring 001 Ring 010 Ring 011 Ring 100 Ring 101 Ring 110 Ring 111 A Root Ring D E O V X Z Ring 0 A ETX Ring 1 D Z V O O Z AT E X D V A T E X D VZ O L = 3 T

21. Alex Shraer, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 21 Name ID Routing Algorithm SendMsg(nameID, msg) { if( LongestPrefix(nameID,localNode.nameID)==0 ) msg.dir = RandomDirection(); else if( nameID<localNode.nameID ) msg.dir = counterClockwise; else msg.dir = clockwise; msg.nameID = nameID; RouteByNameID(msg); } // Invoked at all nodes (including the source and // destination nodes) along the routing path. RouteByNameID(msg) { // Forward along the longest pointer // that is between us and msg.nameID. h = localNode.maxHeight; while (h >= 0) { nbr = localNode.RouteTable[msg.dir][h]; if (LiesBetween(localNode.nameID, nbr.nameID, msg.nameID, msg.dir)) { SendToNode(msg, nbr); return; } h = h - 1; } // h<0 implies we are the closest node. DeliverMessage(msg.msg); } Load Balancing Path Locality

22. Alex Shraer, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 22 Routing By Numeric ID Numeric id’s are random, no ring is sorted by them –We can’t route top-down! Bottom-up Routing –Routing begins at level 0 ring until a node is found whose numeric ID matches the destination numeric ID in the first digit. –Messages forwarded from ring in level h, R h, to a ring in level h+1, R h+1, such that nodes in R h+1 share h+1 digits with destination numeric ID. –Terminates when message delivered, or none the nodes in R h share h+1 digits with destination numeric ID, at a node in R h with closest possible numeric id.

23. Alex Shraer, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 23 Example: Routing by Numeric ID –Hash(“Foo.c”) = 101 Level: L = 0 L = 1 L = 2 Ring 00 Ring 01 Ring 10 Ring 11 Ring000Ring000 Ring 001 Ring 001 Ring 010 Ring 010 Ring 011 Ring 011 Ring 100 Ring 100 Ring 101 Ring 101 Ring 110 Ring 110 Ring 111 Ring 111 Root Ring D M O T V X Z Ring 0 M T X Ring 1 D Z V O O Z AT M X D V A T M X D V Z O L = 3 Foo.c A A

24. Alex Shraer, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 24 Routing by Numeric ID The same routing tables are used for routing by nameID and numericID When Numeric IDs are binary: in each ring R h, in expectation only 2 nodes visited before encountering one belonging to the next ring R h+1 –The number of message hops is O(log N) w.h.p.

25. Alex Shraer, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 25 Routing Algorithm // Invoked at all nodes (including the source and destination nodes) along the routing path. // Initially: msg.ringLvl = -1, msg.startNode = msg.bestNode = null & msg.finalDestination = false RouteByNumericID(msg) { if (msg.numID == localNode.numID || msg.finalDestination) { DeliverMessage(msg.msg); return; } if (localNode == msg.startNode) { // Done traversing current ring. msg.finalDestination = true; SendToNode(msg.bestNode); return; } h = CommonPrefixLen(msg.numID, localNode.numID); if (h > msg.ringLvl) { // Found a higher ring. msg.ringLvl = h; msg.startNode = msg.bestNode = localNode; } else if ( abs(localNode.numID - msg.numID) < abs(msg.bestNode.numID - msg.numID)) { // Found a better candidate for current ring. msg.bestNode = localNode; } // Forward along current ring. nbr = localNode.RouteTable[clockWise][msg.ringLvl]; SendToNode(nbr); }

26. Alex Shraer, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 26 Base (k) for Numeric IDs If a higher base k>2 is used for Numeric IDs the routing is O(klog k N) w.h.p. When we increase k  more rings in each level  less levels  less pointers in routing table  less state but more hops… Optimization - dense routing table (R-Table) Normal (sparse) R-Table + k-1 pointers to contiguous nodes in both directions at each level.  More state but less hops

27. Alex Shraer, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 27 Node Join Two-stage process: (1) bottom-up + (2) top-down Bottom-up: find the top level ring that matches the node’s numeric ID. Top-down: build the new node’s routing table –Find a neighbor in the top ring using name ID search. –Starting from this neighbor, search for the name ID at the next lower level and thus find neighbors at lower level. –Repeated until the search reaches the root. Update of the existing nodes’ routing tables: –after the new node has joined the root ring.

28. Alex Shraer, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 28 Node join illustrated Ring P0 Ring P1 Ring P Only a few in expectation Joining node

29. Alex Shraer, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 29 Node Join - Analysis Key ideas: –Climb to a weakly populated ring. –Search for the node’s neighbors at the lower levels only after finding the neighbors at the higher levels. –The range of traversed nodes at the level = the range of neighbors at the next higher level. Insertion traverses O(log N) hops whp –Expected O(log N) levels, constant number of neighbors at each level.

30. Alex Shraer, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 30 Node Departure/Failure Graceful (notified) vs crash departure Key issue –routing tables’ update Key idea – separate vital info from optimizations –Routing is correct as long as the root level ring is maintained. –Other levels regarded as optimization hints –Does this remind something? Upper-ring membership maintained through a background repair process.

31. Alex Shraer, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 31 Leaf Sets Idea = use redundant pointers at level 0: Store L/2 pointers in each direction SkipNet uses L=16 –Not an original SkipNet idea – used in Pastry. Protect from independent failures Improve the search performance –rout directly using leaf set if got within L/2 of the target

32. Alex Shraer, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 32 Constrained Load Balancing (CLB) Multiple DHTs with differing scopes using a single SkipNet structure –A result of the ability to route in both address spaces Divide data object names into two parts with ! CLB Domain CLB Suffix microsoft.com ! skipnet.html Numeric Routing Name Routing microsoft.com/skipnet.html! – controlled placement !microsoft.com/skipnet.html – Global DHT

33. Alex Shraer, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 33 CLB Example File ID = “com.microsoft ! skipnet.html” –Route by name ID to com.microsoft –Inside com.microsoft, route by numeric ID to hash(“skipnet.html”) com.sun edu.ucb gov.irs com.microsoft skipnet. html

34. Alex Shraer, Principles of Reliable Distributed Systems, Technion EE, Spring 2008 34 SkipNet Path Locality Organizations correspond to contiguous SkipNet segments –Internal routing by NameID remains internal Nodes have left / right pointers com.sun edu.ucb gov.irs com.microsoft com.microsoft.research