Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Principles of Reliable Distributed Systems Lecture 13: SkipNet Spring 2006 Dr. Idit Keidar

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Reading Material SkipNet: A Scalable Overlay Network with Practical Locality Properties Harvey, Jones, Saroiu, Theimer, Wolman Microsoft Research

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Reminder: DHT Advantages Peer-to-peer: no centralized control or infrastructure Scalability: O(log N) routing, routing tables, join time Load-balancing Overlay robustness

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring DHT Disadvantages: SkipNet Motivation No control where data is stored In practice, organizations want: –Content Locality – explicitly place data where we want (inside the organization) –Path Locality – guarantee that local traffic (a user in the organization looks for a file of the organization) remains local No prefix search

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring SkipNet Content Locality Place files at nodes according to names Name ID space (DNS-like) –for files and nodes –node name = reverse DNS name of the host (com.microsoft.host1) –file names have same prefix Problem?

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Constrained Load-Balancing Data uniformly distributed in designated subset of nodes –e.g., inside organization

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring SkipNet’s Two Name Spaces Name ID Space Numerical ID Space com.microsoft.host1 h(com.microsoft.host1) non-uniform uniform

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Skip Lists - Reminder In-memory dictionary data structure. –Sorted linked list with a subset of nodes having additional links to skip over many list elements Perfect (deterministic) skip list: –Search: O (log N), N – number of nodes in the list. –Insertion/deletion: expensive/awkward Probabilistic skip list: –Search, Insert, Delete: O (log N) w.h.p.

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Skip List: Good for Us? The Good: –Sorted list: path locality for name-based search –O(log N) search with skip pointers –Up to log(N) skip pointers: O(log N) instertion The Bad: –Lookup starts from root only –Unequal load

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring SkipNet Global View Ring 000 Ring 001 Ring 010 Ring 011 Ring 100 Ring 101 Ring 110 Ring 111 A D M O T Z X V A M T X D O ZV AT M X O Z D V AT M XZ OD V Ring 00Ring 01Ring 10Ring 11 Ring 0 Ring 1 Root Ring Level L = 0 L = 1 L = 2 L = 3 The full SkipNet routing infrastructure for an 8 node system, including the ring labels.

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring SkipNet Structure Skip Graph = Distributed Skip List –Every node belongs to rings at all levels –Search can start at any node –Use doubly linked lists at each level to account for absence of head and tail nodes. Perfect vs. Probabilistic –Perfect : Pointers at level h point to nodes that are exactly 2 h nodes to the left and right. –Probabilistic : A node in level h probabilistically determines which ring it belongs to.

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring SkipNet Routing Tables Level: L = 0 L = 1 L = 2 Ring 00 Ring 01 Ring 10Ring 11 Ring 000 Ring 001 Ring 010 Ring 011 Ring 100 Ring 101 Ring 110 Ring 111 A Root Ring D M O T V X Z Ring 0 A M T X Ring 1 D Z V O O Z AT M X D V A T M X D V Z O L = 3 Node A’s Routing Table

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring An Alternative View Level 2TT 1MX 0DZ SkipNet nodes ordered by name ID. Routing tables of nodes A and V shown. A D M O T Z X V Level 2DD 1ZO 0XT

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Routing By Name ID Routing in Skip Graph = Search in Skip Lists Simple Rule: –Forward the message to node that is closest to destination, without going too far. Route either clockwise/counterclockwise Terminates when messages arrives at a node whose name ID is closest to destination. Number of hops is O(log N) w.h.p.

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Example: Routing from A to V Level: L = 0 L = 1 L = 2 Ring 00 Ring 01 Ring 10Ring 11 Ring 000 Ring 001 Ring 010 Ring 011 Ring 100 Ring 101 Ring 110 Ring 111 A Root Ring D M O T V X Z Ring 0 A M T X Ring 1 D Z V O O Z AT M X D V A T M X D VZ O L = 3

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Example: Routing from A to V Level: L = 0 L = 1 L = 2 Ring 00 Ring 01 Ring 10Ring 11 Ring 000 Ring 001 Ring 010 Ring 011 Ring 100 Ring 101 Ring 110 Ring 111 A Root Ring D M O T V X Z Ring 0 A M T X Ring 1 D Z V O O Z AT M X D V A T M X D V Z O L = 3 Node T’s Routing Table

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Example: Routing from A to V Level: L = 0 L = 1 L = 2 Ring 00 Ring 01 Ring 10Ring 11 Ring 000 Ring 001 Ring 010 Ring 011 Ring 100 Ring 101 Ring 110 Ring 111 A Root Ring D M O T V X Z Ring 0 A M T X Ring 1 D Z V O O Z AT M X D V A T M X D V Z O L = 3

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Example: Routing to Object Level: L = 0 L = 1 L = 2 Route from A to F -> Terminates at E Ring 00 Ring 01 Ring 10Ring 11 Ring 000 Ring 001 Ring 010 Ring 011 Ring 100 Ring 101 Ring 110 Ring 111 A Root Ring D E O V X Z Ring 0 A ETX Ring 1 D Z V O O Z AT E X D V A T E X D VZ O L = 3 T

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Name ID Routing Algorithm SendMsg(nameID, msg) { if( LongestPrefix(nameID,localNode.nameID)==0 ) msg.dir = RandomDirection(); else if( nameID<localNode.nameID ) msg.dir = counterClockwise; else msg.dir = clockwise; msg.nameID = nameID; RouteByNameID(msg); } // Invoked at all nodes (including the source and // destination nodes) along the routing path. RouteByNameID(msg) { // Forward along the longest pointer // that is between us and msg.nameID. h = localNode.maxHeight; while (h >= 0) { nbr = localNode.RouteTable[msg.dir][h]; if (LiesBetween(localNode.nameID, nbr.nameID, msg.nameID, msg.dir)) { SendToNode(msg, nbr); return; } h = h - 1; } // h<0 implies we are the closest node. DeliverMessage(msg.msg); } Load Balancing Path Locality

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Routing By Numeric ID Numeric id’s are random, no ring is sorted by them –We can’t route top-down! Bottom-up Routing –Routing begins at level 0 ring until a node is found whose numeric ID matches the destination numeric ID in the first digit. –Messages forwarded from ring in level h, R h, to a ring in level h+1, R h+1, such that nodes in R h+1 share h+1 digits with destination numeric ID. –Terminates when message delivered, or none the nodes in R h share h+1 digits with destination numeric ID

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Example: Routing by Numeric ID –Hash(“Foo.c”) = 101 Level: L = 0 L = 1 L = 2 Ring 00 Ring 01 Ring 10 Ring 11 Ring000Ring000 Ring 001 Ring 001 Ring 010 Ring 010 Ring 011 Ring 011 Ring 100 Ring 100 Ring 101 Ring 101 Ring 110 Ring 110 Ring 111 Ring 111 Root Ring D M O T V X Z Ring 0 M T X Ring 1 D Z V O O Z AT M X D V A T M X D V Z O L = 3 Foo.c A A

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Routing by Numeric ID The same routing tables are used for routing by nameID and numericID The number of message hops is O(log N) whp What sequential data structure does this search resemble?

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Routing Algorithm // Invoked at all nodes (including the source and destination nodes) along the routing path. // Initially: msg.ringLvl = -1, msg.startNode = msg.bestNode = null & msg.finalDestination = false RouteByNumericID(msg) { if (msg.numID == localNode.numID || msg.finalDestination) { DeliverMessage(msg.msg); return; } if (localNode == msg.startNode) { // Done traversing current ring. msg.finalDestination = true; SendToNode(msg.bestNode); return; } h = CommonPrefixLen(msg.numID, localNode.numID); if (h > msg.ringLvl) { // Found a higher ring. msg.ringLvl = h; msg.startNode = msg.bestNode = localNode; } else if ( abs(localNode.numID - msg.numID) < abs(msg.bestNode.numID - msg.numID)) { // Found a better candidate for current ring. msg.bestNode = localNode; } // Forward along current ring. nbr = localNode.RouteTable[clockWise][msg.ringLvl]; SendToNode(nbr); }

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Routing Summary It all depends on how we look at the routing tables … What is the data structure consisting of all the pointers in the rings that the specific node’s name ID belongs to? –A Skip List! Search is top-down. What is the data structure consisting of all the rings in respect to searching by numeric id? –A Trie! Search is bottom-up. The search in both directions takes O(log N) messages whp. Ready for join/departure procedures?

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Node Join Two-stage process: (1) bottom-up + (2) top-down Bottom-up: find the top level ring that matches the node’s numeric ID. Top-down: build the new node’s routing table –Find a neighbor in the top ring using name ID search. –Starting from this neighbor, search for the name ID at the next lower level and thus find neighbors at lower level. –Repeated until the search reaches the root. Update of the existing nodes’ routing tables: –after the new node has joined the root ring.

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Node join illustrated Ring P0 Ring P1 Ring P Only a few in expectation Joining node

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Node Join - Analysis Key ideas: –Climb to a weakly populated ring. –Search for the node’s neighbors at the lower levels only after finding the neighbors at the higher levels. –The range of traversed nodes at the level = the range of neighbors at the next higher level. Insertion traverses O(log N) hops whp –Expected O(log N) levels, constant number of neighbors at each level.

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Node Departure/Failure Graceful (notified) vs crash departure Key issue –routing tables’ update Key idea – separate vital info from optimizations –Routing is correct as long as the root level ring is maintained. –Other levels regarded as optimization hints –Does this remind something? Upper-ring membership maintained through a background repair process.

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Leaf Sets Idea = use redundant pointers at level 0: –Protect from independent failures –Improve the search performance Store L/2 pointers in every direction SkipNet uses L=16 –Not an original SkipNet idea – used in Pastry.

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Constrained Load Balancing (CLB) Multiple DHTs with differing scopes using a single SkipNet structure –A result of the ability to route in both address spaces Divide data object names into two parts with ! CLB Domain CLB Suffix microsoft.com ! skipnet.html Numeric Routing Name Routing microsoft.com/skipnet.html! – controlled placement !microsoft.com/skipnet.html – Global DHT

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring CLB Example File ID = “com.microsoft ! skipnet.html” –Route by name ID to com.microsoft –Inside com.microsoft, route by numeric ID to hash(“skipnet.html”) com.sun edu.ucb gov.irs com.microsoft skipnet. html

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring SkipNet Path Locality Organizations correspond to contiguous SkipNet segments –Internal routing by NameID remains internal Nodes have left / right pointers com.sun edu.ucb gov.irs com.microsoft com.microsoft.research

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Course Summary

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Main Topics State machine replication for consistency and availability. –Uses Atomic Broadcast. –Uses Consensus. Asynchronous Message-Passing Models –Consensus impossible –Solvable with eventual synchrony, failure detectors <>S,  –In two communication rounds in “fast” case Shared memory –Convenient model –Can be emulated using message-passing –Good for “data-centric” replication

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Course Summary (What I Hope You Learned…) Distributed systems are subtle. –It’s very easy to get things wrong –Lesson: don’t design a distributed system without proving the algorithm first! Redundancy is the key to reliability. –Multiple replicas: 2t+1, 3t+1, etc.. Strong consistency is attainable but costly and has scalability limitations.

Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring Have a Great Summer!