Weaving a Tapestry Distributed Algorithms for Secure Node Integration, Routing and Fault Handling Ben Y. Zhao (John Kubiatowicz, Anthony Joseph) Fault-tolerant Computing Fall 2000

Why Tapestry? Distributed systems scaling to WAN Larger scale  frequent component faults More data + centralization  performance bottleneck Dynamic environment  manageability complexity More principals  attacks on system (e.g. DoS) more likely Tapestry: Decentralized approach to location and routing focusing on fault-resilience and adaptability Builds on previous work: Plaxton trees

Background: Tapestry/Plaxton Decentralized Routing Local neighbor pointers Finite storage overhead Wide-area Location Each obj maps to “root” node Store backpointers en route to root node Exploits locality Neighbor Map For “6271” (Octal) Routing Levels xxx2 xxx0 xxx3 xxx4 xxx5 xxx6 xxx7 xx01 xx11 xx21 xx31 xx41 xx51 xx x071 x x371 x471 x571 x671 x

New Tapestry Mechanisms Sibling Mesh Nodes in Sibling Mesh of level N share common suffix of length N Neighbors of level N+1 are siblings of level N Clusters connected w/in, possibly disconnected between regions Gateway Nodes Nodes that also serve as integration points Integrates new nodes w/in coverage area

Issues Routing to non-existent node Ids Node integration into Tapestry Populate neighbor maps, divert relevant traffic Goals:  Low latency  Limit stress on system/nodes  Prevent/Limit Denial of Service attacks  Approximate optimal mapping Fault-handling: Fast detection, avoidance, and recovery

Surrogate Routing Messages to non-existent node go to “surrogate” When routing hits null entry in neighbor map … Plaxton algorithm: Find global set of all nodes matching on most # of suffix bits to destination ID Use global ordering to choose 1 determinstically One hop to surrogate Tapestry distributed algorithm: Find local set of routes matching on most suffix bits Deterministically (via pseudo-random hash) choose an existing alternate; route on Terminate when local node is only entry in neighbor map

Surrogate Overhead Additional hops after Plaxton version terminates Function of how even sparseness spreads through the namespace Probabilistic reasoning Assumption: even ID distribution in space If perfectly evenly spaced names in namespace: overhead =< 1 Overhead = function of skew, with high probability is very small constant

Node Weaving Algorithm Phase I: Given desired Guid g, nearby “gateway” Ask gateway to route to g, return hops Router for hop i+1 is sibling at level i For the i th hop router Copy i th neighbor map For each entry E in neighbor list,  Look at E’s higher order siblings  Traverse sibling mesh until local optima reached

Node Weaving Phase I X Gateway Node New Node Optimize

Node Weaving Phase II Notify “nearby” nodes to divert traffic Recognize Neighbor(i+1) = Sibling(i) For each route level Send NotifyMsg to neighbors Msg forwarded w/ small TTL Intuition: Only local nodes need notification Probability dictates distant nodes have better alternative  no notify necessary X x x x xx x xx x x x x Route Notify

Denial of Service Scenario #1 Generate large # of Guids, weave each in Soln: Attach cost function to use of each Guid Key = Bit Sequence L such that  SHA-1(Guid+L+S) = [ ]+[…]  S = random string refreshed periodically per server Solving inv(SHA-1) is costly Showing L with request to integrate is enough Scenario #2 Weave same Guid in using large # of Gateways Soln: Allow Gateways to arbitrarily limit coverage area

Fault Handling Detection: Routing: heartbeats to immediate neighbors Location: inventory beats to routers Resilience: Use secondary neighbors/siblings to route around failed links/servers Repair: Failed nodes marked with “invalid” flag Periodic probes (piggyback msgs) Success  Change back to active status

In Progress Simulations: Java: Simulation of dynamic Tapestry C: Simulation of perfect topology with SGB library Algorithms: Theoretical analysis:  Expected insertion latencies  Optimality of dynamic insertions  Optimality of overlay network distance Routing:  Better routing using landmarks