1. Tapestry: Scalable and Fault-tolerant Routing and Location Stanford Networking Seminar October 2001 Ben Y. Zhao ravenben@eecs.berkeley.edu

2. Stanford Networking Seminar, 10/20012 Challenges in the Wide-area Trends: Exponential growth in CPU, storage Network expanding in reach and b/w Can applications leverage new resources? Scalability: increasing users, requests, traffic Resilience: more components  inversely low MTBF Management: intermittent resource availability  complex management schemes Proposal: an infrastructure that solves these issues and passes benefits onto applications

3. Stanford Networking Seminar, 10/20013 Driving Applications Leverage of cheap & plentiful resources: CPU cycles, storage, network bandwidth Global applications share distributed resources Shared computation: SETI, Entropia Shared storage OceanStore, Gnutella, Scale-8 Shared bandwidth Application-level multicast, content distribution networks

4. Stanford Networking Seminar, 10/20014 Key: Location and Routing Hard problem: Locating and messaging to resources and data Goals for a wide-area overlay infrastructure Easy to deploy Scalable to millions of nodes, billions of objects Available in presence of routine faults Self-configuring, adaptive to network changes Localize effects of operations/failures

5. Stanford Networking Seminar, 10/20015 Talk Outline Motivation Tapestry overview Fault-tolerant operation Deployment / evaluation Related / ongoing work

6. Stanford Networking Seminar, 10/20016 What is Tapestry? A prototype of a decentralized, scalable, fault- tolerant, adaptive location and routing infrastructure (Zhao, Kubiatowicz, Joseph et al. U.C. Berkeley) Network layer of OceanStore Routing: Suffix-based hypercube Similar to Plaxton, Rajamaran, Richa (SPAA97) Decentralized location: Virtual hierarchy per object with cached location references Core API: publishObject(ObjectID, [serverID]) routeMsgToObject(ObjectID) routeMsgToNode(NodeID)

7. Stanford Networking Seminar, 10/20017 Routing and Location Namespace (nodes and objects) 160 bits  2 80 names before name collision Each object has its own hierarchy rooted at Root f (ObjectID) = RootID, via a dynamic mapping function Suffix routing from A to B At h th hop, arrive at nearest node hop(h) s.t. hop(h) shares suffix with B of length h digits Example: 5324 routes to 0629 via 5324  2349  1429  7629  0629 Object location: Root responsible for storing object’s location Publish / search both route incrementally to root

8. Stanford Networking Seminar, 10/20018 Publish / Lookup Publish object with ObjectID: // route towards “virtual root,” ID=ObjectID For (i=0, i<Log 2 (N), i+=j) { //Define hierarchy j is # of bits in digit size, (i.e. for hex digits, j = 4 ) Insert entry into nearest node that matches on last i bits If no matches found, deterministically choose alternative Found real root node, when no external routes left Lookup object Traverse same path to root as publish, except search for entry at each node For (i=0, i<Log 2 (N), i+=j) { Search for cached object location Once found, route via IP or Tapestry to object

9. Stanford Networking Seminar, 10/20019 4 2 3 3 3 2 2 1 2 4 1 2 3 3 1 3 4 1 1 43 2 4 Tapestry Mesh Incremental suffix-based routing NodeID 0x43FE NodeID 0x13FE NodeID 0xABFE NodeID 0x1290 NodeID 0x239E NodeID 0x73FE NodeID 0x423E NodeID 0x79FE NodeID 0x23FE NodeID 0x73FF NodeID 0x555E NodeID 0x035E NodeID 0x44FE NodeID 0x9990 NodeID 0xF990 NodeID 0x993E NodeID 0x04FE NodeID 0x43FE

10. Stanford Networking Seminar, 10/200110 Routing in Detail 5712 0880 3210 7510 4510 Neighbor Map For “5712” (Octal) Routing Levels 1234 xxx1 5712 xxx0 xxx3 xxx4 xxx5 xxx6 xxx7 xx02 5712 xx22 xx32 xx42 xx52 xx62 xx72 x012 x112 x212 x312 x412 x512 x612 5712 0712 1712 2712 3712 4712 5712 6712 7712 5712 012 3 45 6 7 0880 012 3 45 6 7 3210 012 3 45 6 7 4510 012 3 45 6 7 7510 012 3 45 6 7 Example: Octal digits, 2 12 namespace, 5712  7510

11. Stanford Networking Seminar, 10/200111 Object Location Randomization and Locality

12. Stanford Networking Seminar, 10/200112 Talk Outline Motivation Tapestry overview Fault-tolerant operation Deployment / evaluation Related / ongoing work

13. Stanford Networking Seminar, 10/200113 Fault-tolerant Location Minimized soft-state vs. explicit fault-recovery Redundant roots Object names hashed w/ small salts  multiple names/roots Queries and publishing utilize all roots in parallel P(finding reference w/ partition) = 1 – (1/2) n where n = # of roots Soft-state periodic republish 50 million files/node, daily republish, b = 16, N = 2 160, 40B/msg, worst case update traffic: 156 kb/s, expected traffic w/ 2 40 real nodes: 39 kb/s

14. Stanford Networking Seminar, 10/200114 Fault-tolerant Routing Strategy: Detect failures via soft-state probe packets Route around problematic hop via backup pointers Handling: 3 forward pointers per outgoing route (2 backups) 2 nd chance algorithm for intermittent failures Upgrade backup pointers and replace Protocols: First Reachable Link Selection (FRLS) Proactive Duplicate Packet Routing

15. Stanford Networking Seminar, 10/200115 Summary Decentralized location and routing infrastructure Core routing similar to PRR97 Distributed algorithms for object-root mapping, node insertion / deletion Fault-handling with redundancy, soft-state beacons, self-repair Decentralized and scalable, with locality Analytical properties Per node routing table size: bLog b (N) N = size of namespace, n = # of physical nodes Find object in Log b (n) overlay hops

16. Stanford Networking Seminar, 10/200116 Talk Outline Motivation Tapestry overview Fault-tolerant operation Deployment / evaluation Related / ongoing work

17. Stanford Networking Seminar, 10/200117 Deployment Status Java Implementation in OceanStore Running static Tapestry Deploying dynamic Tapestry with fault- tolerant routing Packet-level simulator Delay measured in network hops No cross traffic or queuing delays Topologies: AS, MBone, GT-ITM, TIERS ns2 simulations

18. Stanford Networking Seminar, 10/200118 Evaluation Results Cached object pointers Efficient lookup for nearby objects Reasonable storage overhead Multiple object roots Improves availability under attack Improves performance and perf. stability Reliable packet delivery Redundant pointers approximate optimal reachability FRLS, a simple fault-tolerant UDP protocol

19. Stanford Networking Seminar, 10/200119 First Reachable Link Selection Use periodic UDP packets to gauge link condition Packets routed to shortest “good” link Assumes IP cannot correct routing table in time for packet delivery ABCDEABCDE IP Tapestry No path exists to dest.

20. Stanford Networking Seminar, 10/200120 Talk Outline Motivation Tapestry overview Fault-tolerant operation Deployment / evaluation Related / ongoing work

21. Stanford Networking Seminar, 10/200121 Bayeux Global-scale application-level multicast (NOSSDAV 2001) Scalability Scales to > 10 5 nodes Self-forming member group partitions Fault tolerance Multicast root replication FRLS for resilient packet delivery More optimizations Group ID clustering for better b/w utilization

22. Stanford Networking Seminar, 10/200122 13FE ABFE 1290 239E 73FE 423E 793E 44FE 9990 F990 993E 04FE 093E 29FE F9FE 79FE 555E 035E 23FE 43FE Multicast Root Receiver 793E 79FE 793E 79FE 793E 79FE Bayeux: Multicast

23. Stanford Networking Seminar, 10/200123 13FE ABFE 1290 239E 73FE 423E 793E 44FE 9990 F990 993E 04FE 093E 29FE F9FE 79FE 555E 035E 23FE 43FE Multicast Root Multicast Root Receiver JOIN Bayeux: Tree Partitioning JOIN

24. Stanford Networking Seminar, 10/200124 Overlay Routing Networks CAN: Ratnasamy et al., (ACIRI / UCB) Uses d-dimensional coordinate space to implement distributed hash table Route to neighbor closest to destination coordinate Chord: Stoica, Morris, Karger, et al., (MIT / UCB) Linear namespace modeled as circular address space “Finger-table” point to logarithmic # of inc. remote hosts Pastry: Rowstron and Druschel (Microsoft / Rice ) Hypercube routing similar to PRR97 Objects replicated to servers by name Fast Insertion / Deletion Constant-sized routing state Unconstrained # of hops Overlay distance not prop. to physical distance Simplicity in algorithms Fast fault-recovery Log 2 (N) hops and routing state Overlay distance not prop. to physical distance Fast fault-recovery Log(N) hops and routing state Data replication required for fault-tolerance

25. Stanford Networking Seminar, 10/200125 Ongoing Research Fault-tolerant routing Reliable Overlay Networks (MIT) Fault-tolerant Overlay Routing (UCB) Application-level multicast Bayeux (UCB), CAN (AT&T), Scribe and Herald (Microsoft) File systems OceanStore (UCB) PAST (Microsoft / Rice) Cooperative File System (MIT)

26. Stanford Networking Seminar, 10/200126 For More Information Tapestry: http://www.cs.berkeley.edu/~ravenben/tapestry OceanStore: http://oceanstore.cs.berkeley.edu Related papers: http://oceanstore.cs.berkeley.edu/publications http://www.cs.berkeley.edu/~ravenben/publications ravenben@cs.berkeley.edu