1. Tapestry: Scalable and Fault-tolerant Routing and Location
Stanford Networking Seminar October 2001
Ben Y. Zhao

2. Challenges in the Wide-area
Trends:
Exponential growth in CPU, storage
Network expanding in reach and b/w
Can applications leverage new resources?
Scalability: increasing users, requests, traffic
Resilience: more components  inversely low MTBF
Management: intermittent resource availability  complex management schemes
Proposal: an infrastructure that solves these issues and passes benefits onto applications
Stanford Networking Seminar, 10/2001

3. Driving Applications
Leverage of cheap & plentiful resources: CPU cycles, storage, network bandwidth
Global applications share distributed resources
Shared computation:
SETI, Entropia
Shared storage
OceanStore, Gnutella, Scale-8
Shared bandwidth
Application-level multicast, content distribution networks
Stanford Networking Seminar, 10/2001

4. Key: Location and Routing
Hard problem:
Locating and messaging to resources and data
Goals for a wide-area overlay infrastructure
Easy to deploy
Scalable to millions of nodes, billions of objects
Available in presence of routine faults
Self-configuring, adaptive to network changes
Localize effects of operations/failures
Stanford Networking Seminar, 10/2001

5. Talk Outline Motivation Tapestry overview Fault-tolerant operation
Deployment / evaluation
Related / ongoing work
Stanford Networking Seminar, 10/2001

6. What is Tapestry?
A prototype of a decentralized, scalable, fault-tolerant, adaptive location and routing infrastructure (Zhao, Kubiatowicz, Joseph et al. U.C. Berkeley)
Network layer of OceanStore
Routing: Suffix-based hypercube
Similar to Plaxton, Rajamaran, Richa (SPAA97)
Decentralized location:
Virtual hierarchy per object with cached location references
Core API:
publishObject(ObjectID, [serverID])
routeMsgToObject(ObjectID)
routeMsgToNode(NodeID)
Stanford Networking Seminar, 10/2001

7. Routing and Location Namespace (nodes and objects)
160 bits  280 names before name collision
Each object has its own hierarchy rooted at Root
f (ObjectID) = RootID, via a dynamic mapping function
Suffix routing from A to B
At hth hop, arrive at nearest node hop(h) s.t.
hop(h) shares suffix with B of length h digits
Example: 5324 routes to 0629 via 5324  2349  1429  7629  0629
Object location:
Root responsible for storing object’s location
Publish / search both route incrementally to root
Stanford Networking Seminar, 10/2001

8. Publish / Lookup Publish object with ObjectID: Lookup object
// route towards “virtual root,” ID=ObjectID
For (i=0, i<Log2(N), i+=j) { //Define hierarchy
j is # of bits in digit size, (i.e. for hex digits, j = 4 )
Insert entry into nearest node that matches on last i bits
If no matches found, deterministically choose alternative
Found real root node, when no external routes left
Lookup object
Traverse same path to root as publish, except search for entry at each node
For (i=0, i<Log2(N), i+=j) {
Search for cached object location
Once found, route via IP or Tapestry to object
Stanford Networking Seminar, 10/2001

9. Tapestry Mesh Incremental suffix-based routing4 2 3 1
NodeID
0x43FE
NodeID
0x79FE
NodeID
0x23FE
NodeID
0x993E
NodeID
0x43FE
NodeID
0x73FE
NodeID
0x44FE
NodeID
0xF990
NodeID
0x035E
NodeID
0x04FE
NodeID
0x13FE
NodeID
0x555E
NodeID
0xABFE
NodeID
0x9990
NodeID
0x239E
NodeID
0x73FF
NodeID
0x1290
NodeID
0x423E
Stanford Networking Seminar, 10/2001

10. Routing in Detail Example: Octal digits, 212 namespace, 5712  7510
Neighbor Map
For “5712” (Octal)
Routing Levels 1 2 3 4
xxx1
5712
xxx0
xxx3
xxx4
xxx5
xxx6
xxx7
xx02
xx22
xx32
xx42
xx52
xx62
xx72
x012
x112
x212
x312
x412
x512
x612
0712
1712
2712
3712
4712
6712
7712
5712
5712 1 2 3 4 5 6 7
0880 1 2 3 4 5 6 7
0880
3210 1 2 3 4 5 6 7
3210
4510 1 2 3 4 5 6 7
4510
7510 1 2 3 4 5 6 7
7510
Stanford Networking Seminar, 10/2001

11. Object Location Randomization and Locality
Stanford Networking Seminar, 10/2001

12. Talk Outline Motivation Tapestry overview Fault-tolerant operation
Deployment / evaluation
Related / ongoing work
Stanford Networking Seminar, 10/2001

13. Fault-tolerant Location
Minimized soft-state vs. explicit fault-recovery
Redundant roots
Object names hashed w/ small salts  multiple names/roots
Queries and publishing utilize all roots in parallel
P(finding reference w/ partition) = 1 – (1/2)n where n = # of roots
Soft-state periodic republish
50 million files/node, daily republish, b = 16, N = 2160 , 40B/msg, worst case update traffic: 156 kb/s,
expected traffic w/ 240 real nodes: 39 kb/s
Stanford Networking Seminar, 10/2001

14. Fault-tolerant Routing
Strategy:
Detect failures via soft-state probe packets
Route around problematic hop via backup pointers
Handling:
3 forward pointers per outgoing route (2 backups)
2nd chance algorithm for intermittent failures
Upgrade backup pointers and replace
Protocols:
First Reachable Link Selection (FRLS)
Proactive Duplicate Packet Routing
Stanford Networking Seminar, 10/2001

15. Summary Decentralized location and routing infrastructure
Core routing similar to PRR97
Distributed algorithms for object-root mapping, node insertion / deletion
Fault-handling with redundancy, soft-state beacons, self-repair
Decentralized and scalable, with locality
Analytical properties
Per node routing table size: bLogb(N)
N = size of namespace, n = # of physical nodes
Find object in Logb(n) overlay hops
Stanford Networking Seminar, 10/2001

16. Talk Outline Motivation Tapestry overview Fault-tolerant operation
Deployment / evaluation
Related / ongoing work
Stanford Networking Seminar, 10/2001

17. Deployment Status Java Implementation in OceanStore
Running static Tapestry
Deploying dynamic Tapestry with fault-tolerant routing
Packet-level simulator
Delay measured in network hops
No cross traffic or queuing delays
Topologies: AS, MBone, GT-ITM, TIERS
ns2 simulations
Stanford Networking Seminar, 10/2001

18. Evaluation Results Cached object pointers Multiple object roots
Efficient lookup for nearby objects
Reasonable storage overhead
Multiple object roots
Improves availability under attack
Improves performance and perf. stability
Reliable packet delivery
Redundant pointers approximate optimal reachability
FRLS, a simple fault-tolerant UDP protocol
Stanford Networking Seminar, 10/2001

19. First Reachable Link Selection
Use periodic UDP packets to gauge link condition
Packets routed to shortest “good” link
Assumes IP cannot correct routing table in time for packet delivery
IP Tapestry A B C
D E
No path exists to dest.
Stanford Networking Seminar, 10/2001

20. Talk Outline Motivation Tapestry overview Fault-tolerant operation
Deployment / evaluation
Related / ongoing work
Stanford Networking Seminar, 10/2001

21. Bayeux Global-scale application-level multicast (NOSSDAV 2001)
Scalability
Scales to > 105 nodes
Self-forming member group partitions
Fault tolerance
Multicast root replication
FRLS for resilient packet delivery
More optimizations
Group ID clustering for better b/w utilization
Stanford Networking Seminar, 10/2001

22. Bayeux: Multicast 79FE 23FE 993E F9FE 43FE 44FE 73FE F990 29FE 035E
Receiver
23FE
993E
793E
79FE
F9FE
43FE
44FE
73FE
F990
29FE
035E
04FE
13FE
555E
ABFE
9990
Fanout in the tree is bounded by the node ID base
Height of the tree is bounded by the # of digits in the node IDs
793E
239E
1290
093E
423E
Receiver
Multicast
Root
Stanford Networking Seminar, 10/2001

23. Bayeux: Tree Partitioning
JOIN
79FE
43FE
Receiver
23FE
993E
F9FE
43FE
44FE
73FE
F990
29FE
035E
04FE
13FE
Tree Partitioning Algorithm
Integrate Bayeux multicast root nodes into Tapestry network
Name an object O with the hash of the multicast session name, and place O on each multicast root
Each multicast root advertises O in Tapestry
New member M uses Tapestry location service to route a JOIN message to the nearest multicast root node R
R sends TREE message to M, now a member of R’s receiver set
555E
ABFE
Multicast
Root
9990
JOIN
793E
239E
1290
093E
423E
Stanford Networking Seminar, 10/2001
Multicast
Root
Receiver

24. Overlay Routing Networks
CAN: Ratnasamy et al., (ACIRI / UCB)
Uses d-dimensional coordinate space to implement distributed hash table
Route to neighbor closest to destination coordinate
Chord: Stoica, Morris, Karger, et al., (MIT / UCB)
Linear namespace modeled as circular address space
“Finger-table” point to logarithmic # of inc. remote hosts
Pastry: Rowstron and Druschel (Microsoft / Rice )
Hypercube routing similar to PRR97
Objects replicated to servers by name
Fast Insertion / Deletion
Constant-sized routing state
Unconstrained # of hops
Overlay distance not prop. to physical distance
Simplicity in algorithms
Fast fault-recovery
Log2(N) hops and routing state
Overlay distance not prop. to physical distance
Log(N) hops and routing state
Data replication required for fault-tolerance
Stanford Networking Seminar, 10/2001

25. Ongoing Research Fault-tolerant routing Application-level multicast
Reliable Overlay Networks (MIT)
Fault-tolerant Overlay Routing (UCB)
Application-level multicast
Bayeux (UCB), CAN (AT&T), Scribe and Herald (Microsoft)
File systems
OceanStore (UCB)
PAST (Microsoft / Rice)
Cooperative File System (MIT)
Stanford Networking Seminar, 10/2001

26. For More Information Tapestry: OceanStore: Related papers:
OceanStore:
Related papers:
Stanford Networking Seminar, 10/2001