1. An Overlay Infrastructure for Decentralized Object Location and Routing
Ben Y. Zhao
University of California at Santa Barbara

2. Structured Peer-to-Peer Overlays
Node IDs and keys from randomized namespace (SHA-1)
incremental routing towards destination ID
each node has small set of outgoing routes, e.g. prefix routing
log (n) neighbors per node, log (n) hops between any node pair
ID: ABCE
ABC0
To: ABCD
AB5F
A930
November 8, 2018

3. Related Work Unstructured Peer to Peer Approaches
Napster, Gnutella, KaZaa
probabilistic search (optimized for the hay, not the needle)
locality-agnostic routing (resulting in high network b/w costs)
Structured Peer to Peer Overlays
the first protocols (2001): Tapestry, Pastry, Chord, CAN
then: Kademlia, SkipNet, Viceroy, Symphony, Koorde, Ulysseus…
distinction: how to choose your neighbors
Tapestry, Pastry: latency-optimized routing mesh
distinction: application interface
distributed hash table: put (key, data); data = get (key);
Tapestry: decentralized object location and routing
November 8, 2018

4. Chord NodeIDs are numbers on ring
Closeness defined by numerical proximity
Finger table
keep routes for next node 2i away in namespace
routing table size: log2 n
n = total # of nodes
Routing
iterative hops from source
at most log2 n hops
Node 0/1024
896
128
768
256
640
384
512
November 8, 2018

5. Chord II Pros Cons Application Interface: simplicity
limited flexibility in routing
neighbor choices unrelated to network proximity * but can be optimized over time
Application Interface:
distributed hash table (DHash)
November 8, 2018

6. Tapestry / Pastry incremental prefix routing routing table routing
11110XXX00XX 000X0000
routing table
keep nodes matching at least i digits to destination
table size: b * logb n
routing
recursive routing from source
at most logb n hops
Node 0/1024
896
128
768
256
640
384
512
November 8, 2018

7. Routing in Detail Example: Octal digits, 212 namespace, 2175  0157
Neighbor Map
For “2175” (Octal)
Routing Levels 1 2 3 4
2171
2172
2170
2173
2174
----
2176
2177
210x
211x
212x
213x
214x
215x
216x
20xx
22xx
23xx
24xx
25xx
26xx
27xx
0xxx
1xxx
3xxx
4xxx
5xxx
6xxx
7xxx
2175
2175 1 2 3 4 5 6 7
0880 1 2 3 4 5 6 7
0880
0123 1 2 3 4 5 6 7
0123
0154 1 2 3 4 5 6 7
0154
0157 1 2 3 4 5 6 7
0157
November 8, 2018

8. Tapestry / Pastry II Pros Cons Application Interface
large flexibility in neighbor choice choose nodes closest in physical distance
can tune routing table size and routing hops using parameter b
Cons
more complex than Chord to implement / understand
Application Interface
Tapestry: decentralized object location
Pastry: distributed hash table
November 8, 2018

9. Talk Outline Motivation and background What makes Tapestry different
Tapestry deployment performance
Wrap-up
November 8, 2018

10. So What Makes Tapestry Different?
It’s all about performance
Proximity routing
leverage flexibility in routing rules
for each routing table entry, choose node
that satisfies prefix requirement
and is closest in network latency
result: end to end latency “proportional” to actual IP latency
DOLR interface
applications choose where to place objects
use application-level knowledge to optimize access time
November 8, 2018

11. Why Proximity Routing?
01234
01234
Fewer/shorter IP hops: shorter e2e latency, less bandwidth/congestion, less likely to cross broken/lossy links
November 8, 2018

12. Performance Impact (Proximity)
Simulated Tapestry w/ and w/o proximity on 5000 node transit-stub network
Measure pair-wise routing stretch between 200 random nodes
November 8, 2018

13. Decentralized Object Location & Routing
routeobj(k)
backbone
routeobj(k) k
publish(k) k
where objects are placed is orthogonal, we’re providing a slightly lower level abstraction and allowing application to place data. data placement strategy is area of research
redirect data traffic using log(n) in-network redirection pointers
average # of pointers/machine: log(n) * avg files/machine
keys to performance
proximity-enabled routing mesh with routing convergence
November 8, 2018

14. DOLR vs. Distributed Hash Table
DHT: hash content  name  replica placement
modifications  replicating new version into DHT
DOLR: app places copy near requests, overlay routes msgs to it
November 8, 2018

15. Performance Impact (DOLR)
simulated Tapestry w/ DOLR and DHT interfaces on 5000 node T-S
measure route to object latency from clients in 2 stub networks
DHT: 5 object replicas DOLR: 1 replica placed in each stub network
November 8, 2018

16. Weaving a Tapestry inserting node (0123) into network
route to own ID, find 012X nodes, fill last column
request backpointers to 01XX nodes
measure distance, add to rTable
prune to nearest K nodes
repeat 2—4
ID = 0123
XXXX
0XXX
01XX
012X
1XXX
2XXX
3XXX
00XX
02XX
03XX
010X
011X
013X
0120
0121
0122
Existing Tapestry
November 8, 2018

17. Talk Outline Motivation and background What makes Tapestry different
Tapestry deployment performance
Wrap-up
November 8, 2018

18. Implementation Performance
Java implementation
lines in core Tapestry, downloads
Micro-benchmarks
per msg overhead: ~ 50s, most latency from byte copying
performance scales w/ CPU speedup
5KB msgs on P-IV 2.4Ghz: throughput ~ 10,000 msgs/sec
Routing stretch
route to node: < 2
route to objects/endpoints: < 3 higher stretch for close by objects
November 8, 2018

19. Stability Under Membership Changes
kill nodes
constant churn
large group join
success rate (%)
Routing operations on 40 node Tapestry cluster
Churn: nodes join/leave every 10 seconds, average lifetime = 2mins
November 8, 2018

20. Micro-benchmark Methodology
Sender Control
Receiver Control
LAN Link
Tapestry
Tapestry
Experiment run in LAN, GBit Ethernet
Sender sends messages at full speed
Measure inter-arrival time for last msgs
10000 msgs: remove cold-start effects
50000 msgs: remove network jitter effects
November 8, 2018

21. Micro-benchmark Results (LAN)
100mb/s
Per msg overhead ~ 50s, latency dominated by byte copying
Performance scales with CPU speedup
For 5K messages, throughput = ~10,000 msgs/sec
November 8, 2018

22. Large Scale Methodology
PlanetLab global network
500 machines at 100+ institutions, in North America, Europe, Australia, Asia, Africa
1.26Ghz PIII (1GB RAM), 1.8Ghz P4 (2GB RAM)
North American machines (2/3) on Internet2
Tapestry Java deployment
6-7 nodes on each physical machine
IBM Java JDK 1.30
Node virtualization inside JVM and SEDA
Scheduling between virtual nodes increases latency
November 8, 2018

23. Node to Node Routing (PlanetLab)
Median=31.5, 90th percentile=135
Ratio of end-to-end latency to ping distance between nodes
All node pairs measured, placed into buckets
November 8, 2018

24. Latency to Insert Node
Latency to dynamically insert a node into an existing Tapestry, as function of size of existing Tapestry
Humps due to expected filling of each routing level
November 8, 2018

25. Thanks!
Questions, comments?

26. Object Location (PlanetLab)
90th percentile=158
Ratio of end-to-end latency to client-object ping distance
Local-area stretch improved w/ additional location state
November 8, 2018

27. Bandwidth to Insert Node
Cost in bandwidth of dynamically inserting a node into the Tapestry, amortized for each node in network
Per node bandwidth decreases with size of network
November 8, 2018