1. PASTRY

2. Sources Pastry paper Pastry Homepage
“Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems” by Antony Rowstron (Microsoft Research) and Peter Druschel (Rice University), IFIP/ACM International Conference on Distributed Systems Platforms (Middleware), Heidelberg, Germany, pages , November, 2001
Pastry Homepage

3. Related work Chord [Sigcomm’01] CAN [Sigcomm’01]
Tapestry [TR UCB/CSD ]
PNRP [unpub.]
Viceroy [PODC ’02]
Kademlia [IPTPS ’02]
Small World [Kleinberg ‘99, ‘00]
Plaxton Trees [Plaxton et al. ‘97]
Generalized Hypercube [Bhuyan et al. ‘84]

4. Scalable, fault resilient, self-organizing,
Pastry
Generic p2p location and routing substrate (DHT)
Self-organizing overlay network (join, departures, locality repair)
Consistent hashing
Lookup/insert object in < log2b N routing steps (expected)
O(log N) per-node state
Network locality heuristics
Scalable, fault resilient, self-organizing,
locality aware, secure
Why do we say Pastry is secure?

5. Pastry: Object distribution O
Consistent hashing
128 bit circular id space
nodeIds (uniform random)
objIds/keys (uniform random)
Invariant: node with numerically closest nodeId maintains object
objId/key
Each node has a randomly assigned 128-bit nodeId, circular namespace
Basic operation: A message with key X, sent by any Pastry node, is delivered
to the live node with nodeId closest to X in at most log16 N steps (barring node failures).
Pastry uses a form of generalized hypercube routing, where the routing tables are initialized and updated dynamically.
nodeIds

6. Pastry: Object insertion/lookup O
Msg with key X is routed to live node with nodeId closest to X
Problem: complete routing table not feasible X
Each node has a randomly assigned 128-bit nodeId, circular namespace
Basic operation: A message with key X, sent by any Pastry node, is delivered
to the live node with nodeId closest to X in at most log16 N steps (barring node failures).
Pastry uses a form of generalized hypercube routing, where the routing tables are initialized and updated dynamically.
Route(X)

7. Pastry Node
Represented by 128-bit randomly chosen nodeId (Hash of IP or public key)
NodeId is in base 2b (b is a configuration parameter; b typical value 2 or 4)
Evenly distributed nodeIds along the circular namespace ( – 1 space).
Routes a message in O(log N) steps to destination
N: size of network
Node state contains:
Leaf Set ( L )
Routing table ( R )
Neighborhood Set ( M )
CMPT 880: P2P Systems - SFU

8. Pastry node state
Leaf set: L/2 Numerically closest nodes (L is a configuration parameter = 16, 32 typically )
Routing Table (Prefix-based)
Neighborhood Set: M physically closest nodes

9. Pastry node state (Leaf Set)
Serves as a fall back for routing table and contains:
L/2 numerically closest and larger nodeIds
L/2 numerically closest and smaller nodIds
Size of L is typically 2b or 2 x 2b
Nodes in L are numerically close (could be geographically diverse)
Fall back= rely on=helper
Whenever the object key is among the leaf set, the routing process can be stopped.

10. Pastry node state: Neighborhood set (M)
Contains the IP addresses and nodeIds of closest nodes according to proximity metric
Size of |M| is typically 2b or 2x2b
Not used in routing, but instead for maintaining locality properties

11. Node state: Routing Table
Matrix of Log2b N rows and 2b – 1 columns (N is the number of nodes in the network)
Entries in row n match the first n digits of current nodeId (prefix: from left to right)
Column number follows matched digits: Format: matched digits–column number–rest of ID
Log2b N populated on average

12. Node (2), (b = 2, L= 8) 1 2 3

13. Pastry: Routing Tradeoff O(log N) routing table size
2b * log2bN + 2L
O(log N) message forwarding steps

14. Prefix Routing Node IDs and keys from randomized namespace (SHA-1)
incremental routing towards destination ID
each node has small set of outgoing routes
log (n) neighbors per node, log (n) hops between any node pair
ID: ABCE
ABC0
To: ABCE
AB5F
A930

15. Pastry: Routing table (# 10233102)
L nodes in
leaf set
log2b N Rows
(actually
log2b 2128= 128/b)
2b columns
L neighbors
b=2,
l=2^b=4

16. Pastry: Routing procedure
(1) Node is in the leaf set
(2) Forward message to a closer node (Better match)
(3) Forward towards numerically
Closer node (not a better match)
A is the current node.
D: Message Key
Li: ith closest NodeId in leaf set
shl(A, B): Length of prefix shared
by nodes A and B
Rij: (j, i)th entry of routing table

17. Pastry: Routing procedure
If (destination is within range of our leaf set)
forward to numerically closest member
else
let l = length of shared prefix
let d = value of l-th digit in D’s address
if (Rld exists)
forward to Rld
forward to a known node (from ) that
(a) shares at least as long a prefix
(b) is numerically closer than this node

18. Pastry: Routing procedure
If message with key D is within range of leaf set, forward to numerically closest leaf
Else forward to node that shares at least one more digit with D in its prefix than current nodeId
If no such node exists, forward to node that shares at least as many digits with D as current nodeId but numerically nearer than current nodeId
CMPT 880: P2P Systems - SFU

19. Pastry: Routing Properties log2b N steps O(log N) state d471f1 d467c4
d462ba
d46a1c
d4213f
Properties
log2b N steps
O(log N) state
Each node has a randomly assigned 128-bit nodeId, circular namespace
Basic operation: A message with key X, sent by any Pastry node, is delivered
to the live node with nodeId closest to X in at most log16 N steps (barring node failures).
Pastry uses a form of generalized hypercube routing, where the routing tables are initialized and updated dynamically.
Look for (d46a1c)
d13da3
65a1fc

20. Pastry: Locality properties
Assumption: scalar proximity metric
e.g. ping/RTT delay, # IP hops
traceroute, subnet masks
a node can probe distance to any other node
Proximity invariant:
Each routing table entry refers to a node close
to the local node (in the proximity space), among
all nodes with the appropriate nodeId prefix.

21. Pastry: Geometric Routing in proximity space
d46a1c
Route(d46a1c)
d462ba
d4213f
d13da3
65a1fc
d467c4
d471f1
d467c4
65a1fc
d13da3
d4213f
d462ba
Proximity space
NodeId space
Proximity space is the geometrical space.
The proximity distance traveled by message in each routing step is exponentially increasing (entry in row l is chosen from a set of nodes of size N/2bl)
The distance traveled by message from its source increases monotonically at each step (message takes larger and larger strides)

22. Pastry: Locality properties
Each routing step is local, but there is no guarantee of globally shortest path
Nevertheless, simulations show:
Expected distance traveled by a message in the proximity space is within a small constant of the minimum
Among k nodes with nodeIds closest to the key, message likely to reach the node closest to the source node first

23. Pastry: Self-organization
Initializing and maintaining routing tables and leaf sets
Node addition
Node departure (failure)
The goal is to maintain all routing table entries
to refer to a near node, among all live nodes
with appropriate prefix

24. Pastry: Node addition New node X contacts nearby node A
A routes “join” message to X, which arrives to Z, closest to X
X obtains leaf set from Z, i’th row for routing table from i’th node from A to Z
X informs any nodes that need to be aware of its arrival
X also improves its table locality by requesting neighborhood sets from all nodes X knows
In practice: optimistic approach

25. Pastry: Node addition d471f1 Z=d467c4 d462ba X=d46a1c
New node: X=d46a1c
d4213f
A is X’s neighbor
Route(d46a1c)
Each node has a randomly assigned 128-bit nodeId, circular namespace
Basic operation: A message with key X, sent by any Pastry node, is delivered
to the live node with nodeId closest to X in at most log16 N steps (barring node failures).
Pastry uses a form of generalized hypercube routing, where the routing tables are initialized and updated dynamically.
d13da3
A = 65a1fc

26. Pastry: Node addition X d467c4 d471f1 d467c4 d462ba d46a1c d4213f
New node: d46a1c
d46a1c
Route(d46a1c)
d462ba
d4213f
d13da3
65a1fc
d467c4
d471f1
NodeId space
d467c4
65a1fc
d13da3
d4213f
d462ba
Proximity space
B1 is first row of B
Each node has a randomly assigned 128-bit nodeId, circular namespace
Basic operation: A message with key X, sent by any Pastry node, is delivered
to the live node with nodeId closest to X in at most log16 N steps (barring node failures).
Pastry uses a form of generalized hypercube routing, where the routing tables are initialized and updated dynamically. X
X is close to A, B is close to B1. Why X is close to B1?
The expected distance from B to its row one entries (B1) is much larger
than the expected distance from A to B (chosen from exponentially
decreasing set size)

27. Node departure (failure)
Leaf set repair (eager – all the time):
Leaf set members exchange keep-alive messages
request set from furthest live node in set
Routing table repair (lazy – upon failure):
get table from peers in the same row, if not found – from higher rows
Neighborhood set repair (eager)

28. Pastry: Summary Generic p2p overlay network
Scalable, fault resilient, self-organizing, secure
O(log N) routing steps (expected)
O(log N) routing table size
Network locality properties