1. Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems Antony Rowstron and Peter Druschel, Middleware 2001

2. Outline Introduction Design of Pastry Experimental Results Discussion
Node state & routing
Pastry API
Self-organization and adaptation
Locality
Experimental Results
Discussion

3. Introduction: What is Pastry?
It’s a scalable, distributed, decentralized object location and routing substrate
Serves as a general substrate for building P2P applications: SCRIBE, PAST,…etc.
Seeks to minimize distance messages travel
Pastry’s main capability

4. Pastry Node
Represented by 128-bit randomly chosen nodeId (Hash of IP or public key)
NodeId is in base 2b (b is a configuration parameter; b typical value 2 or 4)
Evenly distributed nodeIds along the circular namespace ( – 1 space).
Routes a message in O(log N) steps to destination
N: size of network
Node state contains:
Leaf Set ( L )
Routing table ( R )
Neighborhood Set ( M )

5. Design of Pastry: Node state
Leaf set: L/2 Numerically closest nodes (L is a configuration parameter = 16, 32 typically )
Routing Table (Prefix-based)
Neighborhood Set: M physically closest nodes

6. Pastry node state (Leaf Set)
Serves as a fall back for routing table and contains:
L/2 numerically closest and larger nodeIds
L/2 numerically closest and smaller nodIds
Size of L is typically 2b or 2 x 2b
Nodes in L are numerically close (could be geographically diverse)

7. Pastry node state: Neighborhood set (M)
Contains the IP addresses and nodeIds of closest nodes according to proximity metric
Size of |M| is typically 2b or 2x2b
Not used in routing, but instead for maintaining locality properties

8. Node state: Routing Table
Matrix of Log2b N rows and 2b – 1 columns (N is the number of nodes in the network)
Entries in row n match the first n digits of current nodeId AND
Column number follows matched digits: Format: matched digits–column number–rest of ID
Log2b N populated on average

9. Node (2), (b = 2, l = 8) 1 2 3

10. Routing(2):
If message with key D is within range of leaf set, forward to numerically closest leaf
Else forward to node that shares at least one more digit with D in its prefix than current nodeId
If no such node exists, forward to node that shares at least as many digits with D as current nodeId but numerically nearer than current nodeId

11. Routing Messages (1) Node is in the leaf set
(2) Forward message to a closer node (Better match)
(3) Forward towards numerically
Closer node (not a better match)
D: Message Key
Li: ith closest NodeId in leaf set
shl(A, B): Length of prefix shared
by nodes A and B
Rij: (j, i)th entry of routing table
Source: Rowstron & Drushel, 2001

12. Routing Example:
Source:

13. Routing Performance: (1) If key is within leaf set:
target one hop away
(2) If key has to be routed:
Number of nodes with longer prefix decreases by 2b
(3) Key is not covered by the leaf set (i.e., failed)
With high probability, one more hop needed
Thus: Number of routing steps needed log 2b N

14. Pastry API: nodeId = pastryInit(Credentials, Application )
operations exported by pastry:
nodeId = pastryInit(Credentials, Application )
Causes a Pastry node to join the network with state initialization. Other application specific information is also established.
route(msg, key)
Routes the message to another node which is numerically closest to the key

15. Pastry API forward(msg, key, nextId) deliver(msg,key)
Operations exported by the application working on top of Pastry
deliver(msg,key)
Called when local node is numerically closest to the key
forward(msg, key, nextId)
forward a message from the local nodeId to the next nodeId ( nextId = Null if local node is final)
newLeafs( leafSet):
Updates the leaf set

16. Pastry node join In process, A,B,…,Z send their state tables to X
X = new node, Z = numerically closest node, A = bootstrap (A is close in proximity space to X)
X sends a join message to A with target nodeId X
A forwards to B C…
Stops at Z, numerically closest to X’s nodeId
In process, A,B,…,Z send their state tables to X

17. Node Join X’s neighborhood set (NS) = A’s NS
X’s Leaf Set = Z’s leaf set
X’s routing table is filled as follows:
X’s Row 0 = A’s row 0 (X0 = A0)
X’s Row 1 = B’s row 1 (X1 = B1)
…etc.
X sends its state to every node in its state tables ( Leaf set, neighborhood set, and routing table)

18. Node Join: Example www.scs.cs.nyu.edu/V22.0480-005/notes/l24.pdf
Source:

19. Node departure (2)
Invalid nodes in leaf set: detected by heartbeat monitor
Repair by inserting node from another leaf’s LS
Heartbeat for neighborhood set (NS)
Query all NS members for their NS tables, choose replacement according to proximity metric
Invalid routing entries detected when attempting to route
Query nodes in row for replacement entry, if failed
Query successive rows until success

20. Node failure in routing table: example If node in red fails
Source: cs514/2003fa/CS514-fa03-lec26v0.pdf

21. Locality in Pastry
Based on proximity metric (i.e., No. of IP hops, geographic distance)
Proximity space is assumed to be Euclidean
The route chosen for a message is likely to be “good“ with respect to the proximity metric
We will discuss locality regarding:
Routing table locality
Route locality
Locating the nearest among k nodes

22. Locality in Routing tables
Invariant: “all routing table entries refer to a node that is near the present node, according to the proximity metric, among all live nodes with a prefix appropriate for the entry.”
We wish to maintain the invariant when adding new nodes.
X joins; A is close to X; X0 = A0, so locality holds in X’s routing table
X1 = B1. Entires in B1 (row 1 of X) are close to B, but are they necessarily close to X?

23. Locality in routing table
Entries of B1 are reasonable close to X Why?
A is much closer to B than entry in B1 to B because every time we choose from an exponentially decreasing set of nodes
To improve proximity approximation:
X Queries nodes in routing table and neighborhood set for their state
Compares distances (from routing table entries) and update route entries with closer nodes if found.

24. Route locality
At each routing step the message is moved closer to the destination in the:
nodeId space (numerically closer nodes)
proximity space: message travels the least possible distance
Given that:
A message routed from A to B at a distance d cannot be routed to a node with a distance of less than d from A. (follows from routing procedure)
Expected distance traveled increases exponentially
Though shortest path is not guaranteed, we still get a good route.

25. Locality among k nodes
In some Pastry-based applications, object is replicated on k nodes on its route (during insertion)
In prefix-base routing: goal is to reach any of k numerically closest nodes that has a copy of object
May miss nearby nodes with different prefix
Use heuristic to determine when close to k nearest nodes
Based on density of nodeIds that store object; using local info
Switch to numerically closest address

26. Arbitrary node failure
Node continues to be responsive, but behaves incorrectly or maliciously.
Repeated queries fail each time because they normally take the same route.
How to solve it? Use randomized routing
The choice among multiple nodes that satisfy the routing criteria should be made randomly

27. Routing Performance |L|=16 * b=4 * |M|=32 * 200,000 lookups
Source: Rowstron & Drushel, 2001

28. Pastry routing
Source: Rowstron & Drushel, 2001

29. Routing with failures
Source: Rowstron & Drushel, 2001

30. |L|=16 * b=4 * |M|=32 * 200,000 lookups
Pastry locality
|L|=16 * b=4 * |M|=32 * 200,000 lookups
Source: Rowstron & Drushel, 2001

31. Summary Pastry is a generic P2P object location and routing substrate
Distributed, and scales well
Used in developing applications like file storage, global file sharing,...etc.
Considers locality when routing messeges

32. References
(1) A. Rowstron and P. Druschel, "Pastry: Scalable, distributed object location and routing for large-scale peer-to-peer systems".  IFIP/ACM International Conference on Distributed Systems Platforms (Middleware), Heidelberg, Germany, pages , November, 2001
(2) Jeff Odom slides: