1. A Common API for Structured Peer-to-Peer Overlays
Frank Dabek, Ben Y. Zhao, Peter Druschel, Ion Stoica
Change motivation to, everyone’s using these, but nobody understands what they’re using… let’s break down the pieces, so that we can remove the commonality and examine the differences.
Change presentation to top down from bottom up. Fill in explanation of DOLR implementation on API, absorb anycast into DOLR. Write out names of systems onto DOLR (tapestry=pastry+scribe), DHT (can=Chord+DHash)

2. Structured Peer-to-Peer Overlay
They are:
Scalable, self-organizing overlay networks
Provide routing to location-independent names
Examples: CAN, Chord, Pastry, Tapestry, …
Basic operation:
Large sparse namespace N (integers: 0–2128 or 0–2160)
Nodes in overlay network have nodeIds  N
Given k  N, a deterministic function maps k to its root node (a live node in the network)
route(msg, k) delivers msg to root(k)
OceanStore / Sahara Retreat
January 14, 2003

3. Current Progress Lots of applications built on top
File systems, archival backup
Application level multicast
Routing for anonymity, attack resilience
But do we really understand them?
What is the core functionality that applications leverage from them?
What are the strengths and weaknesses of each protocol? How can they be exploited by applications?
How can we build new protocols customized to our future needs?
OceanStore / Sahara Retreat
January 14, 2003

4. Our Goals Protocol comparison Towards a common API
Compare and contrast protocol semantics
Identify basic commonalities
Isolate and understand differences
Towards a common API
Easily supportable by old and new protocols
Enables application portability between protocols
Enables common benchmarks
Provides a framework for reusable components
OceanStore / Sahara Retreat
January 14, 2003

5. Talk Outline Motivation DHTs and DOLRs A Flexible Routing API
Usage Examples
OceanStore / Sahara Retreat
January 14, 2003

6. Decomposing Functional Layers
Distributed Hash Tables (DHT)
put(key, data), value = get(key)
Hashtable layered across network
Handles replication; distributes replicas randomly
Routes queries towards replicas by name
Decentralized Object Location and Routing (DOLR)
publish(objectId), route(msg, nodeId), routeObj(msg, objectId, n)
Application controls replication and placement
Cache location pointers to replicas; queries quickly intersect pointers and redirect to nearby replica(s)
OceanStore / Sahara Retreat
January 14, 2003

7. DHT Illustrated OceanStore / Sahara Retreat ravenben@eecs.berkeley.edu
January 14, 2003

8. DOLR Illustrated
OceanStore / Sahara Retreat
January 14, 2003

9. Architecture CFS PAST SplitStream i3 OceanStore Bayeux Tier 2 DHT
Multicast
DOLR
CAN,
Chord+DHash
Tapestry
Pastry+Scribe
Tier 1
Replication
Routing Mesh
Tier 0
OceanStore / Sahara Retreat
January 14, 2003

10. Talk Outline Motivation DHTs and DOLRs A Flexible Routing API
Usage Examples
OceanStore / Sahara Retreat
January 14, 2003

11. Flexible API for Routing
Goal
Consistent API for leveraging routing mesh
Flexible enough to build higher abstractions
Openness promotes new abstractions
Allow competitive selection to determine right abstractions
Three main components
Invoking routing functionality
Accessing namespace mapping properties
Open, flexible upcall interface
OceanStore / Sahara Retreat
January 14, 2003

12. API (routing) Data types Key, nodeId = 160 bit integer
Node = Address (IP + port #), nodeId
Msg: application-specific msg of arbitrary size
Invoking routing functionality
Route(key, msg, [node])
route message to node currently responsible for key
Non-blocking, best effort – message may be lost or duplicated.
node: transport address of the node last associated with key (proposed first hop, optional)
OceanStore / Sahara Retreat
January 14, 2003

13. API (namespace properties)
nextHopSet = local_lookup(key, num, safe)
Returns a set of at most num nodes from the local routing table that are possible next hops towards the key.
Safe: whether choice of nodes is randomly chosen
nodehandle[ ] = neighborSet(max_rank)
Returns unordered set of nodes as neighbors of the current node.
Neighbor of rank i is responsible for keys on this node should all neighbors of rank < i fail
nodehandle[ ] = replicaSet(key, num)
Returns ordered set of up to num nodes on which replicas of the object with key key can be stored.
Result is subset of neighborSet plus local node
boolean = range(node, rank, lkey, rkey)
Returns whether current node would be responsible for the range specified by lkey and rkey, should the previous rank-1 nodes fail.
OceanStore / Sahara Retreat
January 14, 2003

14. API (upcalls) Deliver(key, msg) Forward(&key, &msg, &nextHopNode)
Application
Application
Deliver(key, msg)
Delivers an incoming message to the application. One application per node. Demultiplexing done by including demux key in msg.
Forward(&key, &msg, &nextHopNode)
Synchronous upcall invoked at each node along route
On return, will forward msg to nextHopNode
App may modify key, msg, nextHopNode, or terminate by setting nextHopNode to NULL.
Update(node, boolean joined)
Upcall invoked to inform app of a change in the local node’s neighborSet, either a new node joining or an old node leaving.
deliver
forward
msg
msg
msg
Routing Layer
Routing Layer
OceanStore / Sahara Retreat
January 14, 2003

15. Talk Outline Motivation DHTs and DOLRs A Flexible Routing API
Usage Examples
OceanStore / Sahara Retreat
January 14, 2003

16. DHT Implementation Interface Implementation (source S, root R)
put (key, value)
value = get (key)
Implementation (source S, root R)
Put: route(key, [PUT,value,S], NULL) Reply: route(NULL, [PUT-ACK,key], S)
Get: route(key, [GET,S], NULL) Reply: route(NULL, [value,R], S)
OceanStore / Sahara Retreat
January 14, 2003

17. DOLR Implementation Interface
RouteNode(msg, nodeId)
Publish(objectId)
RouteObj(msg, objectId, n)
Implementation (server S, client C, object O)
RouteNode: route(nodeId, msg, NULL)
Publish: route(objectId, [“publish”,O,S], NULL) Upcall: addLocal([O,S])
RouteObj: route(nodeId, [n,msg], NULL) Upcall: serverSet[] = getLocal(O); if (|serverSet|<n), route(nodeId, [n-|serverSet|,msg], NULL) for first n entries in serverSet, route(serverSet[i], msg, NULL)
OceanStore / Sahara Retreat
January 14, 2003

18. Conclusion Very much ongoing work Ongoing Work
Feedback valuable and appreciated
Ongoing Work
Implementations will support routing API
Working towards higher level abstractions Distributed Hash Table API DOLR publish/route API
For more information, see IPTPS 2003
Thank you…
OceanStore / Sahara Retreat
January 14, 2003

19. Backup Slides Follow…
OceanStore / Sahara Retreat
January 14, 2003

20. Storage API: Overview linsert(key, value); value = lget(key);
OceanStore / Sahara Retreat
January 14, 2003

21. Storage API
linsert(key, value): store the tuple <key, value> into local storage. If a tuple with key already exists, it is replaced. The insertion is atomic wrt to failures of the local node.
value = lget(key): retrieves the value associated with key from local storage. Returns null if no tuple with key exists.
OceanStore / Sahara Retreat
January 14, 2003

22. Following slides contain functions that we haven’t decided on yet…
To Do
Following slides contain functions that we haven’t decided on yet…
OceanStore / Sahara Retreat
January 14, 2003

23. Basic DHT API: Overview
insert(key, value, lease);
value = get(key);
release(key);
Upcalls:
insertData(key, value, lease);
OceanStore / Sahara Retreat
January 14, 2003

24. Basic DHT API
insert(key, value, lease): inserts the tuple <key, value> into the DHT. The tuple is guaranteed to be stored in the DHT only for “lease” time. “value” also includes the type of operations to be performed on insertion. Default operation types include
REPLACE: replace value associated with the same key
APPEND: append value to the existing key
UPCALL: generate an upcall to application before inserting …
Does insert include guarantees?
No assumption about network layer, but what about semantics of this call?
For hard state, decouple from the kind of availability (# of replicas)
For soft state, good for adaptability to node changes, can quantify the worse case loss of availability
Include insert(key, value, leaselength), returns true/false (false includes if node cannot store for that long)
Leaselength = 0  best effort,
What are the semantics of the caches of the soft state. After timeout, the assumption is that it still may be there? The answer can make caching of data easier or harder. Keep question on the table. Cached copies should also keep an associated time out which is derived from the timeout of the primary copy.
What happens when you change data, and stale data that has expired may not be deleted? Enforced delete may lend to an easy cache validation protocol. Solution: let’s have lookup(key) ignore invalidated copies, then from the application standpoint, deletes are “enforced”
Fine with route.
Fine with semantics from insert
Question is not whether we want append (yes we do), but how it’s done.
OceanStore / Sahara Retreat
January 14, 2003

25. Basic DHT API
value = get(key): retrieves the value associated with key. Returns null if no tuple with key exists in the DHT.
Does insert include guarantees?
No assumption about network layer, but what about semantics of this call?
For hard state, decouple from the kind of availability (# of replicas)
For soft state, good for adaptability to node changes, can quantify the worse case loss of availability
Include insert(key, value, leaselength), returns true/false (false includes if node cannot store for that long)
Leaselength = 0  best effort,
What are the semantics of the caches of the soft state. After timeout, the assumption is that it still may be there? The answer can make caching of data easier or harder. Keep question on the table. Cached copies should also keep an associated time out which is derived from the timeout of the primary copy.
What happens when you change data, and stale data that has expired may not be deleted? Enforced delete may lend to an easy cache validation protocol. Solution: let’s have lookup(key) ignore invalidated copies, then from the application standpoint, deletes are “enforced”
Fine with route.
Fine with semantics from insert
Question is not whether we want append (yes we do), but how it’s done.
OceanStore / Sahara Retreat
January 14, 2003

26. Basic DHT API
Release(key): releases any tuples with key from the DHT. After this operations completes, tuples with key are no longer guaranteed to exist in the DHT.
OceanStore / Sahara Retreat
January 14, 2003

27. Basic DHT API: Open questions
Semantics?
Verification/Access control/multiple DHTs?
Caching?
Replication?
Should we have leases? It makes us dependent on secure time sync.
OceanStore / Sahara Retreat
January 14, 2003

28. Replicating DHT API
Insert(key, value, numReplicas); adds a numReplicas argument to insert. Ensures resilience of the tuple to up to numReplicas-1 “simultaneous” node failures.
Open questions:
consistency
OceanStore / Sahara Retreat
January 14, 2003

29. Caching DHT API
Same as basic DHT API. Implementation uses dynamic caching to balance query load.
OceanStore / Sahara Retreat
January 14, 2003

30. Resilient DHT API
Same as replicating DHT API. Implementation uses dynamic caching to balance query load.
OceanStore / Sahara Retreat
January 14, 2003

31. Publish API: Overview Publish(key, object); object = Lookup(key);
Remove(key, object):
OceanStore / Sahara Retreat
January 14, 2003

32. Publish API
Publish(key, object): ensures that the locally stored object can be located using the key. Multiple instances of the object may be published under the same key from different locations.
object = Lookup(key): locates the nearest instance of the object associated with key. Returns null if no such object exists.
OceanStore / Sahara Retreat
January 14, 2003

33. Publish API
Remove(key, object): after this operation completes, the local instance of object can no longer be located using key.
OceanStore / Sahara Retreat
January 14, 2003