1. Outline for today Structured overlay as infrastructures Survey of design solutions Analysis of designs

2. Questions: How many DHTs will there be? Can all applications share one DHT?

3. Benefits of Sharing a DHT Amortizes costs across applications  Maintenance bandwidth, connection state, etc. Facilitates “bootstrapping” of new applications  Working infrastructure already in place Allows for statistical multiplexing of resources  Takes advantage of spare storage and bandwidth Facilitates upgrading existing applications  “Share” DHT between application versions

4. The DHT as a Service K V DHT

5. The DHT as a Service DHT Clients

6. The DHT as a Service DHT

7. The DHT as a Service OpenDHT What is this interface?

8. It’s not lookup() lookup(k) k What does this node do with it? Challenges: 1.Distribution 2.Security

9. What about put/get? Works easily for storage applications Easy to share  No upcalls, so no code distribution or security complications But does it work for rendezvous?  Chat? Sure: put(my-name, my-IP)  What about the others?

10. A Case for Common APIs Lots and lots of peer to peer applications  Decentralized file systems, archival backup  Group communication / coordination  Routing layers for anonymity, attack resilience  Scalable content distribution A number of scalable, self-organizing overlays  E.g. CAN, Chord, Pastry, Tapestry, Kademlia, etc… Semantic differences  Store/get data, locate objects, multicast / anycast  How do these functional layers relate?  What is the smallest common denominator?

11. How are DHT’s used – Some Real Applications Many possibilities  Storage, media delivery, resilient routing… Storage  Distributed file systems  Automatic backup services  Distributed CVS Media delivery  Wide-area multicast systems, pub-sub  Video-on-demand systems  Content distribution networks (CDNs)  Multicast, anycast   Rendezvous Resilient routing  Route around Internet failures

12. Some Abstractions Distributed Hash Tables (DHT)  Simple store and retrieve of values with a key  Values can be of any type Decentralized Object Location and Routing (DOLR)  Decentralized directory service for endpoints/objects  Route messages to nearest available endpoint Multicast / Anycast (CAST)  Scalable group communication  Decentralized membership management

13. Tier 1 Interfaces Distributed Hash Tables (DHT) Decentralized Object Location / Routing (DOLR) Multicast / Anycast (CAST) put (key, data)publish (objectId)join (groupId) remove (key)unpublish (objectId)leave (groupId) value = get (key) sendToObj (msg, objectId, [n]) multicast (msg, gId) anycast (msg, gId)

14. Structured P2P Overlays Key-based Routing Tier 0 Tier 1 Tier 2 CFSPASTSplitStreami3OceanStoreBayeux CASTDHTDOLR

15. The Challenges Maintenance  Many components, many administrative domains  Constant change  Must be self-organizing  Must be self-maintaining  All resources virtualized—no physical names Security  High availability is a hacker’s target-rich environment  Must have end-to-end encryption  Must not place too much trust in any one host

16. The Big Picture For durability (archival layer)  Apply Erasure coding  Replicate copies of fragments across network  Periodically check for level of redundancy For quick access (dissemination layer)  Maintain small # of copies replicated in network  Use access tracking to move copies closer to clients For attack resilience (Byzantine layer)  Various solutions Key decisions made per file by “inner-ring” of servers, Distributed decisions verified by a “threshold signature” Signed data Self-certifying data (SFS File System)