OpenDHT: A Public DHT Service Sean C. Rhea UC Berkeley June 2, 2005 Joint work with: Brighten Godfrey, Brad Karp, John Kubiatowicz, Sylvia Ratnasamy, Scott.

Slides:

Advertisements

Similar presentations

P2P data retrieval DHT (Distributed Hash Tables) Partially based on Hellerstein’s presentation at VLDB2004.

Advertisements

Ion Stoica, Robert Morris, David Karger, M. Frans Kaashoek, Hari Balakrishnan MIT and Berkeley presented by Daniel Figueiredo Chord: A Scalable Peer-to-peer.

Peer-to-Peer Systems Chapter 25. What is Peer-to-Peer (P2P)? Napster? Gnutella? Most people think of P2P as music sharing.

Peer-to-Peer (P2P) Distributed Storage 1Dennis Kafura – CS5204 – Operating Systems.

Clayton Sullivan PEER-TO-PEER NETWORKS. INTRODUCTION What is a Peer-To-Peer Network A Peer Application Overlay Network Network Architecture and System.

CHORD – peer to peer lookup protocol Shankar Karthik Vaithianathan & Aravind Sivaraman University of Central Florida.

Voice over IP Skype.

Chord: A Scalable Peer-to-Peer Lookup Service for Internet Applications Ion Stoica, Robert Morris, David Karger, M. Frans Kaashoek, Hari Balakrishnan Presented.

Distributed Hash Tables: Chord Brad Karp (with many slides contributed by Robert Morris) UCL Computer Science CS M038 / GZ06 27 th January, 2009.

Chord: A Scalable Peer-to-peer Lookup Service for Internet Applications Robert Morris Ion Stoica, David Karger, M. Frans Kaashoek, Hari Balakrishnan MIT.

Massively Distributed Database Systems Distributed Hash Spring 2014 Ki-Joune Li Pusan National University.

1 CS 268: Lecture 22 DHT Applications Ion Stoica Computer Science Division Department of Electrical Engineering and Computer Sciences University of California,

Outline for today Structured overlay as infrastructures Survey of design solutions Analysis of designs.

Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 1 ECSE-6600: Internet Protocols Informal Quiz #13: P2P and Sensor Networks Shivkumar Kalyanaraman:

Peer-to-Peer Networks as a Distribution and Publishing Model Jorn De Boever (june 14, 2007)

Cis e-commerce -- lecture #6: Content Distribution Networks and P2P (based on notes from Dr Peter McBurney © )

What is a P2P system? A distributed system architecture: No centralized control Nodes are symmetric in function Large number of unreliable nodes Enabled.

Topics in Reliable Distributed Systems Lecture 2, Fall Dr. Idit Keidar.

Resource Allocation in OpenHash: a Public DHT Service Sean Rhea with Brad Karp, Sylvia Ratnasamy, Scott Shenker, Ion Stoica, and Harlan Yu.

Chord: A Scalable Peer-to-peer Lookup Service for Internet Applications Ion Stoica, Robert Morris, David Karger, M. Frans Kaashoek and Hari alakrishnan.

Chord-over-Chord Overlay Sudhindra Rao Ph.D Qualifier Exam Department of ECECS.

Fixing the Embarrassing Slowness of OpenDHT on PlanetLab Sean Rhea, Byung-Gon Chun, John Kubiatowicz, and Scott Shenker UC Berkeley (and now MIT) December.

Topics in Reliable Distributed Systems Fall Dr. Idit Keidar.

1 CS 194: Distributed Systems Distributed Hash Tables Scott Shenker and Ion Stoica Computer Science Division Department of Electrical Engineering and Computer.

Data Structures Hash Table (aka Dictionary) i206 Fall 2010 John Chuang Some slides adapted from Marti Hearst, Brian Hayes, Andreas Veneris, Glenn Brookshear,

Peer To Peer Distributed Systems Pete Keleher. Why Distributed Systems? l Aggregate resources! –memory –disk –CPU cycles l Proximity to physical stuff.

Wide-area cooperative storage with CFS

Or, Providing Scalable, Decentralized Location and Routing Network Services Tapestry: Fault-tolerant Wide-area Application Infrastructure Motivation and.

Improving Data Access in P2P Systems Karl Aberer and Magdalena Punceva Swiss Federal Institute of Technology Manfred Hauswirth and Roman Schmidt Technical.

Peer-to-Peer Networks Slides largely adopted from Ion Stoica’s lecture at UCB.

Peer-to-peer approaches for SIP Henning Schulzrinne Dept. of Computer Science Columbia University.

Lecture 10 Naming services for flat namespaces. EECE 411: Design of Distributed Software Applications Logistics / reminders Project Send Samer and me.

1CS 6401 Peer-to-Peer Networks Outline Overview Gnutella Structured Overlays BitTorrent.

Beyond Theory: DHTs in Practice CS Networks Sean C. Rhea April 18, 2005 In collaboration with: Dennis Geels, Brighten Godfrey, Brad Karp, John Kubiatowicz,

A Public DHT Service Sean Rhea, Brighten Godfrey, Brad Karp, John Kubiatowicz, Sylvia Ratnasamy, Scott Shenker, Ion Stoica, and Harlan Yu UC Berkeley and.

CSE 461 University of Washington1 Topic Peer-to-peer content delivery – Runs without dedicated infrastructure – BitTorrent as an example Peer.

Introduction to Peer-to-Peer Networks. What is a P2P network Uses the vast resource of the machines at the edge of the Internet to build a network that.

1 Napster & Gnutella An Overview. 2 About Napster Distributed application allowing users to search and exchange MP3 files. Written by Shawn Fanning in.

Introduction to Peer-to-Peer Networks. What is a P2P network A P2P network is a large distributed system. It uses the vast resource of PCs distributed.

Introduction of P2P systems

Peer to Peer Research survey TingYang Chang. Intro. Of P2P Computers of the system was known as peers which sharing data files with each other. Build.

 A P2P IRC Network Built on Top of the Kademlia Distributed Hash Table.

Chord: A Scalable Peer-to-peer Lookup Protocol for Internet Applications Xiaozhou Li COS 461: Computer Networks (precept 04/06/12) Princeton University.

An Introduction to Peer-to-Peer Networks Presentation for MIE456 - Information Systems Infrastructure II Vinod Muthusamy October 30, 2003.

1 Distributed Hash Tables (DHTs) Lars Jørgen Lillehovde Jo Grimstad Bang Distributed Hash Tables (DHTs)

Vincent Matossian September 21st 2001 ECE 579 An Overview of Decentralized Discovery mechanisms.

T Network Application Frameworks and XML Distributed Hash Tables (DHTs) Sasu Tarkoma.

Peer-to-Peer Network Tzu-Wei Kuo. Outline What is Peer-to-Peer(P2P)? P2P Architecture Applications Advantages and Weaknesses Security Controversy.

1 Peer-to-Peer Technologies Seminar by: Kunal Goswami (05IT6006) School of Information Technology Guided by: Prof. C.R.Mandal, School of Information Technology.

Peer to Peer A Survey and comparison of peer-to-peer overlay network schemes And so on… Chulhyun Park

1 Secure Peer-to-Peer File Sharing Frans Kaashoek, David Karger, Robert Morris, Ion Stoica, Hari Balakrishnan MIT Laboratory.

1 Distributed Hash Table CS780-3 Lecture Notes In courtesy of Heng Yin.

Protocol Requirements draft-bryan-p2psip-requirements-00.txt D. Bryan/SIPeerior-editor S. Baset/Columbia University M. Matuszewski/Nokia H. Sinnreich/Adobe.

Algorithms and Techniques in Structured Scalable Peer-to-Peer Networks

Bruce Hammer, Steve Wallis, Raymond Ho

Two Peer-to-Peer Networking Approaches Ken Calvert Net Seminar, 23 October 2001 Note: Many slides “borrowed” from S. Ratnasamy’s Qualifying Exam talk.

CS 347Notes081 CS 347: Parallel and Distributed Data Management Notes 08: P2P Systems.

P2P Search COP6731 Advanced Database Systems. P2P Computing  Powerful personal computer Share computing resources P2P Computing  Advantages: Shared.

P2P Search COP P2P Search Techniques Centralized P2P systems  e.g. Napster, Decentralized & unstructured P2P systems  e.g. Gnutella.

09/13/04 CDA 6506 Network Architecture and Client/Server Computing Peer-to-Peer Computing and Content Distribution Networks by Zornitza Genova Prodanoff.

1 Open DHT: A Public DHT Service for Developers of Distributed Applications University of California, Irvine Presented By : Ala Khalifeh

Beyond Theory: DHTs in Practice CS Distributed Systems Sean C. Rhea April 18, 2005 In collaboration with: Dennis Geels, Brighten Godfrey, Brad Karp,

1 Secure Peer-to-Peer File Sharing Frans Kaashoek, David Karger, Robert Morris, Ion Stoica, Hari Balakrishnan MIT Laboratory.

Nick McKeown CS244 Lecture 17 Chord: A Scalable Peer-to-peer Lookup Service for Internet Applications [Stoica et al 2001]

Peer-to-Peer (P2P) File Systems. P2P File Systems CS 5204 – Fall, Peer-to-Peer Systems Definition: “Peer-to-peer systems can be characterized as.

CS Spring 2010 CS 414 – Multimedia Systems Design Lecture 24 – Introduction to Peer-to-Peer (P2P) Systems Klara Nahrstedt (presented by Long Vu)

A Case Study in Building Layered DHT Applications

P2P-SIP Using an External P2P network (DHT)

Building Peer-to-Peer Systems with Chord, a Distributed Lookup Service

#02 Peer to Peer Networking

Presentation transcript:

OpenDHT: A Public DHT Service Sean C. Rhea UC Berkeley June 2, 2005 Joint work with: Brighten Godfrey, Brad Karp, John Kubiatowicz, Sylvia Ratnasamy, Scott Shenker, Ion Stoica, and Harlan Yu

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service Peer-to-Peer File Sharing Very simple insight –Most computers unused most of the time Idea: harness this spare capacity to –Quickly download music files [Napster, Gnutella] –Search for aliens –Make free long-distance phone calls [Skype] Question: how to find desired resource(s)? –Early approaches: scoped flooding –Downsides: scalability, accuracy

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service A Better Search Facility: The Distributed Hash Table (DHT) Same interface as a programmatic hash table, –put(key, value) stores value under key –get(key) returns the value(s) stored under key But shared across many machines Implemented via an overlay network

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service A Better Search Facility: The Distributed Hash Table (DHT) K V put(k 1,v 1 ) stores k 1,v 1 get(k 1 ) k1k1 k1k1 k1k1 v1v1 v1v1 v1v1 k1,v1k1,v1 k1,v1k1,v1 k1,v1k1,v1

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service DHTs and File Sharing: DHT Stores Pointers to Files K V pointer to file put(file, IP)

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service DHTs and File Sharing: DHT Stores Pointers to Files K V pointer to file get(file) IP xfer over HTTP

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service DHTs and Spam Detection: Detecting Similar Messages K V put(hash(msg), IP) “I love you!”

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service DHTs and Spam Detection: Detecting Similar Messages K V put(hash(msg), IP) “I love you!” “I love you!”

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service DHTs and Spam Detection: Detecting Similar Messages K V “I love you!” “I love you!” Something’s fishy!

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service DHTs and Spam Detection: Detecting Similar Messages K V put(hash(msg), IP) “I love you!” “I love you!” “I love you!”

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service DHTs and Spam Detection: Detecting Similar Messages K V “I love you!” “I love you!” “I love you!” Something’s fishy!

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service More DHT Applications Distributed Storage Systems –CFS, HiveCache, PAST, Pastiche –OceanStore / Pond Content Distribution Networks / Web Caches –Bslash, Coral, Squirrel Indexing / Naming Systems –Chord-DNS, CoDoNS, DOA, SFR Internet Query Processors –Catalogs, PIER Communication Systems –Bayeux, i3, MCAN, SplitStream

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service Some Areas of DHT Research Better routing protocols –One-hop, degree-optimal Load balancing –Non-uniform key distributions Security –Byzantine fault-tolerant routing Data redundancy and fault tolerance –Replication, erasure-coding Stronger semantics –Supporting read-modify-write

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service How Many DHTs Will There Be? K V File Sharing K V Spam Detection Company Machine: Can’t Share Files Owns Stock in Spam Company K V

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service How Many DHTs Will There Be? K V File Sharing K V Spam Detection Redundant Link K V

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service How Many DHTs Will There Be? K V File Sharing K V Spam Detection Unshared Links K V

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service Benefits of Sharing a DHT Amortizes costs across applications –Maintenance bandwidth, connection state, etc. Facilitates “bootstrapping” of new applications –Working infrastructure already in place Allows for statistical multiplexing of resources –Takes advantage of spare storage and bandwidth Facilitates upgrading existing applications –“Share” DHT between application versions

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service Challenges in Sharing a DHT Robustness –Must be available 24/7 Shared Interface Design –Should be general, yet easy to use Resource Allocation –Must protect against malicious/over-eager users Economics –What incentives are there to provide resources?

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service Challenges in Sharing a DHT Robustness –Must be available 24/7 Shared Interface Design –Should be general, yet easy to use Resource Allocation –Must protect against malicious/over-eager users Economics –What incentives are there to provide resources?

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service The DHT as a Service K V

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service The DHT as a Service K V OpenDHT

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service The DHT as a Service OpenDHT Clients

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service The DHT as a Service OpenDHT

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service The DHT as a Service OpenDHT What is this interface?

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service The Traditional Interface: lookup

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service The Traditional Interface: lookup lookup(k) k On reaching the successor of k, message passed to an “upcall”

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service DHTs and Spam Detection: Detecting Similar Messages K V put(hash(msg), IP) “I love you!” “I love you!” Upcall: I’ve seen this message before!

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service DHTs and Spam Detection: Detecting Similar Messages K V “I love you!” “I love you!” Something’s fishy!

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service Upcall Challenges Distribution –How do we get new upcall code to all nodes?

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service Upcall Challenges lookup(k) k How did the upcall code get here?

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service Upcall Challenges Distribution –How do we get new upcall code to all nodes? –Active networking experience is a warning…

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service Upcall Challenges Distribution –How do we get new upcall code to all nodes? –Active networking experience is a warning… Security –How do we safely run untrusted clients’ upcalls?

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service What about Put/Get? Works great for some applications –File sharing, for example

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service DHTs and File Sharing: DHT Stores Pointers to Files K V pointer to file get(file) IP xfer over HTTP put(file, IP)

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service What about Put/Get? Works great for some applications –File sharing, for example What about applications with upcalls? –Our spam detection application, for example

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service What about Put/Get? Works great for some applications –File sharing, for example What about applications with upcalls? –Our spam detection application, for example Idea: let application nodes run the upcalls –Each node only runs upcalls for the applications that it’s participating in

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service Upcall Example OpenDHT put/get File Sharing I can handle spam detection messages Spam Detection I can handle spam detection messages I can handle spam detection messages

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service Upcall Example OpenDHT put/get File Sharing Spam Detection “I love you!” Who’s handling hash(message)?

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service Upcall Example OpenDHT put/get File Sharing Spam Detection “I love you!” Who’s handling hash(message)? “I love you!”

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service Upcall Example OpenDHT put/get File Sharing Spam Detection “I love you!” “I love you!” Something’s fishy! DHT keeps track of which nodes support which upcalls via Recursive Distributed Rendezvous (ReDiR)

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service ReDiR Goal: Implement two functions using put/get: –join(namespace, node) –node = lookup(namespace, identifier) H(namespace) L0 L1 L2 H(A) A A A H(B)

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service ReDiR Goal: Implement two functions using put/get: –join(namespace, node) –node = lookup(namespace, identifier) L0 L1 L2 H(A) A, B A A H(B)H(C) C

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service ReDiR Goal: Implement two functions using put/get: –join(namespace, node) –node = lookup(namespace, identifier) L0 L1 L2 H(A) A, B A, C A H(B)H(C) C H(D) D D

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service ReDiR Goal: Implement two functions using put/get: –join(namespace, node) –node = lookup(namespace, identifier) L0 L1 L2 H(A) A, B A, C A, D H(B)H(C) C H(D) D D H(E) E

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service ReDiR Goal: Implement two functions using put/get: –join(namespace, node) –node = lookup(namespace, identifier) L0 L1 L2 H(A) A, B A, C A, D H(B)H(C) C H(D) D D, E H(E) E

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service ReDiR Join cost: –Worst case: O(log n) puts and gets –Average case: O(1) puts and gets L0 L1 L2 H(A) A, B A, C A, D H(B)H(C) C H(D) D D, E H(E) E

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service ReDiR Goal: Implement two functions using put/get: –join(namespace, node) –node = lookup(namespace, identifier) L0 L1 L2 H(A) A, B A, C A, D H(B)H(C) C H(D) D D, E H(E) E H(k1) successor

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service ReDiR Goal: Implement two functions using put/get: –join(namespace, node) –node = lookup(namespace, identifier) L0 L1 L2 H(A) A, B A, C A, D H(B)H(C) C H(D) D D, E H(E) E H(k2) no successor successor

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service ReDiR Goal: Implement two functions using put/get: –join(namespace, node) –node = lookup(namespace, identifier) L0 L1 L2 H(A) A, B A, C A, D H(B)H(C) C H(D) D D, E H(E) E H(k3) no successor successor no successor

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service ReDiR Lookup cost: –Worst case: O(log n) gets –Average case: O(1) gets L0 L1 L2 H(A) A, B A, C A, D H(B)H(C) C H(D) D D, E H(E) E

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service ReDiR Performance (On PlanetLab)

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service OpenDHT Design Summary OpenDHT is a common infrastructure for –Storage of values, pointers, etc. –Organizing clients that handle application upcalls Benefits: –Amortizes maintenance costs across applications –Facilitates “bootstrapping” of new applications –Allows for statistical multiplexing of resources

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service Impact ApplicationUses OpenDHT forInterface Croquet Media Managerreplica locationput/get DOAindexingput/get HIPname resolutionput/get Tetherless Computinghost mobilityput/get Place Labrange queriesReDiR QStreammcast tree constr.ReDiR VPN Indexindexingput/get FreeDBstorageput/get Instant Messagingrendezvousput/get CFSstorageput/get i3i3redirectionReDiR

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service Future Work OpenDHT makes a great common substrate for: –Soft-state storage –Naming and rendezvous Many P2P applications also need to: –Traverse NATs –Redirect packets within the infrastructure (as in i3) –Refresh puts while intermittently connected All of these can be implemented with upcalls –Who provides the machines that run the upcalls?

June 2, 2005Sean C. RheaOpenDHT: A Public DHT Service Future Work We don’t want to add upcalls to the core DHT –Keep the main service simple, fast, and robust Can we build a separate upcall service? –Some other set of machines organized with ReDiR –Security: can only accept incoming connections, can’t write to local storage, etc. This should be enough to implement –NAT traversal, reput service –Some (most?) packet redirection What about more expressive security policies?

For more information, see