Distributed Hash Tables

Slides:



Advertisements
Similar presentations
SkipNet: A Scalable Overlay Network with Practical Locality Properties Nick Harvey, Mike Jones, Stefan Saroiu, Marvin Theimer, Alec Wolman Microsoft Research.
Advertisements

Luca Maria Aiello, Università degli Studi di Torino, Computer Science department 1 Tempering Kademlia with a robust identity based system.
P2P data retrieval DHT (Distributed Hash Tables) Partially based on Hellerstein’s presentation at VLDB2004.
Ion Stoica, Robert Morris, David Karger, M. Frans Kaashoek, Hari Balakrishnan MIT and Berkeley presented by Daniel Figueiredo Chord: A Scalable Peer-to-peer.
Peer to Peer and Distributed Hash Tables
Pastry Peter Druschel, Rice University Antony Rowstron, Microsoft Research UK Some slides are borrowed from the original presentation by the authors.
Scalable Content-Addressable Network Lintao Liu
CHORD – peer to peer lookup protocol Shankar Karthik Vaithianathan & Aravind Sivaraman University of Central Florida.
Technische Universität Yimei Liao Chemnitz Kurt Tutschku Vertretung - Professur Rechner- netze und verteilte Systeme Chord - A Distributed Hash Table Yimei.
Technische Universität Chemnitz Kurt Tutschku Vertretung - Professur Rechner- netze und verteilte Systeme Chord - A Distributed Hash Table Yimei Liao.
CHORD: A Peer-to-Peer Lookup Service CHORD: A Peer-to-Peer Lookup Service Ion StoicaRobert Morris David R. Karger M. Frans Kaashoek Hari Balakrishnan Presented.
Chord: A scalable peer-to- peer lookup service for Internet applications Ion Stoica, Robert Morris, David Karger, M. Frans Kaashock, Hari Balakrishnan.
Distributed Hash Tables CPE 401 / 601 Computer Network Systems Modified from Ashwin Bharambe and Robert Morris.
Chord: A Scalable Peer-to-Peer Lookup Service for Internet Applications Ion Stoica, Robert Morris, David Karger, M. Frans Kaashoek, Hari Balakrishnan Presented.
Robert Morris, M. Frans Kaashoek, David Karger, Hari Balakrishnan, Ion Stoica, David Liben-Nowell, Frank Dabek Chord: A scalable peer-to-peer look-up.
Chord: A Scalable Peer-to-peer Lookup Service for Internet Applications Robert Morris Ion Stoica, David Karger, M. Frans Kaashoek, Hari Balakrishnan MIT.
Chord: A Scalable Peer-to-peer Lookup Service for Internet Applications Ion StoicaRobert Morris David Liben-NowellDavid R. Karger M. Frans KaashoekFrank.
Massively Distributed Database Systems Distributed Hash Spring 2014 Ki-Joune Li Pusan National University.
Pastry Peter Druschel, Rice University Antony Rowstron, Microsoft Research UK Some slides are borrowed from the original presentation by the authors.
An Overview of Peer-to-Peer Networking CPSC 441 (with thanks to Sami Rollins, UCSB)
Outline for today Structured overlay as infrastructures Survey of design solutions Analysis of designs.
1 Chord: A Scalable Peer-to-peer Lookup Service for Internet Applications Robert Morris Ion Stoica, David Karger, M. Frans Kaashoek, Hari Balakrishnan.
What is a P2P system? A distributed system architecture: No centralized control Nodes are symmetric in function Large number of unreliable nodes Enabled.
Looking Up Data in P2P Systems Hari Balakrishnan M.Frans Kaashoek David Karger Robert Morris Ion Stoica.
Chord: A Scalable Peer-to-Peer Lookup Protocol for Internet Applications Stoica et al. Presented by Tam Chantem March 30, 2007.
P2P: Advanced Topics Filesystems over DHTs and P2P research Vyas Sekar.
Distributed Lookup Systems
Chord: A Scalable Peer-to-peer Lookup Service for Internet Applications Ion Stoica, Robert Morris, David Karger, M. Frans Kaashoek and Hari alakrishnan.
Structure Overlay Networks and Chord Presentation by Todd Gardner Figures from: Ion Stoica, Robert Morris, David Liben- Nowell, David R. Karger, M. Frans.
Object Naming & Content based Object Search 2/3/2003.
Chord-over-Chord Overlay Sudhindra Rao Ph.D Qualifier Exam Department of ECECS.
Topics in Reliable Distributed Systems Fall Dr. Idit Keidar.
Peer To Peer Distributed Systems Pete Keleher. Why Distributed Systems? l Aggregate resources! –memory –disk –CPU cycles l Proximity to physical stuff.
1 Peer-to-Peer Networks Outline Survey Self-organizing overlay network File system on top of P2P network Contributions from Peter Druschel.
A Backup System built from a Peer-to-Peer Distributed Hash Table Russ Cox joint work with Josh Cates, Frank Dabek, Frans Kaashoek, Robert Morris,
Structured P2P Network Group14: Qiwei Zhang; Shi Yan; Dawei Ouyang; Boyu Sun.
CSE 461 University of Washington1 Topic Peer-to-peer content delivery – Runs without dedicated infrastructure – BitTorrent as an example Peer.
INTRODUCTION TO PEER TO PEER NETWORKS Z.M. Joseph CSE 6392 – DB Exploration Spring 2006 CSE, UT Arlington.
Wide-Area Cooperative Storage with CFS Robert Morris Frank Dabek, M. Frans Kaashoek, David Karger, Ion Stoica MIT and Berkeley.
Wide-area cooperative storage with CFS Frank Dabek, M. Frans Kaashoek, David Karger, Robert Morris, Ion Stoica.
Chord & CFS Presenter: Gang ZhouNov. 11th, University of Virginia.
1 Reading Report 5 Yin Chen 2 Mar 2004 Reference: Chord: A Scalable Peer-To-Peer Lookup Service for Internet Applications, Ion Stoica, Robert Morris, david.
Chord: A Scalable Peer-to-peer Lookup Protocol for Internet Applications Xiaozhou Li COS 461: Computer Networks (precept 04/06/12) Princeton University.
1 Distributed Hash Tables (DHTs) Lars Jørgen Lillehovde Jo Grimstad Bang Distributed Hash Tables (DHTs)
Vincent Matossian September 21st 2001 ECE 579 An Overview of Decentralized Discovery mechanisms.
Ion Stoica, Robert Morris, David Karger, M. Frans Kaashoek, Hari Balakrishnan MIT and Berkeley presented by Daniel Figueiredo Chord: A Scalable Peer-to-peer.
An IP Address Based Caching Scheme for Peer-to-Peer Networks Ronaldo Alves Ferreira Joint work with Ananth Grama and Suresh Jagannathan Department of Computer.
Chord: A Scalable Peer-to-peer Lookup Service for Internet Applications Ion Stoica, Robert Morris, David Karger, M. Frans Kaashoek, Hari Balakrishnan Presented.
SIGCOMM 2001 Lecture slides by Dr. Yingwu Zhu Chord: A Scalable Peer-to-peer Lookup Service for Internet Applications.
Peer to Peer A Survey and comparison of peer-to-peer overlay network schemes And so on… Chulhyun Park
1 Secure Peer-to-Peer File Sharing Frans Kaashoek, David Karger, Robert Morris, Ion Stoica, Hari Balakrishnan MIT Laboratory.
1 Distributed Hash Table CS780-3 Lecture Notes In courtesy of Heng Yin.
Algorithms and Techniques in Structured Scalable Peer-to-Peer Networks
LOOKING UP DATA IN P2P SYSTEMS Hari Balakrishnan M. Frans Kaashoek David Karger Robert Morris Ion Stoica MIT LCS.
Bruce Hammer, Steve Wallis, Raymond Ho
A Sybil-Proof Distributed Hash Table Chris Lesniewski-LaasM. Frans Kaashoek MIT 28 April 2010 NSDI
P2P Search COP6731 Advanced Database Systems. P2P Computing  Powerful personal computer Share computing resources P2P Computing  Advantages: Shared.
1 Secure Peer-to-Peer File Sharing Frans Kaashoek, David Karger, Robert Morris, Ion Stoica, Hari Balakrishnan MIT Laboratory.
Md Tareq Adnan Centralized Approach : Server & Clients Slow content must traverse multiple backbones and long distances Unreliable.
Data Structure of Chord  For each peer -successor link on the ring -predecessor link on the ring -for all i ∈ {0,..,m-1} Finger[i] := the peer following.
CS694 - DHT1 Distributed Hash Table Systems Hui Zhang University of Southern California.
Simple Load Balancing for Distributed Hash tables
Ion Stoica, Robert Morris, David Liben-Nowell, David R. Karger, M
Early Measurements of a Cluster-based Architecture for P2P Systems
EE 122: Peer-to-Peer (P2P) Networks
Building Peer-to-Peer Systems with Chord, a Distributed Lookup Service
Distributed Hash Tables
Consistent Hashing and Distributed Hash Table
P2P: Distributed Hash Tables
#02 Peer to Peer Networking
Presentation transcript:

Distributed Hash Tables David Tam Patrick Pang

Presentation Outline What is DHT (Distributed Hash Table)? Why DHTs? Applications How lookup works? Alternatives to DHTs Performance – Routing Performance – Load Balancing Security – Routing Attack Security – Inconsistent Behaviour Comparison to Other Facilities Current Research Projects Conclusion

Distributed application Distributed hash table What is DHT? Distributed application put(key, data) get (key) data Distributed hash table node …. DHT provides the information look up service for P2P applications. Nodes uniformly distributed across key space Nodes form an overlay network Nodes maintain list of neighbours in routing table Decoupled from physical network topology Make it simple to write decentralized, distributed applications ….. (Figure adopted from Frans Kaashoek)

Why DHTs? Why Middleware? Simplifies the development for large-scale distributed Apps Better security and robustness Simple API Why Do We Need DHTs? Simplifies the development for large-scale distributed Apps Better security and robustness Simple API Exploits P2P resources

Applications Anything that requires a hash table Databases, FSes, storage, archival Web serving, caching Content distribution Query & indexing Naming systems Communication primitives Chat services Application-layer multi-casting Event notification services Publish/subscribe systems ?

How lookup works? start interval succ. 3 [3,4) 5 4 [4,6) 6 [6,10) 7 10 Example: Chord [Stoica et. al.] 1 15 Finger Table for Node 2 2 14 3 start interval succ. 3 [3,4) 5 4 [4,6) 6 [6,10) 7 10 [10,2) 13 4 12 5 11 10 6 7 9 8

How lookup works? start interval succ. 11 [11,12) 12 [12,14) 14 [14,2) Example: Chord 1 15 Finger Table for Node 10 2 14 3 start interval succ. 11 [11,12) 12 [12,14) 14 [14,2) 2 [2,10) 13 4 12 5 11 10 6 7 9 8

How lookup works? start interval succ. 11 [11,12) 12 [12,14) 14 [14,2) Example: Chord 1 15 Finger Table for Node 10 2 14 3 start interval succ. 11 [11,12) 12 [12,14) 14 [14,2) 2 [2,10) 13 4 12 5 11 10 6 7 9 8

How lookup works? start interval succ. 15 [15,0) [0,2) 1 2 [2,6) 6 Example: Chord 1 15 Finger Table for Node 14 2 14 3 start interval succ. 15 [15,0) [0,2) 1 2 [2,6) 6 [6,13) 7 13 4 12 5 11 10 6 7 9 8

How lookup works? start interval succ. 15 [15,0) [0,2) 1 2 [2,6) 6 Example: Chord 1 15 Finger Table for Node 14 2 14 3 start interval succ. 15 [15,0) [0,2) 1 2 [2,6) 6 [6,13) 7 13 4 12 5 11 10 6 7 9 8

How lookup works? Example: Chord 1 15 2 14 3 1 15 2 14 3 Now Node 2 can retrive information for key 0 from Node 1. 4 12 5 11 10 6 7 9 8

Alternatives to DHTs N4 N6 N9 N7 N8 N3 N2 N1 N10 N4 N6 N9 N7 N8 N3 N2 Target Start N6 N9 N7 N8 N3 N2 N1 N10 Distributed file system Centralized lookup P2P flooding queries Server Client Internet N4 Target Start N6 N9 N7 N8 N3 N2 N1 N10 DB (Figures adopted from Frans Kaashoek)

Performance -- Lookup Purpose -- to locate a target node Each step, try to get closer to locating target node Ask a closer neighbour Performance & scalability tied directly to lookup algorithm 2 Aspects to Performance Path latency Lookup path length (# hops) 2 Aspects to Scalability size of routing table – O(log N) lookup path length – O(log N) 3 Techniques proximity lookup proximity neighbour selection geographic layout

Performance -- Load Balancing Issues Hot-spots Content Lookup Heterogeneous nodes & paths System flux Solution Replication is the key Also good for fault-tolerance Cache lookup answers backwards along path

Security – Incorrect Lookup (1) When asked for the “next hop”, give a wrong answer Finger Table for Node 2 1 15 start interval succ. 3 [3,4) 5 4 [4,6) 6 [6,10) 7 10 [10,2) 2 14 3 13 4 12 5 Node 2 to Node 10: Please tell me how to reach key 0 …. 11 10 6 7 9 8

Security – Incorrect Lookup (2) When asked for the “next hop”, give a wrong answer Finger Table for Node 10 1 15 start interval succ. 11 [11,12) 12 [12,14) 14 [14,2) 2 [2,10) 2 14 3 13 4 12 5 Node 2 to Node 10: Please tell me how to reach key 0 …. Node 10 answers: ask Node 14 11 10 6 7 9 8

Security – Incorrect Lookup (3) When asked for the “next hop”, give a wrong answer Finger Table for Node 14 1 15 start interval succ. 15 [15,0) [0,2) 1 2 [2,6) 6 [6,13) 7 2 14 3 13 4 12 5 Node 2 to Node 14: Please tell me how to reach key 0 …. Node 14 answers: ask Node 10 11 10 6 7 9 8

Security – Incorrect Lookup (4) Solution [Sit and Morris]: “Define verifiable system invariant” “Allow the querier to observe lookup progress” Our idea how this can be implemented: Concretely, using an integral monotonically decreasing quantity to implement the idea of “progress”. The concept of “monotonically decreasing quantity” has been used in program construction guaranteeing total correctness. [Parnas]

Security – Inconsistent Behaviour Inconsistent Behaviour, i.e., lie intelligibly Sybil attack [Kaashoek] Solution 1: public key solution

Security – Inconsistent Behaviour Inconsistent Behaviour, i.e., lie intelligibly Sybil attack [Kaashoek] Solution 1: public key solution Solution 2: Byzantine Protocol Byzantine Generals Problem: How to find out the traitors among the Generals? [Lamport]

Security – Inconsistent Behaviour Inconsistent Behaviour, i.e., lie intelligibly Sybil attack [Kaashoek] Solution 1: public key solution Solution 2: Byzantine Protocol Commander Lieutenant 1 Lieutenant 2 Byzantine Generals Problem: How to find out the traitors among the Generals? [Lamport] attack “he said ‘retreat’”

Security – Inconsistent Behaviour Inconsistent Behaviour, i.e., lie intelligibly Sybil attack [Kaashoek] Solution 1: public key solution Solution 2: Byzantine Protocol Commander Lieutenant 1 Lieutenant 2 Byzantine Generals Problem: How to find out the traitors among the Generals? [Lamport] attack retreat “he said ‘retreat’”

Comparison to Other Facilities Facility Abstraction Easy Use/Prg Scalability Load-Balance DHT high yes Centralized Lookup medium low no P2P flooding queries Distributed FS Facility Fault-Tolerance Self-Org Admin DHT high yes low Centralized Lookup no medium P2P flooding queries depends Distributed FS

Research Projects Iris – security & fault-tolerance – US Gov’t Chord – circular key space Pastry – circular key space Tapestry – hypercube space CAN – n-dimensional key space Kelips – n-dimensional key space DDS -- middleware platform for internet service construction -- cluster-based -- incremental scalability

Summary Good middleware platform Exploits P2P networks An exciting new research area

References Lamport, Leslie et. al. The Byzantine Generals Problem Sit, Emil, Morris, Robert. Security Considerations for Peer-to-Peer Distributed Hash Tables Kaashoek, Frans. Distributed Hash Tables – Building large-sacle, robust distributed applications Stoica, Ion et. al. Chord: A scalable peer-to-peer lookup service for Internet applications Parnas, D. L. Connecting Theory to Practice: Software Engineering Programme