1. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 1 Principles of Reliable Distributed Systems Lecture 2: Peer-to-Peer (P2P) Lookup Systems Spring 2007 Idit Keidar

2. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 2 Today’s Material: Papers Looking up Data in P2P Systems –Balakrishnan et al. Chord: A Scalable Peer-to-peer Lookup Service for Internet Applications –Stoica et al.

3. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 3 What Does Peer-to-Peer Mean? Characterization from CFP of IPTPS 2004: –decentralized, –self-organizing –distributed systems, –in which all or most communication is symmetric.

4. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 4 Typical Characteristics Lots of nodes (e.g., millions) Dynamic: frequent join, leave, failure Little or no infrastructure –no central server Communication possible between every pair of nodes (cf. the Internet) All nodes are “peers” – have same role; don’t have lots of resources

5. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 5 The Main Challenge To design and implement a robust and scalable distributed system composed of inexpensive, individually unreliable computers in unrelated administrative domains “Looking up Data in P2P Systems”, Balakrishnan et al., CACM Feb 2003

6. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 6 It All Started with Lookup Goal: Make billions of objects available to millions of concurrent users –e.g., music files Need a mechanism to keep track of them –map files to their locations First There was Napster –centralized server/database –pros and cons?

7. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 7 Traditional Scalability Solution Hierarchy –tree overlay: organize nodes into a spanning tree; communicate on links of the tree –structured lookup: know where to forward the query next –e.g., DNS. Pros and cons?

8. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 8 Overlay Networks A virtual structure imposed over the physical network (e.g., the Internet) –over the Internet, there is a (IP level) unicast channel between every pair of hosts –an overlay uses a fixed subset of these –nodes that have the capability to communicate directly with each other do not use it What is this good for?

9. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 9 Symmetric Lookup Algorithms All nodes were created equal No hierarchy –overlay not a tree So how does the search go? –depends….

10. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 10 Searching in Overlay Networks Take I: Gnutella Build a decentralized unstructured overlay –each node has several neighbors; –holds several keys in its local database When asked to find a key X –check local database if X is known –if yes, return, if not, ask your neighbors What is the communication pattern?

11. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 11 Resolve Query by Flooding 2 3 S 2 3 3 4 5 2 3 4 4 5 6 6 4 X 5 3 5 5 4 4 Time-To-Live (TTL)=5 would have been enough

12. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 12 How Come It Works? Search is fast –what people care about People don’t care so much about wasting bandwidth –may change as ISPs start charging for bandwidth Scalability is limited –normally no more than ~40,000 peers Files are replicated many times so flooding with a small TTL usually finds the file –even if there are multiple connected components

13. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 13 Take II: FastTrack, KaZaA, eDonkey Improve scalability by re-introducing a hierarchy –though not a tree –super-peers have more resources, more neighbors, know more keys –search goes through super-peers Pros and Cons?

14. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 14 Structured Lookup Overlays Many recent academic systems – –CAN, Chord, D2B, Kademlia, Koorde, Pastry, Tapestry, Viceroy, … OverNet based on the Kademlia algorithm Symmetric, no hierarchy Decentralized self management Structured overlay – data stored in a defined place, search goes on a defined path Implement Distributed Hash Table (DHT) abstraction

15. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 15 Reminder: Hashing Data structure supporting the operations: –void insert( key, item ) –item search( key ) Implementation uses hash function for mapping keys to array cells Expected search time O(1) –provided that there are few collisions

16. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 16 Distributed Hash Tables (DHTs) Nodes store table entries –the role of array cells Good abstraction for lookup? Why? Requirements for an application being able to use DHTs? –data identified with unique keys. –nodes can (agree to) store keys for each other location of object or actual object

17. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 17 The DHT Service Interface lookup( key ) returns the location of the node currently responsible for this key key is usually numeric (in some range)

18. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 18 Using the DHT Interface How do you publish a file? How do you find a file?

19. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 19 What Does a DHT Implementation Need to Do? Map keys to nodes –needs to be dynamic as nodes join and leave –how does this affect the service interface? Route a request to the appropriate node –routing on the overlay

20. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 20 Lookup Example K V insert (K 1,V 1 ) K V (K 1,V 1 ) lookup(K 1 )

21. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 21 Mapping Keys to Nodes Goal: load balancing –why? Typical approach: –give an m-bit identifier to each node and each key (e.g., using SHA-1 on the key, IP address) –map key to node whose id is “close” to the key (need distance function). –how is load balancing achieved?

22. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 22 Routing Issues Each node must be able to forward each lookup query to a node closer to the destination Maintain routing tables adaptively –each node knows some other nodes –must adapt to changes (joins, leaves, failures) –goals?

23. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 23 Handling Join/Leave When a node joins it needs to assume responsibility for some keys –ask the application to move these keys to it –how many keys will need to be moved? When a nodes fails or leaves, its keys have to be moved to others –what else is needed in order to implement this?

24. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 24 P2P System Interface Lookup Join Move keys

25. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 25 Chord Stoica, Morris, Karger, Kaashoek, and Balakrishnan

26. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 26 Chord Logical Structure m-bit ID space (2 m IDs), usually m=160. Think of nodes as organized in a logical ring according to their IDs. N1 N8 N10 N14 N21 N30 N38 N42 N48 N51 N56

27. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 27 Consistent Hashing: Assigning Keys to Nodes Key k is assigned to first node whose ID equals or follows k – successor(k) N1 N8 N10 N14 N21 N30 N38 N42 N48 N51 N56 K54

28. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 28 Moving Keys upon Join/Leave When a node joins, it becomes responsible for some keys previously assigned to its successor –local change –assuming load is balanced, how many keys should move? And what happens when a node leaves?

29. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 29 Consistent Hashing Guarantees For any set of N nodes and K keys, with high probability: –Each node is responsible for at most (1 +  )K/N keys –When an (N + 1)st node joins or leaves the network, responsibility for O(K/N) keys changes hands (and only to or from the joining or leaving node). When consistent hashing is implemented as described above, the bound is with  = O(logN). The consistent hashing paper shows that can be reduced to an arbitrarily small constant by having each node run (logN) virtual nodes, each with its own identifier.

30. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 30 Simple Routing Solutions Each node knows only its successor. –routing around the circle Each node knows all other nodes –O(1) routing –cost?

31. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 31 Chord Skiplist Routing Each node has “fingers” to nodes ½ way around the ID space from it, ¼ the way… finger[i] at n contains successor(n+2 i-1 ) successor is finger[1] N0 N8 N10 N14 N21 N30 N38 N42 N48 N51 N56 How many fingers in the finger table?

32. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 32 Example: Chord Fingers N0 N10 N21 N30 N47 finger[1..4] N72 N82 N90 N114 finger[5] finger[6] finger[7] m entries log N distinct fingers with high probability

33. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 33 Chord Data Structures (At Each Node) Finger table First finger is successor Predecessor

34. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 34 Forwarding Queries Query for key k is forwarded to finger with highest ID not exceeding k K54 Lookup( K54 ) N0 N8 N10 N14 N21 N30 N38 N42 N48 N51 N56

35. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 35 How long does it take? Remote Procedure Call (RPC)

36. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 36 Routing Time Node n looks up a key stored at node p p is in n’s ith interval: p  ((n+2 i-1 )mod 2 m, (n+2 i )mod 2 m ] n contacts f=finger[i] –the interval is not empty (because p is in it) so: f  ((n+2 i-1 )mod 2 m, (n+2 i )mod 2 m ] –RPC f f is at least 2 i-1 away from n p is at most 2 i-1 away from f The distance is halved: maximum m steps

37. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 37 Routing Time Refined Assuming uniform node distribution around the circle, the number of nodes in the search space is halved at each step: –expected number of steps: log N Note that: –m = 160 –for 1,000,000 nodes, log N = 20

38. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 38 Joining Chord Goals? Required steps: –find your successor –initialize finger table and predecessor –notify other nodes that need to change their finger table and predecessor pointer O(log 2 N) –learn the keys that you are responsible for; notify others that you assume control over them

39. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 39 Join Algorithm: Take II Observation: for correctness, successors suffice –fingers only needed for performance Upon join, update successor only Periodically, –check that successors and predecessors are consistent –fix fingers

40. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 40 Creation and Join

41. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 41

42. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 42 Join Example joiner finds successor gets keys stabilize fixes successor stabilize fixes predecessor

43. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 43 Join Stabilization Guarantee If any sequence of join operations is executed interleaved with stabilizations, then at some time after the last join the successor pointers will form a cycle on all the nodes in the network. Model assumptions?

44. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 44 Performance with Concurrent Joins If we take a stable network with N nodes with correct finger pointers, and another set of up to N nodes joins the network, and all successor pointers (but perhaps not all finger pointers) are correct, then lookups will still take O(logN) time with high probability. Model assumptions?

45. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 45 Failure Handling Periodically fixing fingers List of r successors instead of one successor Periodically probing predecessors:

46. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 46 The Model? Reliable messages among correct nodes –No network partitions Node failures can be accurately detected! Properties hold as long as failure is bounded: –If we use a successor list of length r = (logN) in a network that is initially stable, and then every node fails with probability 1/2, then with high probability find successor returns the closest living successor to the query key. –In a network that is initially stable, if every node then fails with probability 1/2, then the expected time to execute find successor is O(logN).

47. Idit Keidar, Principles of Reliable Distributed Systems, Technion EE, Spring 2007 47 What About Moving Keys? Left up to the application Solution: keep soft state, refreshed periodically –every refresh operation performs lookup(key) before storing the key in the right place How can we increase reliability for the time between failure and refresh?