Symmetric Replication in Structured Peer-to-Peer Systems Ali Ghodsi, Luc Onana Alima, Seif Haridi

29 July 2005 Scope of the Talk Where to place replicas in a DHT?

29 July 2005 Background DKS System ( –Were initially replicating on successor-lists (Similar to Chord/Koorde) –Turned out to have several problems…

29 July 2005 Outline Background Proposed scheme: symmetric replication Extensions Related Work & Summary

29 July 2005 What is Replication on the Successor-List? In one sentence: –Replicate node n ’s data to its f closest neighbors for replication degree f

29 July 2005 Successor-list Example: Chord A Chord-alike system, with nodes 0, 3, 5, 10,

29 July 2005 Successor-list Example: Chord With a replication factor of f= Data: 12, 13, 14, 15, 0 Data: 1, 2, 3 Data: 4, 5 Data: 11 Data: 6, 7, 8, 9, 10

29 July 2005 Successor-list Example: Chord With a replication factor of f= Data: 12, 13, 14, 15, 0 Data: 1, 2, 3 Data: 4, 5 Data: 11 Data: 6, 7, 8, 9, 10 Node 0: 12, 13, 14, 15, 0 Node 3: 1, 2, 3 Node 5: 4, 5 Node 10: 6, 7, 8, 9, 10 Node 11: 11

29 July 2005 Successor-list Example: Chord With a replication factor of f= Data: 12, 13, 14, 15, 0 Data: 1, 2, 3 Data: 4, 5 Data: 11 Data: 6, 7, 8, 9, 10 Node 0: 12, 13, 14, 15, 0 Node 3: 1, 2, 3 Node 5: 4, 5 Node 10: 6, 7, 8, 9, 10 Node 11: 11 Node 0: 12, 13, 14, 15, 0 Node 3: 1, 2, 3 Node 5: 4, 5 Node 10: 6, 7, 8, 9, 10

29 July 2005 Successor-list Example: Chord With a replication factor of f= Data: 12, 13, 14, 15, 0 Data: 1, 2, 3 Data: 4, 5 Data: 11 Data: 6, 7, 8, 9, 10 Node 0: 12, 13, 14, 15, 0 Node 3: 1, 2, 3 Node 5: 4, 5 Node 10: 6, 7, 8, 9, 10 Node 11: 11 Node 0: 12, 13, 14, 15, 0 Node 3: 1, 2, 3 Node 5: 4, 5 Node 10: 6, 7, 8, 9, 10 Node 11: 11 Node 0: 12, 13, 14, 15, 0 Node 3: 1, 2, 3 Node 5: 4, 5 Node 10: 6, 7, 8, 9, 10

29 July 2005 Problem: Dynamism What happens in case of leaves or failures? f= Data: 12, 13, 14, 15, 0 Data: 1, 2, 3 Data: 4, 5 Data: 11, 10, 9, 8, 7, 6 Node 0: 12, 13, 14, 15, 0 Node 3: 1, 2, 3 Node 10: 6, 7, 8, 9, 10 Node 11: 11 Node 0: 12, 13, 14, 15, 0 Node 5: 4, 5 Node 10: 6, 7, 8, 9, 10 Node 11: 11 Node 3: 1, 2, 3 Node 5: 4, 5 Node 10: 6, 7, 8, 9, 10

29 July 2005 Problem: Dynamism What happens in case of leaves or failures? f= Data: 12, 13, 14, 15, 0 Data: 1, 2, 3 Data: 4, 5 Data: 11, 10, 9, 8, 7, 6 Node 0: 12, 13, 14, 15, 0 Node 3: 1, 2, 3 Node 11: 11,10,9,8,7,6 Node 0: 12, 13, 14, 15, 0 Node 5: 4, 5 Node 11: 11 Node 3: 1, 2, 3 Node 5: 4, 5

29 July 2005 Problem: Dynamism Need to move items to restore replication degree! Data: 12, 13, 14, 15, 0 Data: 1, 2, 3 Data: 4, 5 Data: 11, 10, 9, 8, 7, 6 Node 0: 12, 13, 14, 15, 0 Node 3: 1, 2, 3 Node 11: 11,10,9,8,7,6 Node 0: 12, 13, 14, 15, 0 Node 5: 4, 5 Node 11: 11,10, 9, 8, 7, 6 Node 3: 1, 2, 3 Node 5: 4, 5 Get items Data: 4, 5 Node 5: 4, 5

29 July 2005 Problem: Dynamism Need to move items to restore replication degree! Data: 12, 13, 14, 15, 0 Data: 1, 2, 3 Data: 4, 5 Data: 11, 10, 9, 8, 7, 6 Node 0: 12, 13, 14, 15, 0 Node 3: 1, 2, 3 Node 11: 11,10,9,8,7,6 Node 0: 12, 13, 14, 15, 0 Node 5: 4, 5 Node 11: 11,10, 9, 8, 7, 6 Node 3: 1, 2, 3 Node 5: 4, 5 Get items Data: 1, 2, 3 Node 5: 4, 5 Node 3: 1, 2, 3

29 July 2005 Problem: Dynamism Need to move items to restore replication degree! Data: 12, 13, 14, 15, 0 Data: 1, 2, 3 Data: 4, 5 Data: 11, 10, 9, 8, 7, 6 Node 0: 12, 13, 14, 15, 0 Node 3: 1, 2, 3 Node 11: 11,10,9,8,7,6 Node 0: 12, 13, 14, 15, 0 Node 5: 4, 5 Node 11: 11,10, 9, 8, 7, 6 Node 3: 1, 2, 3 Node 5: 4, 5 Get items Data: 12, 13, 14, 15, 0 Node 5: 4, 5 Node 3: 1, 2, 3 Node 0: 12,13,14,15,0

29 July 2005 Problem: Dynamism Need to move items to restore replication degree! Data: 12, 13, 14, 15, 0 Data: 1, 2, 3 Data: 4, 5 Data: 11, 10, 9, 8, 7, 6 Node 0: 12, 13, 14, 15, 0 Node 3: 1, 2, 3 Node 11: 11,10,9,8,7,6 Node 0: 12, 13, 14, 15, 0 Node 5: 4, 5 Node 11: 11,10, 9, 8, 7, 6 Node 3: 1, 2, 3 Node 5: 4, 5 Get items Data: 10, 9, 8, 7, 6

29 July 2005 Problem: Dynamism Every join/leave/failure requires: –  (f) messages, to restore a replication degree of f –Need coordination of the restoration What if the coordinator fails?

29 July 2005 Problem: Performance & Security How do we contact 2 nd, 3 rd, 4 th … replica? –Have to contact the 1 st replica! The 1 st replica is –Bottleneck –Can fail –Can be malicious (distributed voting useless)

29 July 2005 Outline Background Proposed scheme: symmetric replication Extensions Related Work & Summary

29 July 2005 Symmetric Replication Goal of Symmetric Replication: –Simple algorithms Each join/leave should require O(1) messages –Enable concurrent requests Load-balancing without a bottleneck –Increase security Use distributed voting –Add proximity information Route through closest replica

29 July 2005 Symmetric Replication Idea: –For replication degree f, partition the identifier space into m equivalence classes: Use the congruence classes modulo f –The cardinality of each class is f –Each node replicates the equivalence class of every identifier it is responsible for

29 July 2005 Symmetric replication Replication degree f=4, Id={0,…,15} 4 equivalence classes: Data: 15, 0 Data: 1, 2, 3 Data: 4, 5 Data: 14, 13, 12, 11 Data: 6, 7, 8, 9, 10 0, 4, 8, 12 1, 5, 9, 13 2, 6, 10, 14 3, 7, 11, 15

29 July 2005 Symmetric replication Replication degree f=4, Id={0,…,15} 4 equivalence classes: Data: 15, 0 Data: 1, 2, 3 Data: 4, 5 Data: 14, 13, 12, 11 Data: 6, 7, 8, 9, 10 0, 4, 8, 12 1, 5, 9, 13 2, 6, 10, 14 3, 7, 11, 15 Data: 10, 9, 8, 7 Data: 11, 12 Data: 13, 14, 15 Data: 0, 1 Data: 2, 3, 4, 5, 6 Data: 6, 5, 4, 3 Data: 2, 1, 0, 15 Data: 7, 8 Data: 3, 4 Data: 9, 10, 11 Data: 5, 6, 7 Data: 12, 13 Data: 8, 9 Data: 14, 15, 0, 1, 2 Data: 10, 11, 12, 13, 14

29 July 2005 Symmetric replication: joins! Replication degree f=4, Id={0,…,15} 4 equivalence classes: Data: 15, 0 Data: 1, 2, 3 Data: 4, 5 Data: 14, 13, Data: 6, 7, 8, 9, 10 0, 4, 8, 12 1, 5, 9, 13 2, 6, 10, 14 3, 7, 11, 15 Data: 10, 9 Data: 11, 12 Data: 13, 14, 15 Data: 0, 1 Data: 2, 3, 4, 5, 6 Data: 6, 5 Data: 2, 1 Data: 7, 8 Data: 3, 4 Data: 9, 10, 11 Data: 5, 6, 7 Data: 12, 13 Data: 8, 9 Data: 14, 15, 0, 1, 2 Data: 10, 11, 12, 13, 14 Data: 11, 12 Data: 7, 8 Data: 3, 4 Data: 0, 15 Get items Data: 11, 12, 7, 8, 3, 4, 0, 15

29 July 2005 Symmetric replication: failures Data: 15, 0 Data: 1, 2, 3 Data: 4, 5 Data: 14, 13, Data: 6, 7, 8, 9, 10 0, 4, 8, 12 1, 5, 9, 13 2, 6, 10, 14 3, 7, 11, 15 Data: 10, 9 Data: 11, 12 Data: 13, 14, 15 Data: 0, 1 Data: 2, 3, 4, 5, 6 Data: 6, 5 Data: 2, 1 Data: 7, 8 Data: 3, 4 Data: 9, 10, 11 Data: 5, 6, 7 Data: 12, 13 Data: 8, 9 Data: 14, 15, 0, 1, 2 Data: 10, 11, 12, 13, 14 Successor has to fetch [11], [12] from closest replica

29 July 2005 Restoring Replication Degee: failures Assume a node responsible for (10, 14) fails –Classes [10]={1, 10, 19} [14]={5, 14, 23} We decide to fetch replicas from (1, 5) Use restricted broadcast to fetch (1, 5) –[ansary-iptps02, ghodsi-pdcs02] –Reliable, O(log R) time complexity to reach R nodes Several nodes might be responsible for (1, 5) –Given uniform distribution of node identifiers E[ peers in range (1,5)] = 1

29 July 2005 Outline Background Proposed scheme: symmetric replication Extensions Related Work & Summary

29 July 2005 Extensions to Symmetric Replication Augment the routing information to: –Add proximity awareness Example: node 2, f=3 [2]={2, 5, 8} Routing Table: node 2 Finger 1 Succ(2+2 0 ) Finger 2 Succ(2+2 1 ) Finger 3 Succ(2+2 2 ) Finger 4 Succ(2+2 3 )

29 July 2005 Extensions to Symmetric Replication Augment the routing information to: –Add proximity awareness Example: node 2, f=3 [2]={2, 5, 8} Routing Table: node 2 Finger 1 Succ(2+2 0 )Succ(5+2 0 )Succ(8+2 0 ) Finger 2 Succ(2+2 1 )Succ(5+2 1 )Succ(8+2 1 ) Finger 3 Succ(2+2 2 )Succ(5+2 2 )Succ(8+2 2 ) Finger 4 Succ(2+2 3 )Succ(5+2 3 )Succ(8+2 3 )

29 July 2005 Extensions: Erasure Codes –Symmetric Replication with Erasure Codes –Encode data into n blocks, only k needed to reconstruct original data Example: n=10, k=5 –Replication degree= n ( # encoded blocks) fetch any k blocks to reconstruct data

29 July 2005 Outline Background Proposed scheme: symmetric replication Extensions Related Work & Summary

29 July 2005 Related Work Several systems use multiple hash functions –Content Addressable Networks (CAN) –Tapestry –Simple Load Balancing in Chord

29 July 2005 Problem with Multiple Hash Functions Hash 1 (“music.mp3”)=11 Hash 2 (“music.mp3”)= Data: 15, 0 Data: 1, 2, 3 Data: 4, 5 Data: 14, 13, Data: 6, 7, 8, 9, 10 Node 12 crashed, We need the inverse of Hash 1 -1 (11)=“music.mp3”

29 July 2005 Summary A general replica placement scheme –Algorithms for join, leave, fail, insert in paper O(1) messages to restore replication degree f instead of  (f) Can access any replica without a bottleneck: –Concurrent Requests –Load balance queries –Distributed Voting Current Use: –Implemented in DKS,

29 July 2005 Thank you for listening!