1. Symmetric Replication in Structured Peer-to-Peer Systems Ali Ghodsi, Luc Onana Alima, Seif Haridi

2. 29 July 2005 ali@sics.se2 Scope of the Talk Where to place replicas in a DHT?

3. 29 July 2005 ali@sics.se3 Background DKS System (http://dks.sics.se/)http://dks.sics.se/ –Were initially replicating on successor-lists (Similar to Chord/Koorde) –Turned out to have several problems…

4. 29 July 2005 ali@sics.se4 Outline Background Proposed scheme: symmetric replication Extensions Related Work & Summary

5. 29 July 2005 ali@sics.se5 What is Replication on the Successor-List? In one sentence: –Replicate node n ’s data to its f closest neighbors for replication degree f

6. 29 July 2005 ali@sics.se6 Successor-list Example: Chord A Chord-alike system, with nodes 0, 3, 5, 10, 11 0 1 2 15 14 133 12 11 4 5 6 9 8 7 10

7. 29 July 2005 ali@sics.se7 Successor-list Example: Chord With a replication factor of f=1 0 1 2 15 14 133 12 11 4 5 6 9 8 7 10 Data: 12, 13, 14, 15, 0 Data: 1, 2, 3 Data: 4, 5 Data: 11 Data: 6, 7, 8, 9, 10

8. 29 July 2005 ali@sics.se8 Successor-list Example: Chord With a replication factor of f=2 0 1 2 15 14 133 12 11 4 5 6 9 8 7 10 Data: 12, 13, 14, 15, 0 Data: 1, 2, 3 Data: 4, 5 Data: 11 Data: 6, 7, 8, 9, 10 Node 0: 12, 13, 14, 15, 0 Node 3: 1, 2, 3 Node 5: 4, 5 Node 10: 6, 7, 8, 9, 10 Node 11: 11

9. 29 July 2005 ali@sics.se9 Successor-list Example: Chord With a replication factor of f=3 0 1 2 15 14 133 12 11 4 5 6 9 8 7 10 Data: 12, 13, 14, 15, 0 Data: 1, 2, 3 Data: 4, 5 Data: 11 Data: 6, 7, 8, 9, 10 Node 0: 12, 13, 14, 15, 0 Node 3: 1, 2, 3 Node 5: 4, 5 Node 10: 6, 7, 8, 9, 10 Node 11: 11 Node 0: 12, 13, 14, 15, 0 Node 3: 1, 2, 3 Node 5: 4, 5 Node 10: 6, 7, 8, 9, 10

10. 29 July 2005 ali@sics.se10 Successor-list Example: Chord With a replication factor of f=4 0 1 2 15 14 133 12 11 4 5 6 9 8 7 10 Data: 12, 13, 14, 15, 0 Data: 1, 2, 3 Data: 4, 5 Data: 11 Data: 6, 7, 8, 9, 10 Node 0: 12, 13, 14, 15, 0 Node 3: 1, 2, 3 Node 5: 4, 5 Node 10: 6, 7, 8, 9, 10 Node 11: 11 Node 0: 12, 13, 14, 15, 0 Node 3: 1, 2, 3 Node 5: 4, 5 Node 10: 6, 7, 8, 9, 10 Node 11: 11 Node 0: 12, 13, 14, 15, 0 Node 3: 1, 2, 3 Node 5: 4, 5 Node 10: 6, 7, 8, 9, 10

11. 29 July 2005 ali@sics.se11 Problem: Dynamism What happens in case of leaves or failures? f=4 0 1 2 15 14 133 12 11 4 5 6 9 8 7 10 Data: 12, 13, 14, 15, 0 Data: 1, 2, 3 Data: 4, 5 Data: 11, 10, 9, 8, 7, 6 Node 0: 12, 13, 14, 15, 0 Node 3: 1, 2, 3 Node 10: 6, 7, 8, 9, 10 Node 11: 11 Node 0: 12, 13, 14, 15, 0 Node 5: 4, 5 Node 10: 6, 7, 8, 9, 10 Node 11: 11 Node 3: 1, 2, 3 Node 5: 4, 5 Node 10: 6, 7, 8, 9, 10

12. 29 July 2005 ali@sics.se12 Problem: Dynamism What happens in case of leaves or failures? f=4 0 1 2 15 14 133 12 11 4 5 6 9 8 7 10 Data: 12, 13, 14, 15, 0 Data: 1, 2, 3 Data: 4, 5 Data: 11, 10, 9, 8, 7, 6 Node 0: 12, 13, 14, 15, 0 Node 3: 1, 2, 3 Node 11: 11,10,9,8,7,6 Node 0: 12, 13, 14, 15, 0 Node 5: 4, 5 Node 11: 11 Node 3: 1, 2, 3 Node 5: 4, 5

13. 29 July 2005 ali@sics.se13 Problem: Dynamism Need to move items to restore replication degree! 0 1 2 15 14 133 12 11 4 5 6 9 8 7 10 Data: 12, 13, 14, 15, 0 Data: 1, 2, 3 Data: 4, 5 Data: 11, 10, 9, 8, 7, 6 Node 0: 12, 13, 14, 15, 0 Node 3: 1, 2, 3 Node 11: 11,10,9,8,7,6 Node 0: 12, 13, 14, 15, 0 Node 5: 4, 5 Node 11: 11,10, 9, 8, 7, 6 Node 3: 1, 2, 3 Node 5: 4, 5 Get items Data: 4, 5 Node 5: 4, 5

14. 29 July 2005 ali@sics.se14 Problem: Dynamism Need to move items to restore replication degree! 0 1 2 15 14 133 12 11 4 5 6 9 8 7 10 Data: 12, 13, 14, 15, 0 Data: 1, 2, 3 Data: 4, 5 Data: 11, 10, 9, 8, 7, 6 Node 0: 12, 13, 14, 15, 0 Node 3: 1, 2, 3 Node 11: 11,10,9,8,7,6 Node 0: 12, 13, 14, 15, 0 Node 5: 4, 5 Node 11: 11,10, 9, 8, 7, 6 Node 3: 1, 2, 3 Node 5: 4, 5 Get items Data: 1, 2, 3 Node 5: 4, 5 Node 3: 1, 2, 3

15. 29 July 2005 ali@sics.se15 Problem: Dynamism Need to move items to restore replication degree! 0 1 2 15 14 133 12 11 4 5 6 9 8 7 10 Data: 12, 13, 14, 15, 0 Data: 1, 2, 3 Data: 4, 5 Data: 11, 10, 9, 8, 7, 6 Node 0: 12, 13, 14, 15, 0 Node 3: 1, 2, 3 Node 11: 11,10,9,8,7,6 Node 0: 12, 13, 14, 15, 0 Node 5: 4, 5 Node 11: 11,10, 9, 8, 7, 6 Node 3: 1, 2, 3 Node 5: 4, 5 Get items Data: 12, 13, 14, 15, 0 Node 5: 4, 5 Node 3: 1, 2, 3 Node 0: 12,13,14,15,0

16. 29 July 2005 ali@sics.se16 Problem: Dynamism Need to move items to restore replication degree! 0 1 2 15 14 133 12 11 4 5 6 9 8 7 10 Data: 12, 13, 14, 15, 0 Data: 1, 2, 3 Data: 4, 5 Data: 11, 10, 9, 8, 7, 6 Node 0: 12, 13, 14, 15, 0 Node 3: 1, 2, 3 Node 11: 11,10,9,8,7,6 Node 0: 12, 13, 14, 15, 0 Node 5: 4, 5 Node 11: 11,10, 9, 8, 7, 6 Node 3: 1, 2, 3 Node 5: 4, 5 Get items Data: 10, 9, 8, 7, 6

17. 29 July 2005 ali@sics.se17 Problem: Dynamism Every join/leave/failure requires: –  (f) messages, to restore a replication degree of f –Need coordination of the restoration What if the coordinator fails?

18. 29 July 2005 ali@sics.se18 Problem: Performance & Security How do we contact 2 nd, 3 rd, 4 th … replica? –Have to contact the 1 st replica! The 1 st replica is –Bottleneck –Can fail –Can be malicious (distributed voting useless)

19. 29 July 2005 ali@sics.se19 Outline Background Proposed scheme: symmetric replication Extensions Related Work & Summary

20. 29 July 2005 ali@sics.se20 Symmetric Replication Goal of Symmetric Replication: –Simple algorithms Each join/leave should require O(1) messages –Enable concurrent requests Load-balancing without a bottleneck –Increase security Use distributed voting –Add proximity information Route through closest replica

21. 29 July 2005 ali@sics.se21 Symmetric Replication Idea: –For replication degree f, partition the identifier space into m equivalence classes: Use the congruence classes modulo f –The cardinality of each class is f –Each node replicates the equivalence class of every identifier it is responsible for

22. 29 July 2005 ali@sics.se22 Symmetric replication Replication degree f=4, Id={0,…,15} 4 equivalence classes: 0 1 2 15 14 133 12 11 4 5 6 9 8 7 10 Data: 15, 0 Data: 1, 2, 3 Data: 4, 5 Data: 14, 13, 12, 11 Data: 6, 7, 8, 9, 10 0, 4, 8, 12 1, 5, 9, 13 2, 6, 10, 14 3, 7, 11, 15

23. 29 July 2005 ali@sics.se23 Symmetric replication Replication degree f=4, Id={0,…,15} 4 equivalence classes: 0 1 2 15 14 133 12 11 4 5 6 9 8 7 10 Data: 15, 0 Data: 1, 2, 3 Data: 4, 5 Data: 14, 13, 12, 11 Data: 6, 7, 8, 9, 10 0, 4, 8, 12 1, 5, 9, 13 2, 6, 10, 14 3, 7, 11, 15 Data: 10, 9, 8, 7 Data: 11, 12 Data: 13, 14, 15 Data: 0, 1 Data: 2, 3, 4, 5, 6 Data: 6, 5, 4, 3 Data: 2, 1, 0, 15 Data: 7, 8 Data: 3, 4 Data: 9, 10, 11 Data: 5, 6, 7 Data: 12, 13 Data: 8, 9 Data: 14, 15, 0, 1, 2 Data: 10, 11, 12, 13, 14

24. 29 July 2005 ali@sics.se24 Symmetric replication: joins! Replication degree f=4, Id={0,…,15} 4 equivalence classes: 0 1 2 15 14 133 12 11 4 5 6 9 8 7 10 Data: 15, 0 Data: 1, 2, 3 Data: 4, 5 Data: 14, 13, Data: 6, 7, 8, 9, 10 0, 4, 8, 12 1, 5, 9, 13 2, 6, 10, 14 3, 7, 11, 15 Data: 10, 9 Data: 11, 12 Data: 13, 14, 15 Data: 0, 1 Data: 2, 3, 4, 5, 6 Data: 6, 5 Data: 2, 1 Data: 7, 8 Data: 3, 4 Data: 9, 10, 11 Data: 5, 6, 7 Data: 12, 13 Data: 8, 9 Data: 14, 15, 0, 1, 2 Data: 10, 11, 12, 13, 14 Data: 11, 12 Data: 7, 8 Data: 3, 4 Data: 0, 15 Get items Data: 11, 12, 7, 8, 3, 4, 0, 15

25. 29 July 2005 ali@sics.se25 Symmetric replication: failures 0 1 2 15 14 133 12 11 4 5 6 9 8 7 10 Data: 15, 0 Data: 1, 2, 3 Data: 4, 5 Data: 14, 13, Data: 6, 7, 8, 9, 10 0, 4, 8, 12 1, 5, 9, 13 2, 6, 10, 14 3, 7, 11, 15 Data: 10, 9 Data: 11, 12 Data: 13, 14, 15 Data: 0, 1 Data: 2, 3, 4, 5, 6 Data: 6, 5 Data: 2, 1 Data: 7, 8 Data: 3, 4 Data: 9, 10, 11 Data: 5, 6, 7 Data: 12, 13 Data: 8, 9 Data: 14, 15, 0, 1, 2 Data: 10, 11, 12, 13, 14 Successor has to fetch [11], [12] from closest replica

26. 29 July 2005 ali@sics.se26 Restoring Replication Degee: failures Assume a node responsible for (10, 14) fails –Classes [10]={1, 10, 19} [14]={5, 14, 23} We decide to fetch replicas from (1, 5) Use restricted broadcast to fetch (1, 5) –[ansary-iptps02, ghodsi-pdcs02] –Reliable, O(log R) time complexity to reach R nodes Several nodes might be responsible for (1, 5) –Given uniform distribution of node identifiers E[ peers in range (1,5)] = 1

27. 29 July 2005 ali@sics.se27 Outline Background Proposed scheme: symmetric replication Extensions Related Work & Summary

28. 29 July 2005 ali@sics.se28 Extensions to Symmetric Replication Augment the routing information to: –Add proximity awareness Example: node 2, f=3 [2]={2, 5, 8} Routing Table: node 2 Finger 1 Succ(2+2 0 ) Finger 2 Succ(2+2 1 ) Finger 3 Succ(2+2 2 ) Finger 4 Succ(2+2 3 )

29. 29 July 2005 ali@sics.se29 Extensions to Symmetric Replication Augment the routing information to: –Add proximity awareness Example: node 2, f=3 [2]={2, 5, 8} Routing Table: node 2 Finger 1 Succ(2+2 0 )Succ(5+2 0 )Succ(8+2 0 ) Finger 2 Succ(2+2 1 )Succ(5+2 1 )Succ(8+2 1 ) Finger 3 Succ(2+2 2 )Succ(5+2 2 )Succ(8+2 2 ) Finger 4 Succ(2+2 3 )Succ(5+2 3 )Succ(8+2 3 )

30. 29 July 2005 ali@sics.se30 Extensions: Erasure Codes –Symmetric Replication with Erasure Codes –Encode data into n blocks, only k needed to reconstruct original data Example: n=10, k=5 –Replication degree= n ( # encoded blocks) fetch any k blocks to reconstruct data

31. 29 July 2005 ali@sics.se31 Outline Background Proposed scheme: symmetric replication Extensions Related Work & Summary

32. 29 July 2005 ali@sics.se32 Related Work Several systems use multiple hash functions –Content Addressable Networks (CAN) –Tapestry –Simple Load Balancing in Chord

33. 29 July 2005 ali@sics.se33 Problem with Multiple Hash Functions Hash 1 (“music.mp3”)=11 Hash 2 (“music.mp3”)=2 0 1 2 15 14 133 12 11 4 5 6 9 8 7 10 Data: 15, 0 Data: 1, 2, 3 Data: 4, 5 Data: 14, 13, Data: 6, 7, 8, 9, 10 Node 12 crashed, We need the inverse of Hash 1 -1 (11)=“music.mp3”

34. 29 July 2005 ali@sics.se34 Summary A general replica placement scheme –Algorithms for join, leave, fail, insert in paper O(1) messages to restore replication degree f instead of  (f) Can access any replica without a bottleneck: –Concurrent Requests –Load balance queries –Distributed Voting Current Use: –Implemented in DKS, http://dks.sics.se/http://dks.sics.se/

35. 29 July 2005 ali@sics.se35 Thank you for listening!