1. Databases and Distributed Systems Maintaining Replicas in Unstructured P2P Systems CoNEXT, Madrid, 12/11/2008 Christof Leng, TU Darmstadt Wesley W. Terpstra, TU Darmstadt Bettina Kemme, McGill University Wilhelm Stannat, TU Darmstadt Alejandro P. Buchmann, TU Darmstadt http://www.bubblestorm.net http://www.dvs.tu-darmstadt.de

2. Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk2 Replicate, what?

3. Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk3 Replicas far and small  Our research focuses on peer-to-peer search  The data we replicate is usually small (< 10kB)  Modern unstructured overlays (Sarshar’04, Ferreira’05, BubbleStorm) can have several hundred copies of an object  Why replicate so far?  The more copies, the easier it is to find one  The more providers, the harder to overload them  When a node leaves, its copy is gone with the wind To be clear: this talk is not about replicating files

4. Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk4 Two kinds of replicated data Maintained  “I’m online @132.160.222.1”  “Tell me when a paper is published with ‘P2P’&‘search’”  “I can provide these files”  “I am waiting for event X”  Service Lists  Subscriptions Collective  Wiki articles  Information about physical objects outside the network  Distributed file systems  System backups  “Persistent” information

5. Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk5 Who maintains replicated data? A Maintainer  Ideal for data that should never outlive its owner  The owner can manage it  If the owner crashes, any remaining replicas are junk The Collective  Ideal for data that should live until explicitly deleted  No clear managing authority  Replicas of undeleted objects should remain in the system

6. Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk6 Our Paper: Maintainer-based “Let there be replicas!”

7. Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk7 Maintaining Replicas  We want to be able to  Ensure the system has no junk replicas (of objects whose maintainer has left) – they cause bad search results!  Ensure that there are exactly the number of replicas requested  We need to be able to  Keep junk contained so the system remains useful  Increase/decrease the density of replicas in the system  Hold the density of replicas steady against network churn  Update or destroy all the replicas of an object...you can’t always get what you want…

8. Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk8 An INformal model

9. Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk9 The Good (aka assumptions)  Nodes all run our software – so they mostly cooperate  Computing a sum over participants in the network is easy …and since we do unstructured peer-to-peer:  We don’t care too much about who has which particular replica

10. Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk10 The Bad (aka Reality)  Churn is out of our control  We can’t stop it  Its rate changes over time  We cannot influence participant lifetimes  Nodes sometimes crash  They don’t say good-bye; they just leave  This happens a lot

11. Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk11 The Ugly  Storage Providing Peers crash silently fixed replication is impossible  Guarantee probability distribution? sufficient to prove the correctness of Stochastic algorithms  Maintainers crashes are silent replicas should have been deleted Zero junk is impossible  Perhaps guarantee below a threshold? sufficient to prove the performance of Stochastic algorithms

12. Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk12 Don’t try this at home

13. Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk13 A common maintenance strategy Maintainer: 1.Push desired replicas into system 2.Wait for X minutes 3.Goto 1 Storage Providing Peer:  After Y minutes, replica is deleted

14. Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk14 Why this is bad  What should the parameters X and Y be?  If X is too slow:  If Y is too slow:  If X or Y are too fast, more traffic is expended than necessary  Problem: Churn rate is out of our control (unboundable)  Correctness requires setting X for the worst possible situation (costly)

15. Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk15 Pull on Join

16. Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk16 Observation: Density  When a storage providing peer leaves the system  Replica count might be reduced  Expected replica density remains the same  Holding replica density fixed dodges the problem of crashes  Not just a semantic difference!  Must compensate by replicating whenever peers join  We can adjust the density as the population changes  It is still possible to hold replicas at a fixed value (or √n)

17. Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk17 Our replication algorithm  Maintainer:  Push out initial replicas  Record who has received replicas (superset)  Push extra replicas to increase density  Probabilistically delete existing replicas to decrease density  Storage Providing Peer:  On join, ask randomly selected maintainers for replicas  Accept replicas pushed out from maintainers

18. Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk18 Visual Example “Give me more replicas!” “Hello!” “Bye Bye!” “I don’t need so many.” … “Chow!”

19. Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk19 Convergence to the Binomial

20. Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk20 For the whole network

21. Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk21 Flushing Junk

22. Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk22 Observation: Pulls have no Junk  Recall: Junk is bad  Unnecessarily consumes storage  Results in spurious query results  When a node joins, all the replicas it receives are valid  So, to control junk…  Blow everything away  Reload fresh replicas

23. Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk23 But the cost?!  Sounds expensive  Not all storage providing peers (c)rash, only c %  We require only g % of replicas to be (g)ood  If c=10% and g=80%, the overhead of flushing is  ½ c/(1/g – 1) = 20% extra replica transfers  only applies to especially long-lived storage peers  It might be possible to optimize this cost away. We don’t.  Be careful! Most of the obvious optimizations are wrong  It’s easy to introduce statistical defects that accumulate over time

24. Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk24 When to flush?  Expected number of replicas stored is easy to compute (a sum)  Flush when: stored replicas > expected replicas + tolerable junk  Only one problem:  Peers that happen to store more than average flush earlier  those peers are preferentially destroyed  there are less replicas than there should be

25. Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk25 Independence is needed  P(v stores o | v flushes) = P(v stores o)  This equality fails if a node flushes because it is full  Solution:  Use the flow of replicas through one bucket to flush the other  The buckets’ replica flows are statistically independent: done!

26. Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk26 Distribution of Junk

27. Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk27 Things you’ll find in the paper  How pull on join handles different replication densities  The details of the flush threshold equation  Formal Stochastic proof of correctness  How to support peers with different storage capacities  How to support maintainers behind NAT/firewalls  Cost (in operations) or the proposed algorithms  Simulations of what-if scenarios

28. Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk28 Conclusion  Providing replication guarantees in peer-to-peer is feasible  Strong guarantees are impossible  Probabilistic guarantees are tricky  Seemingly innocent choices result in statistically bad behaviour  Maintainer-based replication is a relevant sub-problem  Required for service lists and query subscriptions  Compared to collective replication, maintainer-based  requires junk control  has an obvious node (maintainer/owner) to manage replication

29. Replicate, what? | An INformal model | Don't try this at home | Pull on Join | Flushing Junk29 Thanks for listening! ? Questions http://www.bubblestorm.net http://www.dvs.tu-darmstadt.de