1. Distributed Search with Rendezvous Search Systems Christof Leng Dipl.-Inform. Christof Leng Databases & Distributed Systems Department of Computer Science Technische Universität Darmstadt http://www.dvs.tu-darmstadt.de cleng@dvs.tu-darmstadt.de

2. Distributed Search: A Traditional View Christof Leng | Rendezvous Search Systems | KuVS/ITG Fachgespräch "Future Internet" 2 | 27.01.12 |  Implement distributed key-value lookup  e.g., a distributed hash table (DHT)  scalable & simple  Build everything else on top of it  keyword search, XPath, range queries  map every operation to a number of lookups  Downside:  implement every query language yourself  query processing not portable between DHTs  unforeseen side-effects & interactions  Experts needed for every application  application domain  P2P networking  information retrieval Application Developer P2P Networking Expert Information Retrieval Expert … need sophisticated queries …

3. Rendezvous Search  “Every query will meet each data item somewhere in the network”  don’t worry how it gets there  black box approach  Implement the match functionality  as if it was local  re-use existing libraries  Advantages:  separation of concerns  flexible  scalable (O(√n) search cost)  robust (O(√n) replicas) | 27.01.12 | Christof Leng | Rendezvous Search Systems | KuVS/ITG Fachgespräch "Future Internet" 3 Application Developer Rendezvous Search System Information Retrieval Library … need sophisticated queries … glue code

4. Example Application Christof Leng | Rendezvous Search Systems | KuVS/ITG Fachgespräch "Future Internet" 4 | 27.01.12 | P2P Wiki Lucene Keyword Search Lucene Keyword Search Rendezvous Middleware Overlay Network

5. Rendezvous Search Genealogy | 27.01.12 | Christof Leng | Rendezvous Search Systems | KuVS/ITG Fachgespräch "Future Internet" 5

6. Data Center Solutions | 27.01.12 | Christof Leng | Rendezvous Search Systems | KuVS/ITG Fachgespräch "Future Internet" 6

7. Grid Pro:  simple  efficient  flexible Con:  not fault-tolerant  not size-adaptive  not load-adaptive  two-dimensional structure Reference: L. A. Barroso, J. Dean, and U. Hölzle. Web Search for a Planet: The Google Cluster Architecture. IEEE Micro, 23(2):22–28, 2003. | 27.01.12 | Christof Leng | Rendezvous Search Systems | KuVS/ITG Fachgespräch "Future Internet" 7 Q Q Q Q Q Q Q Q Q Q Data Query D D D D D D D D D D D D Q Q R R Q Q Q Q Q Q Q Q Q Q Data Query D D D D D D D D Q Q R R Q Q Q Q Q Q

8. ROAR – Rendezvous on a Ring Pro:  simple mapping to 1 dim.  size-adaptive  load-adaptive  fault-tolerant Con:  centralized or  inefficient Reference: C. Raiciu, F. Huici, M. Handley, and D. S. Rosenblum. ROAR: Increasing the Flexibility and Performance of Distributed Search. In Proceedings of SIGCOMM’09, pages 291–302, 2009. ACM. | 27.01.12 | Christof Leng | Rendezvous Search Systems | KuVS/ITG Fachgespräch "Future Internet" 8

9. Structured Peer-to-Peer | 27.01.12 | Christof Leng | Rendezvous Search Systems | KuVS/ITG Fachgespräch "Future Internet" 9

10. Bit Zipper Pro:  z-order mapping  DHT-based = decentralized  fast tree-based replication Con:  relies on DHT’s reliability  no replica maintenance Reference: W. W. Terpstra, S. Behnel, L. Fiege, J. Kangasharju, and A. Buchmann. Bit Zipper Rendezvous—Optimal Data Placement for General P2P Queries. In EDBT’04 Workshop on Peer-to-Peer Computing and Databases, 2004. | 27.01.12 | Christof Leng | Rendezvous Search Systems | KuVS/ITG Fachgespräch "Future Internet" 10

11. Deetoo Pro:  2 rings for projection  includes replica maintenance Con:  maintenance overhead for 2 rings  no responsibility ranges (probabilistic approach)  additional overhead Reference: T. W. Choi and P. O. Boykin. Deetoo: Scalable unstructured Search built on a Structured Overlay. In HOTP2P - International Workshop on Hot Topics in Peer-to-Peer Systems, Los Alamitos, CA, USA, 2010. IEEE Computer Society. | 27.01.12 | Christof Leng | Rendezvous Search Systems | KuVS/ITG Fachgespräch "Future Internet" 11

12. Unstructured Peer-to-Peer | 27.01.12 | Christof Leng | Rendezvous Search Systems | KuVS/ITG Fachgespräch "Future Internet" 12

13. Ferreira et al. – Random Walks Christof Leng | Rendezvous Search Systems | KuVS/ITG Fachgespräch "Future Internet" 13 | 27.01.12 | Pro:  very resilient topology  proven correctness Con:  probabilistic  additional overhead  slow & fragile random walks  no topology protocol Reference: R. A. Ferreira, M. K. Ramanathan, A. Awan, A. Grama, and S. Jagannathan. Search with Probabilistic Guarantees in Unstructured Peer-to-Peer Networks. In Proceedings of P2P’05, pages 165–172, Washington, DC, USA, 2005. IEEE Computer Society.

14. BubbleStorm Christof Leng | Rendezvous Search Systems | KuVS/ITG Fachgespräch "Future Internet" 14 | 27.01.12 | Pro:  resilient & fast  size-adaptive  load-adaptive  topology & replica maintenance included Con:  probabilistic  additional overhead Reference: W. W. Terpstra, J. Kangasharju, C. Leng, and A. P. Buchmann. BubbleStorm: Resilient, Probabilistic, and Exhaustive Peer-to-Peer Search. In Proceedings of SIGCOMM’07, pages 49–60, New York, NY, USA, 2007. ACM.

15. Hautakorpi & Schultz Christof Leng | Rendezvous Search Systems | KuVS/ITG Fachgespräch "Future Internet" 15 | 27.01.12 | Pro:  unstructured solution on top of DHT  incremental deployment possible Con:  probabilistic  additional overhead  fragile Reference: J. Hautakorpi and G. Schultz. A Feasibility Study of an Arbitrary Search in Structured Peer-to-Peer Networks. In Proceedings of 19th International Conference on Computer Communications and Networks (ICCCN), pages 1–8. IEEE, Aug. 2010.

16. Semi-Structured Peer-to-Peer | 27.01.12 | Christof Leng | Rendezvous Search Systems | KuVS/ITG Fachgespräch "Future Internet" 16

17. SplitQuest Christof Leng | Rendezvous Search Systems | KuVS/ITG Fachgespräch "Future Internet" 17 | 27.01.12 | Pro:  combines unstructured resilience with structured efficiency Con:  probabilistic  lacks analysis and prototype evaluation Reference: P. Lopes and R. A. Ferreira. SplitQuest: Controlled and Exhaustive Search in Peer-to-Peer Networks. In Proceedings of the 9th International Conference on Peer-to-Peer Systems, IPTPS’10, Berkeley, CA, USA, 2010. USENIX Association.

18. Comparison Data CenterStructured NameGridROARBit ZipperDeetoo ProsimpleadaptiveDHT add-onmaintenance Connot adaptivecentralizedmaintenance unclearprobabilistic | 27.01.12 | Christof Leng | Rendezvous Search Systems | KuVS/ITG Fachgespräch "Future Internet" 18 UnstructuredSemi-Structured NameFerreiraBubbleStormHautakorpiSplitQuest Prosimple resilient & adaptive increm. DHT add-onefficient & resilient Confragile & slowprobabilistic immature

19. Q Q Q Q Q Q Q Q Q Q Data Query D D D D D D D D Q Q R R Q Q Q Q Q Q Rendezvous Search & Cloud Computing Christof Leng | Rendezvous Search Systems | KuVS/ITG Fachgespräch "Future Internet" 19 | 27.01.12 |  similar problems  similar requirements  less research history could benefit from  size adaptivity  traffic adaptivity  fault tolerance  algorithm diversity Sharding Key-Value Store Scalability Replication D D D D D D Q Q Q Q Q Q M M Q Q Q Q Q Q

20. Conclusion Christof Leng | Rendezvous Search Systems | KuVS/ITG Fachgespräch "Future Internet" 20 | 27.01.12 |  Rendezvous Search is a powerful abstraction  scalable  flexible  robust  A wide range of solutions is out there  data center  unstructured P2P  structured P2P  semi-structured P2P  Ever needed decentralized but powerful search in a project?  don’t try to reinvent the wheel!

21. Thank you! ? Questions http://www.bubblestorm.net http://www.dvs.tu-darmstadt.de | 27.01.12 | Christof Leng | Rendezvous Search Systems | KuVS/ITG Fachgespräch "Future Internet" 21