1. Network software system laboratory Rana Shahout & Ibrahim Baransi supervisor : Edward Bortnikov Winter 2011 Real-Time Search EngineReal-Time Search Engine

2. Agenda The problem & motivation Background in search systems The architecture CIP policies Software design

3. What? What is the project goal? Serving fresh search results when the data is constantly changing Nowadays websites changes in a high frequency, such as Twitter, Facebook, news.

4. Background in search systems Search caches Why is that a problem ? Search engine uses cache optimization which makes the search engine faster and efficient, when the data a dynamic data, some of cache’s information become irrelevant. Search engines search for the queries first in the cache, and only if there is cache miss they search in the Index. Thus, when the data is dynamic, it is existing in the cache, and the search engine returns UNCORRECT result

5. General picture

6. Why?

7. The Architecture

8. Data structures required for implementation Index - Lucene Index Directory : Lucene is a free text-indexing and -searching API written in Java, a typical Lucene index is stored in a single directory in the file system on a hard disk Cache - It was implemented as a linked-list with hash table. Replacement policy is LRU

9. CIP-- CACHE INVALIDATION PREDICTORS The CIP is formed of two major parts: Synopsis generator is responsible for preparing synopses of the new documents coming in. Invalidator interacts with the runtime system and decides which cached entries to invalidate according to two policies.

10. Invalidation Policies Basic: invalidates each query (in the cache) which appear in the synopsis. Score: Find out all the queries (in the cache) which are contained in the synopsis, for each one of them compute score(q,d)- where d is the added/updated document – and invalidate top K results.

11. Illustration

12. ValueKey President Mubarak, Egypt MubarakMubarak President Obama, Barak ObamaObama Facebook features, Facebook account Facebook Cache President Barak Obama meets Mubarak in London Added Document Basic Invalidation

13. ValueKey President Mubarak, Egypt MubarakMubarak President Obama, Barak ObamaObama Facebook features, Facebook account Facebook Cache Added Document Basic Invalidation President Barak Obama meets Mubarak in London

14. ValueKey President Mubarak, Egypt MubarakMubarak President Obama, Barak ObamaObama Facebook features, Facebook account Facebook Cache Added Document Basic Invalidation President Barak Obama meets Mubarak in London

15. ValueKey President Mubarak, Egypt MubarakMubarak President Obama, Barak ObamaObama Facebook features, Facebook account Facebook Cache Added Document Basic Invalidation President Barak Obama meets Mubarak in London

16. ValueKey President Mubarak, Egypt MubarakMubarak President Obama, Barak ObamaObama Facebook features, Facebook account Facebook Cache President Barak Obama meets Mubarak in London Added Document Basic Invalidation

17. ValueKey President Mubarak, Egypt MubarakMubarak President Obama, Barak ObamaObama Facebook features, Facebook account Facebook Cache Added Document Basic Invalidation President Barak Obama meets Mubarak in London

18. ValueKey President Mubarak, Egypt MubarakMubarak President Obama, Barak ObamaObama Facebook features, Facebook account Facebook Cache Added Document Basic Invalidation President Barak Obama meets Mubarak in London

19. ValueKey President Mubarak, Egypt MubarakMubarak President Obama, Barak ObamaObama Facebook features, Facebook account Facebook Cache Added Document Basic Invalidation President Barak Obama meets Mubarak in London

20. ValueKey President Mubarak, Egypt MubarakMubarak President Obama, Barak ObamaObama Facebook features, Facebook account Facebook Cache Added Document Basic Invalidation President Barak Obama meets Mubarak in London

21. ValueKey President Mubarak, Egypt MubarakMubarak President Obama, Barak ObamaObama Facebook features, Facebook account Facebook Cache Added Document Basic Invalidation President Barak Obama meets Mubarak in London

22. ValueKey President Mubarak, Egypt MubarakMubarak President Obama, Barak ObamaObama Facebook features, Facebook account Facebook Cache Added Document Basic Invalidation President Barak Obama meets Mubarak in London

23. ValueKey President Mubarak, Egypt MubarakMubarak President Obama, Barak ObamaObama Facebook features, Facebook account Facebook Cache Added Document Basic Invalidation President Barak Obama meets Mubarak in London

24. ValueKey President Mubarak, Egypt MubarakMubarak President Obama, Barak ObamaObama Facebook features, Facebook account Facebook Cache Added Document Basic Invalidation CIP Will help here ! President Barak Obama meets Mubarak in London

25. ValueKey President Mubarak, Egypt MubarakMubarak President Obama, Barak ObamaObama Facebook features, Facebook account Facebook Cache Added Document Basic Invalidation President Barak Obama meets Mubarak in London

26. ValueKey President Mubarak, Egypt MubarakMubarak President Obama, Barak ObamaObama Facebook features, Facebook account Facebook Cache Added Document Basic Invalidation President Barak Obama meets Mubarak in London

27. ValueKey President Mubarak, Egypt MubarakMubarak President Obama, Barak ObamaObama Facebook features, Facebook account Facebook Cache Added Document Basic Invalidation President Barak Obama meets Mubarak in London

28. ValueKey President Mubarak, Egypt MubarakMubarak President Obama, Barak ObamaObama Facebook features, Facebook account Facebook Cache Added Document Basic Invalidation President Barak Obama meets Mubarak in London

29. ValueKey President Mubarak, Egypt MubarakMubarak President Obama, Barak ObamaObama Facebook features, Facebook account Facebook Cache Added Document Basic Invalidation President Barak Obama meets Mubarak in London

30. ValueKey President Mubarak, Egypt MubarakMubarak President Obama, Barak ObamaObama Facebook features, Facebook account Facebook Cache President Barak Obama meets Mubarak in London Added Document Basic Invalidation

31. ValueKey President Mubarak, Egypt MubarakMubarak President Obama, Barak ObamaObama Facebook features, Facebook account Facebook Cache Added Document Basic Invalidation President Barak Obama meets Mubarak in London

32. ValueKey President Mubarak, Egypt MubarakMubarak President Obama, Barak ObamaObama Facebook features, Facebook account Facebook Cache Added Document Basic Invalidation President Barak Obama meets Mubarak in London

33. ValueKey President Mubarak, Egypt MubarakMubarak President Obama, Barak ObamaObama Facebook features, Facebook account Facebook Cache Added Document Basic Invalidation My work is done President Barak Obama meets Mubarak in London

34. ValueKey President Mubarak, Egypt MubarakMubarak President Obama, Barak ObamaObama Facebook features, Facebook account Facebook Cache Added Document Basic Invalidation President Barak Obama meets Mubarak in London

35. ValueKey President Mubarak, Egypt MubarakMubarak President Obama, Barak ObamaObama Facebook features, Facebook account Facebook Cache Added Document Basic Invalidation President Barak Obama meets Mubarak in London

36. ValueKey President Mubarak, Egypt MubarakMubarak President Obama, Barak ObamaObama Facebook features, Facebook account Facebook Cache Added Document Score Invalidation- K=1 President Barak Obama meets Mubarak in London

37. ValueKey President Mubarak, Egypt MubarakMubarak President Obama, Barak ObamaObama Facebook features, Facebook account Facebook Cache Added Document-d Score(q,d)Query 0.56President Obama 0.32President Mubarak 0.001Barak Obama Score Invalidation- K=1 President Barak Obama meets Mubarak in London

38. ValueKey President Mubarak, Egypt MubarakMubarak President Obama, Barak ObamaObama Facebook features, Facebook account Facebook Cache Added Document-d Score(q,d)Query 0.56President Obama 0.32President Mubarak 0.001Barak Obama Score Invalidation- K=1 President Barak Obama meets Mubarak in London

39. ValueKey President Mubarak, Egypt MubarakMubarak President Obama, Barak ObamaObama Facebook features, Facebook account Facebook Cache Added Document-d Score Invalidation- K=1 President Barak Obama meets Mubarak in London

40. Software Design – UML Diagrams Search Query, with miss in cache

41. Software Design – UML Diagrams Add a document to index with basic invalidation

42. Skills We acquired the following skills in this project: Knowledge: reading scientific publications Java (& Advanced Java topics) Working with Web-server.(apache) Learning Lucene features and how to use it. Building software Cache. UML XML parsing HTML