1. Building a Distributed Full-Text Index for the Web S. Melnik, S. Raghavan, B.Yang, H. Garcia-Molina

2. Introduction. Testbed architecture. Design of the indexer. Distributed indexing.

3. Introduction. Testbed architecture. Design of the indexer. Distributed indexing.

4. Pig Cat Fish Cat Fly Dog Pig Dog Cat Fish Dog 123 Inverted list Cat-> (1,2), (1,4), (3,2) Dog->(2,2), (3,1), (3,4) Fish->(1,3), (3,3) Pig->(1,1), (2,3) Inverted index location

5. Inverted index consist of an inverted lists for each sorted term. Inverted list consist of a locations in sorted way. Location consist of (page identifier, position in the page). Posting consist of (index term, location).

6. Building an inverted index over a collection of web pages involves: 1. Processing each page to extract postings. 2. Building for each term inverted list. 3. Writing out on disk.

7. Important problems when building web-scale inverted index: 1. Scale and growth rate. 2. Rate of change

8. Introduction. Testbed architecture. Design of the indexer. Distributed indexing.

10. Distributors. Indexers. Query servers.

11. Distributed inverted index organization: 1.Local inverted files. 2. Global inverted files.

12. Global inverted files Query server 1 Cat->(1,2), (1,4), (3,2) Dog->(2,2), (3,1), (3,4) a-e Query server 2 Fish->(1,3), (3,3) Pig->(1,1), (2,3) f-z Dog Cat Fish Dog Fly Dog Pig Cat Fish Cat 213

13. Local inverted files Query server 1 Cat->(1,2), (1,4) Dog->(2,2) Fish->(1,3) Fly->(2,1) Pig->(1,1), (2,3) a-e Query server 2 Cat->(3,2) Dog->(3,1), (3,4) Fish->(3,3) f-z Dog Cat Fish Dog Fly Dog Pig Cat Fish Cat 213

14. Local vs. Global Resilience to failures. Network load.

15. Testbed environment: The indexers and the query servers are single processor PC’s with 350-500 MHz processors, 300-500 MB of main memory, and equipped with multiple disks. All the machines are interconnected by a 100 Mbps Ethernet LAN network.

16. The WebBase collection: To study some properties of web pages that are relevant to text indexing, we analyzed 5 samples, of 100,000 pages each, from different portions of the WebBase repository.

17. valueProperty 438Average number of words per page 171Average number of distinct words per page 8650Average size of each page (as HTML) 2815Average size of each page after removing HTML tags 8Average size of a word in the vocabulary Table 1: Properties of the WebBase collection

19. Introduction. Testbed architecture. Design of the indexer. Distributed indexing.

21. Design of the Indexer Software pipeline. The storage of the inverted files generated by the process.

22. Software pipeline The process can logically be split into 3 phases: Processing -> CPU intensive. Flushing -> disk. loading -> network.

24. The goal of our pipelining technique is to design an execution schedule for the different indexing phases that will result in minimal overall running time. Examples: L P F Execution of the pipeline

26. t Pipeline time

27. Theoretical analysis vs. experimental results

30. Design of the Indexer Software pipeline. The storage of the inverted files generated by the process.

31. Storage schemes: We consider ed three storage schemes for storing inverted files as sets of (key, value) pairs in a B-tree: 1. Full list. 2. Single payload. 3. Mixed list.

33. A qualitative comparison of these storage schemes: Index size Zig-zag joins Hot updates

34. Zig-zag join using ordered indexes 12347918 1 7911171219

35. Experimental results (using mixed list)

36. Index size (%age) Index size (GB) Input size (GB) Number of pages(million) 6.170.050.810.1 6.700.274.030.5 7.011.1316.112.0 6.902.7840.285.0 Table 5:Mixed-list scheme index sizes Only one posting was generated for all the occurrences of a word in a page

38. Introduction. Testbed architecture. Design of the indexer. Distributed indexing.

39. Two problems that must be addressed when building an inverted index on a distributed architecture: Page distribution: The question of when and how to distribute pages to the indexing nodes. Collecting global statistics: the question of where, when, and how to compute and distribute global statistics.

40. Two strategies for page distribution: A priori distribution. Runtime distribution.

41. Three advantages of runtime distribution: Space. Load balancing. Effective pipelining.

42. Collecting global statistics A dedicated server known as the statistician. Parallel computation. Minimize the number of conversations among servers. Avoid extra disk I/O Reduces network overhead.

43. Two strategies for sending information to the statistician: ME Strategy: sending local information during merging. FL Strategy: sending local information during flushing.

46. comparison