1. Introduction to Information Retrieval Introduction to Information Retrieval CS276 Information Retrieval and Web Search Christopher Manning and Prabhakar Raghavan Lecture: Index construction

2. Introduction to Information Retrieval Index construction  How do we construct an index?  What strategies can we use with limited main memory?  What about dynamic text collections? Ch. 4

3. Introduction to Information Retrieval Recall hardware basics  Disk I/O is block-based: Reading and writing of entire blocks (as opposed to smaller chunks).  Block sizes: 8KB to 256 KB.  Servers used in IR systems now typically have several GB of main memory, sometimes tens of GB.  Available disk space is several (2–3) orders of magnitude larger.  Fault tolerance is very expensive: It’s much cheaper to use many regular machines rather than one fault tolerant machine. Sec. 4.1

4. Introduction to Information Retrieval

5.  Documents are parsed to extract words and these are saved with the Document ID. I did enact Julius Caesar I was killed i' the Capitol; Brutus killed me. Doc 1 So let it be with Caesar. The noble Brutus hath told you Caesar was ambitious Doc 2 Recall: index construction Sec. 4.2

6. Introduction to Information Retrieval Key step  After all documents have been parsed, the inverted file is sorted by terms. We focus on this sort step. Sec. 4.2

7. Introduction to Information Retrieval Scaling index construction  In-memory index construction does not scale.  How can we construct an index for very large collections?  Taking into account...memory, disk, speed, etc. Sec. 4.2

8. Introduction to Information Retrieval Sort-based index construction  As we build the index, we parse docs one at a time.  While building the index, we cannot easily exploit compression tricks (you can, but much more complex)  The final postings are incomplete until the end.  At 12 bytes per non-positional postings entry (term, doc, freq), demands a lot of space for large collections.  T = 100,000,000 in the case of Reuters  So … we can do this in memory in 2009, but typical collections are much larger. E.g. the New York Times provides an index of >150 years of newswire  We need to store intermediate results on disk. Sec. 4.2

9. Introduction to Information Retrieval Use the same algorithm for disk?  multi-way merge-sort aka BSBI: Blocked sort-based Indexing Sec. 4.2

10. Introduction to Information Retrieval Multi-way Mergesort  Take Wikipedia in Italian, and compute word freq:  Few GBs  n  10 9 words  How do you proceed ??  Tokenize into a sequence of strings  Sort the strings  Create tuples

11. Introduction to Information Retrieval Binary Merge-Sort Merge-Sort(A,i,j) 01 if (i < j) then 02 m = (i+j)/2; 03 Merge-Sort(A,i,m); 04 Merge-Sort(A,m+1,j); 05 Merge(A,i,m,j) Merge-Sort(A,i,j) 01 if (i < j) then 02 m = (i+j)/2; 03 Merge-Sort(A,i,m); 04 Merge-Sort(A,m+1,j); 05 Merge(A,i,m,j) Divide Conquer Combine 1 2 8 10 7 9 13 19 127 Merge is linear in the #items to be merged

12. Introduction to Information Retrieval Few key observations  Items = (short) strings = atomic...   (n log n) memory accesses (I/Os ??)  [5ms] * n log 2 n ≈ 3 years In practice it is a “faster”, why?

13. Introduction to Information Retrieval Implicit Caching… 102 2 10 51 1 5 1319 13 19 97 7 9 154 4 15 83 3 8 1217 12 17 611 6 11 1 2 5 107 9 13 193 4 8 156 11 12 17 1 2 5 7 9 10 13 193 4 6 8 11 12 15 17 1 2 3 4 5 6 7 8 9 10 11 12 13 15 17 19 log 2 N M N/M runs, each sorted in internal memory (no I/Os) 2 passes (one Read/one Write) = 2 * (N/B) I/Os — I/O-cost for binary merge-sort is ≈ 2 (N/B) log 2 (N/M) Log 2 (N/M) 2 passes (R/W)

14. Introduction to Information Retrieval B A key inefficiency 1 2 4 7 9 10 13 193 5 6 8 11 12 15 17 B After few steps, every run is longer than B !!! B We are using only 3 pages But memory contains M/B pages ≈ 2 30 /2 15 = 2 15 B Output Buffer Disk 1, 2, 3 Output Run 4,...

15. Introduction to Information Retrieval Multi-way Merge-Sort  Sort N items with main-memory M and disk-pages B:  Pass 1: Produce (N/M) sorted runs.  Pass i: merge X = M/B-1 runs  log X N/M passes Main memory buffers of B items Pg for run1 Pg for run X Out Pg Disk Pg for run 2...

16. Introduction to Information Retrieval Cost of Multi-way Merge-Sort  Number of passes = log X N/M  log M/B (N/M)  Total I/O-cost is  ( (N/B) log M/B N/M ) I/Os Large fan-out (M/B) decreases #passes In practice M/B ≈ 10 5  #passes = 1  few mins Tuning depends on disk features Compression would decrease the cost of a pass!

17. Introduction to Information Retrieval Keep attention...  If sorting needs to manage arbitrarily long strings Key observations:  Array A is an “array of pointers to objects”  For each object-to-object comparison A[i] vs A[j]:  2 random accesses to 2 memory locations A[i] and A[j]   (n log n) random memory accesses (I/Os ??) Memory containing the strings A Again chaching helps, But it may be less effective than before Indirect sort

18. Introduction to Information Retrieval SPIMI: Single-pass in-memory indexing  Key idea #1: Generate separate dictionaries for each block.  Key idea #2: Don’t sort. Accumulate postings in postings lists as they occur.  With these two ideas we can generate a complete inverted index for each block.  These separate indexes can then be merged into one big index.  Easy append with 1 file per posting (docID are increasing among blocks), otherwise emulate multi-way merging Sec. 4.3

19. Introduction to Information Retrieval SPIMI-Invert Merging of blocks  BSBI (now dictionary-driven). Sec. 4.3

20. Introduction to Information Retrieval SPIMI: Compression  Compression makes SPIMI even more efficient.  Compression of terms  Compression of postings  See next lectures Sec. 4.3

21. Introduction to Information Retrieval Distributed indexing  For web-scale indexing: must use a distributed computing cluster  Individual machines are fault-prone  Can unpredictably slow down or fail  How do we exploit such a pool of machines? Sec. 4.4

22. Introduction to Information Retrieval Google data centers  Google data centers mainly contain commodity machines.  Data centers are distributed around the world.  Estimate: Google installs 100,000 servers each quarter.  Based on expenditures of 200–250 million dollars per year  This is 10% of the computing capacity of the world !?! Sec. 4.4

23. Introduction to Information Retrieval Distributed indexing  Maintain a master machine directing the indexing job – considered “safe”.  Break up indexing into sets of (parallel) tasks.  Master machine assigns tasks to idle machines Sec. 4.4

24. Introduction to Information Retrieval Parallel tasks  We will use two sets of parallel tasks  Parsers  Inverters  Break the input document collection into splits  Each split is a subset of documents Sec. 4.4 Term-based partition one machine handles a subrange of terms Doc-based partition one machine handles a subrange of documents Term-based partition one machine handles a subrange of terms Doc-based partition one machine handles a subrange of documents

25. Introduction to Information Retrieval Data flow: doc-based partitioning splits Parser Master Inverter Postings IL 1 assign Sec. 4.4 IL 2 IL k Set 1 Set 2 Set k Query must be also distributed

26. Introduction to Information Retrieval Data flow: term-based partitioning splits Parser Master a-fg-pq-z a-fg-pq-z a-fg-pq-z Inverter Postings a-f g-p q-z assign Sec. 4.4 Set 1 Set 2 Set k Each query-term goes to one machine

27. Introduction to Information Retrieval Parsers  Master assigns a split to an idle parser machine  Parser reads a document at a time and emits (term, doc) pairs  Parser writes pairs into j partitions  Each partition is for a range of terms’ first letters (e.g., a-f, g-p, q-z) – here j = 3. Sec. 4.4

28. Introduction to Information Retrieval Inverters  An inverter collects all (term,doc) pairs (= postings) for one term-partition.  Sorts and writes to postings lists Sec. 4.4

29. Introduction to Information Retrieval MapReduce  Index construction is an instance of MapReduce  a robust and conceptually simple framework for distributed computing  … without having to write code for the distribution part.  Google indexing system (ca. 2002) consists of a number of phases, each implemented in MapReduce. Sec. 4.4

30. Introduction to Information Retrieval Data flow: term-based partitioning splits Parser Master a-fg-pq-z a-fg-pq-z a-fg-pq-z Inverter Postings a-f g-p q-z assign Map phase Reduce phase Sec. 4.4

31. Introduction to Information Retrieval Dynamic indexing  Up to now, we have assumed static collections.  It was rare but now more frequently occurs that:  Documents come in over time and need to be inserted.  Documents are deleted and modified.  And this induces:  Postings updates for terms already in dictionary  New terms added/deleted to/from dictionary Sec. 4.5

32. Introduction to Information Retrieval Simplest approach  Maintain “big” main index  New docs go into “small” auxiliary index  Search across both, and merge the results  Deletions  Invalidation bit-vector for deleted docs  Filter search results (i.e. docs) by the invalidation bit-vector  Periodically, re-index into one main index Sec. 4.5

33. Introduction to Information Retrieval Issues with main and auxiliary indexes  Merges may touch a lot of stuff  Poor performance  Merging of the auxiliary index into the main index is efficient if we keep a separate file for each postings list [docIDs are greater].  Merge is the same as a simple append.  But this needs a lot of files – inefficient for O/S.  Assumption: The index is one big file.  In reality: Use a scheme somewhere in between (e.g., split very large postings lists, collect postings lists of length 1 in one file etc.) Sec. 4.5

34. Introduction to Information Retrieval Logarithmic merge  Maintain a series of indexes, each twice as large as the previous one.  Keep smallest (Z 0 ) in memory  Larger ones (I 0, I 1, …) on disk  If Z 0 gets too big (> n), write to disk as I 0 or merge with I 0 (if I 0 already exists) as Z 1  Either write merge Z 1 to disk as I 1 (if no I 1 ) or merge with I 1 to form Z 2  etc. Sec. 4.5

35. Introduction to Information Retrieval Logarithmic merge  Auxiliary and main index: index construction time is O(T 2 ) as each posting is touched in each merge.  Logarithmic merge: Each posting is merged O(log T) times, so complexity is O(T log T) So logarithmic merge is much more efficient for index construction, but its query processing requires the merging of O(log T) indexes Sec. 4.5

36. Introduction to Information Retrieval Dynamic indexing at search engines  Most search engines now support dynamic indexing  News items, blogs, new topical web pages  But (sometimes/typically) they also periodically reconstruct the index from scratch  Query processing is then switched to the new index, and the old index is then deleted Sec. 4.5

37. Introduction to Information Retrieval Other sorts of indexes  Positional indexes  Same type of sorting problem … just larger  Character n-gram indexes:  For each n-gram, need pointers to all dictionary terms containing it – the “postings”.  Permuterm index:  Consider all cyclic rotations of T$, where T is any term Sec. 4.5