1. CS276A Text Information Retrieval, Mining, and Exploitation Lecture 3 8 Oct 2002

2. Recap of last time Dictionary storage by blocking Wild card queries Query optimization and processing Phrase search Precision and Recall

3. Blocking Store pointers to every kth on term string. Need to store term lengths (1 extra byte) ….7systile9syzygetic8syzygial6syzygy11szaibelyite8szczecin9szomo…. Save 9 bytes  on 3  pointers. Lose 4 bytes on term lengths.

4. Wild-card queries mon*: find all docs containing any word beginning “mon”. Solution: index all k-grams occurring in any doc (any sequence of k chars). e.g., from text “April is the cruelest month” we get the 2-grams (bigrams) $ is a special word boundary symbol $a,ap,pr,ri,il,l$,$i,is,s$,$t,th,he,e$,$c,cr,ru,ue,el,le,es,st,t$, $m,mo,on,nt,h$

5. Skip pointers At query time: As we walk the current candidate list, concurrently walk inverted file entry - can skip ahead (e.g., 8,21). Skip size: recommend about  (list length) 2,4,6,8,10,12,14,16,18,20,22,24,...

6. Positional indexes Search for “to be or not to be” No longer suffices to store only entries Store, for each term, entries <number of docs containing term; doc1: position1, position2 … ; doc2: position1, position2 … ; etc.>

7. Today’s topics Index size Index construction time and strategies Dynamic indices - updating Additional indexing challenges from the real world

8. Index size In Lecture 1 we studied the space for postings also considered stemming, case folding, stop words... Lecture 2 covered the dictionary What is the effect of stemming, etc. on index size?

9. Index size Stemming/case folding cut number of terms by ~40% number of pointers by 10-20% total space by ~30% Stop words Rule of 30: ~30 words account for ~30% of all term occurrences in written text Eliminating 150 commonest terms from indexing will cut almost 25% of space

10. Positional index size Need an entry for each occurrence, not just once per document Index size depends on average document size Average web page has <1000 terms SEC filings, books, even some epic poems … easily 100,000 terms Consider a term with frequency 0.1% Why? 1001100,000 111000 Positional postings Postings Document size

11. Rules of thumb Positional index size factor of 2-4 over non- positional index Positional index size 35-50% of volume of original text Caveat: all of this holds for “English-like” languages

12. Index construction Thus far, considered index space What about index construction time? What strategies can we use with limited main memory?

13. Somewhat bigger corpus Number of docs = n = 40M Number of terms = m = 1M Use Zipf to estimate number of postings entries: n + n/2 + n/3 + …. + n/m ~ n ln m = 560M entries No positional info yet Check for yourself

14. Documents are parsed to extract words and these are saved with the Document ID. I did enact Julius Caesar I was killed i' the Capitol; Brutus killed me. Doc 1 So let it be with Caesar. The noble Brutus hath told you Caesar was ambitious Doc 2 Recall index construction

15. Key step After all documents have been parsed the inverted file is sorted by terms

16. Index construction As we build up the index, cannot exploit compression tricks parse docs one at a time, final postings entry for any term incomplete until the end (actually you can exploit compression, but this becomes a lot more complex) At 10-12 bytes per postings entry, demands several temporary gigabytes

17. System parameters for design Disk seek ~ 1 millisecond Block transfer from disk ~ 1 microsecond per byte (following a seek) All other ops ~ 10 microseconds

18. Bottleneck Parse and build postings entries one doc at a time To now turn this into a term-wise view, must sort postings entries by term (then by doc within each term) Doing this with random disk seeks would be too slow If every comparison took 1 disk seek, and n items could be sorted with nlog 2 n comparisons, how long would this take?

19. Sorting with fewer disk seeks 12-byte (4+4+4) records (term, doc, freq). These are generated as we parse docs. Must now sort 560M such 12-byte records by term. Define a Block = 10M such records can “easily” fit a couple into memory. Will sort within blocks first, then merge the blocks into one long sorted order.

20. Sorting 56 blocks of 10M records First, read each block and sort within: Quicksort takes about 2 x (10M ln 10M) steps Exercise: estimate total time to read each block from disk and and quicksort it. Exercise: estimate total time to read each block from disk and and quicksort it. 56 times this estimate - gives us 56 sorted runs of 10M records each. Need 2 copies of data on disk, throughout.

21. Merging 56 sorted runs Merge tree of log 2 56 ~ 6 layers. During each layer, read into memory runs in blocks of 10M, merge, write back. Disk 1 34 2 42 13

22. Merging 56 runs Time estimate for disk transfer: 6 x 56 x (120M x 10 -6 ) x 2 ~ 22 hours. disk block transfer time Work out how these transfers are staged, and the total time for merging.

23. Exercise - fill in this table TimeStep 56 initial quicksorts of 10M records each Read 2 sorted blocks for merging, write back Merge 2 sorted blocks 1 2 3 4 5 Add (2) + (3) = time to read/merge/write 56 times (4) = total merge time ?

24. Large memory indexing Suppose instead that we had 16GB of memory for the above indexing task. Exercise: how much time to index? Repeat with a couple of values of n, m. In practice, spidering interlaced with indexing. Spidering bottlenecked by WAN speed and many other factors - more on this later.

25. Improving on merge tree Compressed temporary files compress terms in temporary dictionary runs Merge more than 2 runs at a time maintain heap of candidates from each run …. 1 52436

26. Dynamic indexing Docs come in over time postings updates for terms already in dictionary new terms added to dictionary Docs get deleted

27. Simplest approach Maintain “big” main index New docs go into “small” auxiliary index Search across both, merge results Deletions Invalidation bit-vector for deleted docs Filter docs output on a search result by this invalidation bit-vector Periodically, re-index into one main index

28. More complex approach Fully dynamic updates Only one index at all times No big and small indices Active management of a pool of space

29. Fully dynamic updates Inserting a (variable-length) record e.g., a typical postings entry Maintain a pool of (say) 64KB chunks Chunk header maintains metadata on records in chunk, and its free space Record Header Free space

30. Global tracking In memory, maintain a global record address table that says, for each record, the chunk it’s in. Define one chunk to be current. Insertion if current chunk has enough free space extend record and update metadata. else look in other chunks for enough space. else open new chunk.

31. Changes to dictionary New terms appear over time cannot use a static perfect hash for dictionary OK to use term character string w/pointers from postings as in lecture 2.

32. Index on disk vs. memory Most retrieval systems keep the dictionary in memory and the postings on disk Web search engines frequently keep both in memory massive memory requirement feasible for large web service installations, less so for standard usage where query loads are lighter users willing to wait 2 seconds for a response More on this when discussing deployment models

33. Distributed indexing Suppose we had several machines available to do the indexing how do we exploit the parallelism Two basic approaches stripe by dictionary as index is built up stripe by documents

34. Indexing in the real world Typically, don’t have all documents sitting on a local filesystem Documents need to be spidered Could be dispersed over a WAN with varying connectivity Must schedule distributed spiders/indexers Could be (secure content) in Databases Content management applications Email applications http often not the most efficient way of fetching these documents - native API fetching

35. Indexing in the real world Documents in a variety of formats word processing formats (e.g., MS Word) spreadsheets presentations publishing formats (e.g., pdf) Generally handled using format-specific “filters” convert format into text + meta-data Documents in a variety of languages automatically detect language(s) in a document tokenization, stemming, are language- dependent

36. “Rich” documents (How) Do we index images? Researchers have devised Query Based on Image Content (QBIC) systems “show me a picture similar to this orange circle” watch for next week’s lecture on vector space retrieval In practice, image search based on meta- data such as file name e.g., monalisa.jpg

37. Passage/sentence retrieval Suppose we want to retrieve not an entire document matching a query, but only a passage/sentence - say, in a very long document Can index passages/sentences as mini- documents But then you lose the overall relevance assessment from the complete document - more on this as we study relevance ranking More on this when discussing XML search

38. Digression: food for thought What if a doc consisted of components Each component has its own access control list. Your search should get a doc only if your query meets one of its components that you have access to. More generally: doc assembled from computations on components e.g., in Lotus databases or in content management systems Welcome to the real world … more later.

39. Resources, and beyond MG 5. Comprehensive paper on parallel spidering: http://www2002.org/CDROM/refereed/108/index.ht ml Thus far, documents either match a query or do not. It’s time to become more discriminating - how well does a document match a query? Gives rise to ranking and scoring Vector space models Relevance ranking