1. | 1 Gertjan van Noord2014 Zoekmachines Lecture 2: vocabulary, posting lists

2. Agenda for today Questions Chapter 1 Chapter 2: Term vocabulary & posting lists Chapter 2: Posting lists with positions Homework/lab assignment

3. Questions chapter 1

4. Chapter 2 Overview Preprocessing of documents choose the unit of indexing (granularity) tokenization (removing punctuation, splitting in words) stop list? normalization: case folding, stemming versus lemmatizing,... extensions to postings lists

5. Tokens, types and terms tokeneach separate word in the text typesame words belong to one type (index) termfinally included in the index index term is an equivalence class of tokens and/or types

6. Tokens, types and terms The Lord of the Rings Number of tokens? 5 Number of types? 4 Number of terms? 4? 2? 1? 26-01-12

7. Equivalence classes Casefolding Diacritics Stemming/lemmatisation Decompounding Synonym lists Variant spellings 26-01-12

8. Equivalence classes Implicit: mapping rules Relational: query expansion Relational: double indexing Mapping should be done: – Indexing – Querying 26-01-12

9. Diacritics 26-01-12

10. Words and word forms Inflection (D: verbuiging/vervoeging) -changing a word to express person, case, aspect,... -for determiners, nouns, pronouns, adjectives: declination (D: verbuiging) -for verbs: conjugation (D: vervoeging) Derivation (D: afleiding) -formation of a new word from another word (e.g. by adding an affix (prefix or suffix) or changing the grammatical category)

11. Inflection examples Determiners E: the D: de, het G: der, des, dem, den, die, das Adjectives E: young D: jonge, jonge G: junger, junge, junges, jungen Nouns E: man, men D: man, mannen G: mann, mannes, Verbs E write / writes / wrote / written D schrijf/ schrijft /schrijven / schreef/ schreven / geschreven G schreibe/ schreibst / schreibt / schreiben / schrieben / geschrieben

12. Derivation examples to browse -> a browser red -> to redden, reddish Google -> to google arm(s) -> to arm, to disarm -> disarmament, disarming

13. Stemming and lemmatizing verb formsinform, informs, informed, informing derivationsinformation, informative, informal?? steminform lemma inform, information, informative, informal verb formssing: sings, sang, sung, singing derivations singer, singers, song, songs stemsing, sang, sung, song, lemmasing, singer, song

14. Discussion Why is stemming used when lemmatizing is much more precise? Lemmatizing is a more complex process it needs - a vocabulary (problem: new words) - morphologic analysis (knowledge of inflection rules) - syntactic analysis, parsing (noun or verb?)

15. Compound splitting Marketingjargon -> marketing AND jargon Increased retrieval Decreased precision Must be applied to both query and index! But what to do with the query marketing jargon ? And with spreekwoord appel boom ? 26-01-12

16. Chapter 2 Overview Preprocessing of documents choose the unit of indexing (granularity) tokenization (removing punctuation, splitting in words) stop list? normalization: case folding, stemming versus lemmatizing,... extensions to postings lists

17. Efficient merging of postings For X AND Y, we have to intersect 2 lists Most documents will contain only one of the two terms

18. Recall basic intersection algorithm

19. Skip pointers

20. Makes intersection of 2 lists more efficient think of millions of list items How many skip pointers and where? Trade-off: More pointers, often useful but small skips. Less pointers … Heuristic: distance √n, evenly distributed

21. Skip pointers: useful? Yes, certainly in the past With very fast CPUs less important Especially in a rather static index If a list keeps changing less effective

22. Extensions of the simple term index To support phrase queries “information retrieval” “retrieval of information” Different approaches biword indexes phrase indexes positional indexes combinations

23. Biword and phrase indexes Holding terms together in the index Simple biword index: retrieval of, of information Sophisticated: POS tagger selects nouns N x* N retrieval of this information Phrase index: includes variable lengths of word sequences terms of 1 and 2 words both included

24. Positional index Add in the postings lists for each doc the list of positions of the term for phrase queries for proximity search Example [information, 4] : [1:, 2:, …] [retrieval, 2] : [1:, 2: ]

25. Combination schemes Often queried combinations: phrase index names of persons and organization esp. combinations of common terms (!) find out from query log For other phrases a positional index Williams e.a.: next word index added

26. H.E. Williams, J.Zobel, and D.Bahle (2004) Fast Phrase Querying With Combined Indexes (ACM Dig Library): Phrase querying with a combination of three approaches (next word index, phrase index and inverted file)... is more than 60% faster on average than using an inverted index alone... requires structures that total only 20% of the size of the collection.

27. A nextword index (Williams e.a.) docfreq,(,<doc, freq, [..] No of matching docs Doc ID No of occurrences in doc position