1. Information Retrieval Document Parsing

2. Basic indexing pipeline Tokenizer Token stream. Friends RomansCountrymen Linguistic modules Modified tokens. friend romancountryman Indexer Inverted index. Documents to be indexed. Friends, Romans, countrymen.

3. Parsing a document What format is it in? pdf/word/excel/html? What language is it in? What character set is in use? Plain ASCII, UTF-8, UTF-16,… Each of these is a classification problem, with many complications…

4. Tokenization: Issues Chinese/Japanese no spaces between words: Not always guaranteed a unique tokenization Dates/amounts in multiple formats フォーチュン 500 社は情報不足のため時間あた $500K( 約 6,000 万円 ) KatakanaHiraganaKanji“Romaji” What about DNA sequences ? ACCCGGTACGCAC... Definition of Tokens  What you can search !!

5. Case folding Reduce all letters to lower case exception: upper case (in mid-sentence?) e.g., General Motors USA vs. usa Morgen will ich in MIT … Is this the German “mit”?

6. Stemming Reduce terms to their “roots” language dependent e.g., automate(s), automatic, automation all reduced to automat. e.g., casa, casalinga, casata, casamatta, casolare, casamento, casale, rincasare, case reduced to cas

7. Porter’s algorithm Commonest algorithm for stemming English Conventions + 5 phases of reductions phases applied sequentially each phase consists of a set of commands sample convention: Of the rules in a compound command, select the one that applies to the longest suffix. Full morphologial analysis  modest benefit !! sses  ss, ies  i, ational  ate, tional  tion

8. Thesauri Handle synonyms and homonyms Hand-constructed equivalence classes e.g., car = automobile e.g., macchina = automobile = spider List of words important for a given domain For each word it specifies a list of correlated words (usually, synonyms, polysemic or phrases for complex concepts ). Co-occurrence Pattern: BT (broader term), NT (narrower term) Vehicle (BT)  Car  Fiat 500 (NT) How to use it in SE ??

10. Dmoz Directory

11. Yahoo! Directory

12. Information Retrieval Statistical Properties of Documents

13. Statistical properties of texts Token are not distributed uniformly They follow the so called “Zipf Law” Few tokens are very frequent A middle sized set has medium frequency Many are rare The first 100 tokens sum up to 50% of the text Many of these tokens are stopwords

14. K-th most frequent term has frequency approximately 1/k; or the product of the frequency (f) of a token and its rank (r) is almost a constant The Zipf Law, in detail f = c |T| / r   r * f = c |T| f = c |T| / r General Law Sum after the k-th element is ≤ f k k/(z-1) For the initial top-elements is a constant

15. An example of “Zipf curve”

16. Zipf’s law log-log plot

17. Consequences of Zipf Law Do exist many not frequent tokens that do not discriminate. These are the so called “stop words” English: to, from, on, and, the,... Italian: a, per, il, in, un,… Do exist many tokens that occur once in a text and thus are poor to discriminate (error?). English: Calpurnia Italian: Precipitevolissimevolmente (o, paklo) Words with medium frequency Words that discriminate

18. Other statistical properties of texts The number of distinct tokens grows as The so called “Heaps Law” (|T|  where  ) Hence the token length is  (log |T|) Interesting words are the ones with Medium frequency (Luhn)

19. Frequency vs. Term significance (Luhn)