School of EECS, Peking University

School of EECS, Peking University
Web-based Information Architecture 02: The term vocabulary and postings list Hongfei Yan School of EECS, Peking University 3/4/2014 Phrase queries: PEKING UNIVERSITY Proximity: find GATES NEAR MICROSOFT We need an index that captures position information for phrase queries and proximity queries.

Contents 09 Relevance feedback & query expansion 01 Boolean retrieval
02 The term vocabulary & postings lists 03 Dictionaries and tolerant retrieval 04 Index construction 05 Index compression 06 Scoring, term weighting & the vector space model 07 Computing scores in a complete search system 08 Evaluation in information retrieval 09 Relevance feedback & query expansion 10 XML retrieval 11 Probabilistic information retrieval 12 Language models for information retrieval 13 Text classification & Naive Bayes 14 Vector space classification 15 Support vector machines & machine learning on documents 16 Flat clustering 17 Hierarchical clustering 18 Matrix decompositions & latent semantic indexing 19 Web search basics 20 Web crawling and indexes 21 Link analysis

02: The term vocabulary and postings list
2.1 Document delineation and character sequence decoding 2.2 Determining the vocabulary of terms 2.3 Faster postings list intersection via skip pointers 2.4 Positional postings and phrase queries Delineation 英 [dɪˌlɪnɪ'eɪʃn] n.描绘网络描述；圈定；轮廓；描写

Recap: Inverted Index dictionary postings
For each term t, we store a list of all documents that contain t. dictionary postings A posting A postings list The two parts of an inverted index. The dictionary is commonly kept in memory, with pointers to each postings list, which is stored on disk. The dictionary has been sorted by alphabetically and each postings list is sorted by document ID.

Recap: Intersecting two posting lists

Does Google use the Boolean model?
On Google, the default interpretation of a query [w1 w wn] is w1 AND w2 AND . . .AND wn Cases where you get hits that do not contain one of the wi : anchor text page contains variant of wi (morphology, spelling correction, synonym) long queries (n large) Boolean expression generates very few hits Simple Boolean vs. Ranking of result set Simple Boolean retrieval returns matching documents in no particular order. Google (and most well designed Boolean engines) rank the result set – they rank good hits (according to some estimator of relevance) higher than bad hits.

Documents Last lecture: Simple Boolean retrieval system
Our assumptions were: We know what a document is. We can “machine-read” each document. This can be complex in reality.

Parsing a document We need to deal with format and language of each document. What format is it in? pdf, word, excel, html etc. What language is it in? What character set is in use? Each of these is a classification problem, which we will study later in this course (IIR 13). Alternative: use heuristics

Format/Language: Complications
A single index usually contains terms of several languages. Sometimes a document or its components contain multiple languages/formats. French with Spanish pdf attachment What is the document unit for indexing? A file? An ? An with 5 attachments? A group of files (ppt or latex in HTML)? Upshot: Answering the question “what is a document?” is not trivial and requires some design decisions. Also: XML upshot D.J.[ˈʌpʃɔt] n. 结果

2.1 Document delineation and character sequence decoding 2.2 Determining the vocabulary of terms 2.2.1 Tokenization 2.2.2 Dropping common words: stop words 2.2.3 Normalization (equivalence classing of terms) 2.2.4 Stemming and lemmatization 2.3 Faster postings list intersection via skip pointers 2.4 Positional postings and phrase queries

Definitions Word – A delimited string of characters as it appears in the text. Term – A “normalized” word (case, morphology, spelling etc); an equivalence class of words. Token – An instance of a word or term occurring in a document. Type – The same as a term in most cases: an equivalence class of tokens.

Recall: Inverted index construction
Input: Output: Each token is a candidate for a postings entry. What are valid tokens to emit?

In June, the dog likes to chase the cat in the barn.
Exercises In June, the dog likes to chase the cat in the barn. How many word tokens? How many word types? Why tokenization is difficult– even in English. Tokenize: Mr. O’Neill thinks that the boys’ stories about Chile’s capital aren’t amusing. Chile n.英 ['tʃɪlɪ] 智利（南美洲西南部的一个国家,首都是圣地亚哥Santiago）

Tokenization problems: One word or two? (or several)
Hewlett-Packard State-of-the-art co-education the hold-him-back-and-drag-him-away maneuver data base San Francisco Los Angeles-based company cheap San Francisco-Los Angeles fares York University vs. New York University San Francisco D.J.[ˌsænfrənˈsiskəu] n. 旧金山(美国加利福尼亚西部港市) Los Angeles [lɔsˈændʒələs] n. 洛杉矶(美国城市) York [jɔ:k] n. =Yorkshire [英]【史】约克王朝( ) 约克(美国宾夕法尼亚州东南部一城市) maneuver 英 [mə‘nu:və] v.& n.机动，演习 n.策略，谋略，花招 vi.移动，用策略 vt.操纵网络手法；操作法；擦略；调遣 Splitting tokens on spaces can cause bad retrieval results, for example, if a search for York University mainly returns documents containing New York University. The problems of hyphens and non-separating whitespace can even interact. Advertisements for air fares frequently contain items like San Francisco-Los Angeles, where simply doing whitespace splitting would give unfortunate results.

Numbers 3/12/91 in the United States, 3/12/91 is Mar. 12,1991 Whereas in Europe it is 3 Dec. 1991 Mar 12, 1991 B-52 (800) Older IR systems may not index numbers . . . . . . but generally it’s a useful feature.

Chinese: No whitespace

Ambiguous segmentation in Chinese
The two characters can be treated as one word meaning ‘monk’ or as a sequence of two words meaning ‘and’ and ‘still’.

Other cases of “no whitespace”
Compounds in Dutch, German, Swedish Computerlinguistik → Computer + Linguistik Lebensversicherungsgesellschaftsangestellter→ leben + versicherung + gesellschaft + angestellter Inuit: tusaatsiarunnanngittualuujunga (I can’t hear very well.) Many other languages with segmentation difficulties: Finnish, Urdu, . . . Dutch D.J.[dʌtʃ] adj. 荷兰人的;荷兰的;荷兰语的 n. 荷兰语荷兰人 The British and Dutch belong to the same race. 英国人和荷兰人是同一种族。 Finnish [ˈfiniʃ] adj. 芬兰的, 芬兰人的芬兰语的词性变化 Finnish [ˈfiniʃ] n. 芬兰语 Urdu : n. An Indic language that is the official literary language of Pakistan, essentially identical to Hindi in its spoken form but in its literary form heavily influenced by Persian and Arabic and written in an Arabic alphabet.

Japanese 4 different “alphabets”: Chinese characters, hiragana syllabary for inflectional endings and functional words, katakana syllabary for transcription of foreign words and other uses, and latin. No spaces (as in Chinese). End user can express query entirely in hiragana! alphabet 字母表 syllabary :n. A list of syllables. A set of written characters for a language, each character representing a syllable. Syllable, 音节 Hiragana 英 [ˌhɪrə'ɡɑ:nə] n. <日>平假名（日文字母的草体） Katakana 英 [ˌkætə'kɑ:nə] n. <日>（日本字母）片假名 Latin 英 [ˈlætn] n.拉丁文；拉丁语，拉丁文学；拉丁美洲人；拉丁姆人 adj.拉丁的，拉丁语的；古罗马的；拉丁姆的；拉丁美洲的 inflectional morphology, 曲折形态学. Derivational morphology, 派生形态学 Chap 3 of the book “Speech and language processing” 3.1 ENGLISH MORPHOLOGY 英语形态学 There are many ways to combine morphemes to create words. Four of these methods are common and play important roles in speech and language processing: inflection, derivation, compounding, INFLECTION and cliticization. Inflection is the combination of a word stem with a grammatical morpheme, usually resulting in a word of the same class as the original stem, and usually filling some syntactic function like agreement. For example, English has the inflectional morpheme -s for marking the plural on nouns, and the inflectional morpheme -ed for marking the past tense on verbs. Derivation is the combination of a word stem with a grammatical morpheme, usually resulting in a word of a different class, often with a meaning hard to predict exactly. For example the verb computerize can take the derivational suffix -ation to produce the noun computerization. Compounding is the combination of multiple word stems together. For example the noun doghouse is the concatenation of the morpheme dog with the morpheme house. Finally, cliticization is the combination of CLITIC a word stem with a clitic. A clitic is a morpheme that acts syntactically like a word, but is reduced in form and attached (phonologically and sometimes orthographically) to another word. For example the English morpheme ’ve in the word I’ve is a clitic, as is the French definite article l’ in the word l’opera. In the following sections we give more details on these processes. 3.1.1 Inflectional

Arabic script It is not easy to learn Arabic. 学阿拉伯语不容易。
Arabic script reads from right to left. 阿拉伯文是从右至左读的。 Arabic, n. A Semitic language consisting of numerous dialects that is the principal language of Arabia, Jordan, Syria, Iraq, Lebanon, Egypt, and parts of northern Africa. Semitic 英 [sə‘mɪtɪk] adj. 闪米特人的；闪族语系的 n. 闪语；闪语族 dialect 英 [ˈdaɪəlekt] n.方言，土语；语调； [语]语支；专业用语网络方言；非标准语言；方言，土语，地方话；方言中

Arabic script: Bidirectionality
← → ← → ← START ‘Algeria achieved its independence in 1962 after 132 years of French occupation.’ Bidirectionality is not a problem if text is coded in Unicode. Algeria n. 阿尔及利亚

Stop words stop words = extremely common words which would appear to be of little value in helping select documents matching a user need Examples: a, an, and, are, as, at, be, by, for, from, has, he, in, is, it, its, of, on, that, the, to, was, were, will, with Stop word elimination used to be standard in older IR systems. But you need stop words for phrase queries, e.g. “King of Denmark” Most web search engines index stop words. Denmark 英 ['denmɑ:k] n. 丹麦(北欧国家,首都哥本哈根)

2.1 Document delineation and character sequence decoding 2.2 Determining the vocabulary of terms 2.2.1 Tokenization 2.2.2 Dropping common words: stop words 2.2.3 Normalization (equivalence classing of terms) 2.2.4 Stemming and lemmatization 2.3 Faster postings list intersection via skip pointers 2.4 Positional postings and phrase queries normalization 英 [ˌnɔ:məlaɪ'zeɪʃn] n.正常化；标准化；正态化 lemmatize 英 ['lemətaɪz] vt.把（文中的词）按屈折变化形式（或异体形式）进行归类

Normalization Need to “normalize” terms in indexed text as well as query terms into the same form. Example: We want to match U.S.A. and USA We most commonly implicitly define equivalence classes of terms. Alternatively: do asymmetric expansion (query expansion list) window → window, windows windows → Windows, windows Windows (no expansion) More powerful, but less efficient Why don’t you want to put window, Window, windows, and Windows in the same equivalence class? Asymmetric 英 [ˌeɪsɪˈmetrɪk] adj. 不对称的，不匀称的； <逻>非对称的

Normalization: Other languages
Normalization and language detection interact. PETER WILL NICHT MIT. → MIT = mit He got his PhD from MIT. → MIT ≠ mit

Accents and diacritics
Accents: résumé vs. resume (simple omission of accent) Umlauts: Universität vs. Universitaet (substitution with special letter sequence “ae”) Most important criterion: How are users likely to write their queries for these words? Even in languages that standardly have accents, users often do not type them. (Polish?) Diacritic adj. 可区别的,读音符号的 n. A mark, such as the cedilla of façade or the acute accent of resumé, added to a letter to indicate a special phonetic value or distinguish words that are otherwise graphically identical. cedilla [siˈdilə] n. 【语】下加符(法语中C字母下所附加的符号, 表示读作“S”音) Umlaut英 ['ʊmlaʊt] n. (日耳曼语系中的)元音变音 n. A change in a vowel sound caused by partial assimilation especially to a vowel or semivowel occurring in the following syllable Polish D.J.[ˈpəʊlɪʃ] adj. 波兰的,与波兰有关的 a brief outline of Polish history 波兰史纲波兰人的,波兰语的

Case folding Reduce all letters to lower case
Possible exceptions: capitalized words in mid-sentence MIT vs. mit Fed vs. fed It’s often best to lowercase everything since users will use lowercase regardless of correct capitalization.

More equivalence classing
Soundex: IIR 3 (phonetic equivalence, Muller = Mueller) Thesauri: IIR 9 (semantic equivalence, car = automobile)

Lemmatization Reduce inflectional/variant forms to base form
Example: am, are, is → be Example: car, cars, car’s, cars’ → car Example: the boy’s cars are different colors → the boy car be different color Lemmatization implies doing “proper” reduction to dictionary headword form (the lemma). Inflectional morphology (cutting → cut) vs. derivational morphology (destruction → destroy) derivational 英 [ˌderɪ'veɪʃənəl] adj.诱导的，得来的 morphology 英 [mɔ:ˈfɒlədʒi] n.形态学（尤指动植物形态学或词语形态学） Inflectional 英 [ɪn'flekʃənəl] adj.抑扬的，屈曲的，屈折的

Stemming Definition of stemming: Crude heuristic process that chops off the ends of words in the hope of achieving what “principled” lemmatization attempts to do with a lot of linguistic knowledge. Language dependent Often inflectional and derivational Example for derivational: automate, automatic, automation all reduce to automat automatic 英 [ˌɔ:tə'mætɪk] automate 英 ['ɔ:təmeɪt] Automat 英 [ˈɔ:təmæt] n.<美>自助餐馆；自助食堂；自动机网络自动机；自动装置；自动售货机；自动贩卖式餐馆

Porter Algorithm Most common algorithm for stemming English
Results suggest that it is at least as good as other stemming options Conventions + 5 phases of reductions Phases are applied sequentially Each phase consists of a set of commands. Sample command: Delete final ement if what remains is longer than 1 character replacement → replac cement → cement Sample convention: Of the rules in a compound command, select the one that applies to the longest suffix.

Porter stemmer: A few rules
Example SSES  SS caresses  caress IES  I ponies  poni SS -> SS caress  caress S  cats  cat

Three stemmers: A comparison
Sample text: Such an analysis can reveal features that are not easil visible from the variations in the individual genes and can lead to a picture of expression that is more biologically transparent and accessible to interpretation Porter stemmer: such an analysi can reveal featur that ar not easili visibl from the variat in the individu gene and can lead to pictur of express that is more biolog transpar and access to interpret Lovins stemmer: such an analys can reve featur that ar not eas vis from th vari in th individu gen and can lead to a pictur of expres that is mor biolog transpar and acces to interpres Paice stemmer: such an analys can rev feat that are not easy vis from the vary in the individ gen and can lead to a pict of express that is mor biolog transp and access to interpret

Does stemming improve effectiveness?
In general, stemming increases effectiveness for some queries, and decreases effectiveness for others. Queries where stemming is likely to help: [tartan sweaters], [sightseeing tour san francisco] (equivalence classes: {sweater,sweaters}, {tour,tours}) Porter Stemmer equivalence class oper contains all of operate operating operates operation operative operatives operational. Queries where stemming hurts: [operational AND research], [operating AND system], [operative AND dentistry] operative 英 [ˈɔpərətɪv, -əˌreɪtɪv, ˈɔprə-] adj.手术的；（计划、法律等）实施中的，起作用的；最适合的；关键词句 n.工作人员；（尤指）体力劳动者；密探；（尤指政府的）特工人员 operational research 运筹学 operative dentistry 牙科手术

Exercise: What does Google do?
Stop words Normalization Tokenization Lowercasing Stemming Non-latin alphabets Umlauts Compounds Numbers stop words：google索引中保留停用词，结果包含去停用词的查询结果和不去停用词的查询结果。一般情况下，查询中非停用词已经具有显著意思时，停用词会被忽略的查询结果靠前，例如apples and pears，而当查询中停用词与其他词构成特殊词组时，包含停用词的查询结果靠前，如face to face，to be or not to be。 normalization：google对部分明显的等价词项会进行归一化处理。例如就USA和U.S.A这样比较明显的等价词来说，google进行了归一化处理，搜索结果相同；而像car和automobile这样差异较大的同义词则没有进行归一化。 tokenization：google对中文这些无明显区分的语言也支持tokenization。就英语来说google对部分有歧义的词会进行切分，例如boy's会被切开当作boy或boys，而I'm不会切开。 lowercasting：google查询不区分大小写，例如ibm和IBM查询结果相同。 stemming：google查询结果不是简单使用stemming，例如social和society的查询结果不同。但normalization应该使用到了stemming技术。 non-latin alphabets: google支持non-latin alphabets，比如中文、日文等等。一般查询结果只会返回与查询使用的语言一致的结果 Umlauts：google会适当考虑音标，例如希腊语中αλλήλους的查询结果包含ἀλλήλους，而ἀλλήλους的查询结果靠前的不包含αλλήλους。 Compounds：google会对复合词切分，同时也保留复合词。例如hard-working查询结果包含hard working，hard-working，hardworking。 numbers：google支持数字查询，例如电话号码，日期等。日期会在一定范围内转换，例如中式的2013/3/12结果中包含，2013年3月12日，美式的12/3/2012结果中包含December 3rd, 2012。

2.1 Document delineation and character sequence decoding 2.2 Determining the vocabulary of terms 2.3 Faster postings list intersection via skip pointers 2.4 Positional postings and phrase queries

Recall basic intersection algorithm
Linear in the length of the postings lists. Can we do better?

Skip pointers Skip pointers allow us to skip postings that will not figure in the search results. This makes intersecting postings lists more efficient. Some postings lists contain several million entries so efficiency can be an issue even if basic intersection is linear. Where do we put skip pointers? How do we make sure intersection results are correct? figure 英 [ˈfiɡə] n.数字；算术；图解；轮廓 vi.计算；出现；扮演角色 vt.计算在内；估计；推测；认为

Basic idea We have matched 8 on each list and moved it to the results list. The smallest item is then the element 34 on the upper list and 17 on the lower list. The smallest item is then the element 17 on the lower list. Rather than simply advancing the upper pointer, We first check the skip list pointer and note that 31 is also less than 34.

Skip lists: larger example

Intersection with skip pointers

Where do we place skips? Tradeoff: number of items skipped vs. frequency skip can be taken More skips: Each skip pointer skips only a few items, but we can frequently use it. Fewer skips: Each skip pointer skips many items, but we can not use it very often.

Where do we place skips? (cont)
Simple heuristic: for postings list of length P, use evenly-spaced skip pointers. This ignores the distribution of query terms. Easy if the index is static; harder in a dynamic environment because of updates. How much do skip pointers help? They used to help a lot. With today’s fast CPUs, they don’t help that much anymore.

2.1 Document delineation and character sequence decoding 2.2 Determining the vocabulary of terms 2.3 Faster postings list intersection via skip pointers 2.4 Positional postings and phrase queries 2.4.1 Biword indexes 2.4.2 Positional indexes 2.4.3 Combination schemes

Phrase queries We want to answer a query such as [Stanford University] – as a phrase. Thus The inventor Stanford Ovshinsky never went to university. is not a match. The concept of phrase query has proven easily understood by users. About 10% of web queries are phrase queries. Consequence for inverted index: it no longer suffices to store docIDs in postings lists. Two ways of extending the inverted index: biword index positional index

Biword indexes Index every consecutive pair of terms in the text as a phrase. For example, Friends, Romans, Countrymen would generate two biwords: “friends romans” and “romans countrymen” Each of these biwords is now a vocabulary term. Two-word phrases can now easily be answered.

Longer phrase queries A long phrase like “stanford university palo alto” can be represented as the Boolean query “STANFORD UNIVERSITY” AND “UNIVERSITY PALO” AND “PALO ALTO” We need to do post-filtering of hits to identify subset that actually contains the 4-word phrase.

Extended biwords Parse each document and perform part-of-speech tagging Bucket the terms into (say) nouns (N) and articles/prepositions (X) Now deem any string of terms of the form NX*N to be an extended biword Examples: catcher in the rye N X X N king of Denmark N X N Include extended biwords in the term vocabulary Queries are processed accordingly Preposition 英 [ˌprepəˈzɪʃn] n.介词；前置词网络前置词；前置词，介词；方位词；介词，前置词

Issues with biword indexes
Why are biword indexes rarely used? False positives, as noted above Index blowup due to very large term vocabulary

Positional indexes Positional indexes are a more efficient alternative to biword indexes. Postings lists in a nonpositional index: each posting is just a docID Postings lists in a positional index: each posting is a docID and a list of positions

Positional indexes: Example
Query: “to1 be2 or3 not4 to5 be6” TO, : ‹ 1,6: ‹7, 18, 33, 72, 86, 231›; 2,5: ‹1, 17, 74, 222, 255›; 4,5: ‹8, 16, 190, 429, 433›; 5,2: ‹363, 367›; 7,3: ‹13, 23, 191›; › BE, : ‹ 1,2: ‹17, 25›; 4,5: ‹17, 191, 291, 430, 434›; 5,3: ‹14, 19, 101›; › Document 4 is a match!

Proximity search We just saw how to use a positional index for phrase searches. We can also use it for proximity search. For example: employment /4 place Employment agencies that place healthcare workers are seeing growth is a hit. Employment agencies that have learned to adapt now place healthcare workers is not a hit. Find all documents that contain EMPLOYMENT and PLACE within 4 words of each other.

Proximity search Use the positional index
Simplest algorithm: look at cross-product of positions of (i) EMPLOYMENT in document and (ii) PLACE in document Very inefficient for frequent words, especially stop words Note that we want to return the actual matching positions, not just a list of documents. This is important for dynamic summaries etc.

“Proximity” intersection

Combination scheme Biword indexes and positional indexes can be profitably combined. Many biwords are extremely frequent: Michael Jackson, Britney Spears etc For these biwords, increased speed compared to positional postings intersection is substantial. Combination scheme: Include frequent biwords as vocabulary terms in the index. Do all other phrases by positional intersection. Williams et al. (2004) evaluate a more sophisticated mixed indexing scheme. Faster than a positional index, at a cost of 26% more space for index.

“Positional” queries on Google
For web search engines, positional queries are much more expensive than regular Boolean queries. Let’s look at the example of phrase queries. Why are they more expensive than regular Boolean queries? Can you demonstrate on Google that phrase queries are more expensive than Boolean queries? The number of items to check is now bounded not by the number of documents but by the total number of tokens in the document collection T.

Summary Understanding of the basic unit of classical information retrieval systems: words and documents: What is a document, what is a term? Tokenization: how to get from raw text to words (or tokens) More complex indexes: skip pointers and phrases

School of EECS, Peking University

Similar presentations

Presentation on theme: "School of EECS, Peking University"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

School of EECS, Peking University

Similar presentations

Presentation on theme: "School of EECS, Peking University"— Presentation transcript:

Similar presentations

About project

Feedback