INFORMATION RETRIEVAL TECHNIQUES BY DR. ADNAN ABID Lecture # 19 Permuterm index k-gram

ACKNOWLEDGEMENTS The presentation of this lecture has been taken from the underline sources “Introduction to information retrieval” by Prabhakar Raghavan, Christopher D. Manning, and Hinrich Schütze “Managing gigabytes” by Ian H. Witten, ‎Alistair Moffat, ‎Timothy C. Bell “Modern information retrieval” by Baeza-Yates Ricardo, ‎  “Web Information Retrieval” by Stefano Ceri, ‎Alessandro Bozzon, ‎Marco Brambilla

Outline Permuterm index k-gram Soundex

Permuterm index For term hello, index under: Queries: hello$, ello$h, llo$he, lo$hel, o$hell where $ is a special symbol. Queries: X lookup on X$ X* lookup on $X* *X lookup on X$* *X* lookup on X* X*Y lookup on Y$X* X*Y*Z ??? Exercise! a*e*i*o*u  a * u  u$a*  look up B tree X, X*, *X are all simple and can be handled without permuterm. X*Y and X*Y*Z can be handled with permuterm. 00:01:45  00:03:32(For term) 00:04:50  00:05:10 00:06:35  00:08:15 00:12:07  00:13:10 00:16:35  00:17:00(a*e*) 00:19:00  00:20:00(Permuterm Index) Permuterm index increases the size of dictionary 4 times FOR ENGLINSH Depends on average length of a word Query = hel*o X=hel, Y=o Lookup o$hel*

Bigram (k-gram) indexes Enumerate all k-grams (sequence of k chars) occurring in any term e.g., from text “April is the cruelest month” we get the 2-grams (bigrams) $ is a special word boundary symbol Maintain a second inverted index from bigrams to dictionary terms that match each bigram. $a,ap,pr,ri,il,l$,$i,is,s$,$t,th,he,e$,$c,cr,ru, ue,el,le,es,st,t$, $m,mo,on,nt,h$ 00:21:35  00:22:30 00:22:55  00:24:20

Bigram index example The k-gram index finds terms based on a query consisting of k-grams (here k=2). $m mace madden mo among amortize on along among 00:25:10  00:26:15 00:26:35  00:27:25

Processing wild-cards Query mon* can now be run as $m AND mo AND on Gets terms that match AND version of our wildcard query. But we’d enumerate moon. Must post-filter these terms against query. Surviving enumerated terms are then looked up in the term- document inverted index. Fast, space efficient (compared to permuterm). 00:28:25  00:30:00

K-gram Index Larger k-grams would be lesser flexibility in query processing. Normally bi and tri k-gram proffered. 00:41:40  00:42:00

00:42:55 00:43:05

Processing wild-card queries As before, we must execute a Boolean query for each enumerated, filtered term. Wild-cards can result in expensive query execution (very large disjunctions…) pyth* AND prog* If you encourage “laziness” people will respond! Which web search engines allow wildcard queries? Search 00:44:00  00:45:00 Type your search terms, use ‘*’ if you need to. E.g., Alex* will match Alexander.

Soundex

Soundex Class of heuristics to expand a query into phonetic equivalents Language specific – mainly for names E.g., chebyshev  tchebycheff Invented for the U.S. census … in 1918 00:45:50  00:46:20

Soundex – typical algorithm Retain the first letter of the word. Change all occurrences of the following letters to '0' (zero):   'A', E', 'I', 'O', 'U', 'H', 'W', 'Y'. Change letters to digits as follows: B, F, P, V  1 C, G, J, K, Q, S, X, Z  2 D,T  3 L  4 M, N  5 R  6 e.g. Herman H0rm0n H0r505 H06505 00:48:00  00:50:50(Layering)

Will hermann generate the same code? Soundex continued Remove all pairs of consecutive digits. Remove all zeros from the resulting string. Pad the resulting string with trailing zeros and return the first four positions, which will be of the form <uppercase letter> <digit> <digit> <digit>. E.g., Herman becomes H655. 00:50:54  00:52:30(Layering) Will hermann generate the same code?

Soundex Soundex is the classic algorithm, provided by most databases (Oracle, Microsoft, …) How useful is soundex? Generally used for Proper Nouns. Database systems also provide soundex option. Get names whose last name is same as soundex(SMITH) We may get Lindsay and William It lowers precision. But increases Recall. Can be used by Interpol to check names. Generally such algorithm are tailored according to European names. There are other variants which work well for IR. 00:53:14  00:54:40

Resources Peter Norvig: How to write a spelling corrector IIR 3, MG 4.2 Efficient spell retrieval: K. Kukich. Techniques for automatically correcting words in text. ACM Computing Surveys 24(4), Dec 1992. J. Zobel and P. Dart.  Finding approximate matches in large lexicons.  Software - practice and experience 25(3), March 1995. http://citeseer.ist.psu.edu/zobel95finding.html Mikael Tillenius: Efficient Generation and Ranking of Spelling Error Corrections. Master’s thesis at Sweden’s Royal Institute of Technology. http://citeseer.ist.psu.edu/179155.html Nice, easy reading on spell correction: Peter Norvig: How to write a spelling corrector http://norvig.com/spell-correct.html