1. INFORMATION RETRIEVAL TECHNIQUES BY DR. ADNAN ABID
Lecture # 19
Permuterm index
k-gram

2. ACKNOWLEDGEMENTS
The presentation of this lecture has been taken from the underline sources
“Introduction to information retrieval” by Prabhakar Raghavan, Christopher D. Manning, and Hinrich Schütze
“Managing gigabytes” by Ian H. Witten, ‎Alistair Moffat, ‎Timothy C. Bell
“Modern information retrieval” by Baeza-Yates Ricardo, ‎ 
“Web Information Retrieval” by Stefano Ceri, ‎Alessandro Bozzon, ‎Marco Brambilla

3. Outline
Permuterm index
k-gram
Soundex

4. Permuterm index For term hello, index under: Queries:
hello$, ello$h, llo$he, lo$hel, o$hell
where $ is a special symbol.
Queries:
X lookup on X$ X* lookup on $X*
*X lookup on X$* *X* lookup on X*
X*Y lookup on Y$X* X*Y*Z ??? Exercise!
a*e*i*o*u  a * u 
u$a*  look up B tree
X, X*, *X are all simple and can be handled without permuterm.
X*Y and X*Y*Z can be handled with permuterm.
00:01:45  00:03:32(For term)
00:04:50  00:05:10
00:06:35  00:08:15
00:12:07  00:13:10
00:16:35  00:17:00(a*e*)
00:19:00  00:20:00(Permuterm Index)
Permuterm index increases the size of dictionary 4 times
FOR ENGLINSH
Depends on average length of a word
Query = hel*o
X=hel, Y=o
Lookup o$hel*

5. Bigram (k-gram) indexes
Enumerate all k-grams (sequence of k chars) occurring in any term
e.g., from text “April is the cruelest month” we get the 2-grams (bigrams)
$ is a special word boundary symbol
Maintain a second inverted index from bigrams to dictionary terms that match each bigram.
$a,ap,pr,ri,il,l$,$i,is,s$,$t,th,he,e$,$c,cr,ru,
ue,el,le,es,st,t$, $m,mo,on,nt,h$
00:21:35  00:22:30
00:22:55  00:24:20

6. Bigram index example
The k-gram index finds terms based on a query consisting of k-grams (here k=2). $m
mace
madden mo
among
amortize on
along
among
00:25:10  00:26:15
00:26:35  00:27:25

7. Processing wild-cards
Query mon* can now be run as
$m AND mo AND on
Gets terms that match AND version of our wildcard query.
But we’d enumerate moon.
Must post-filter these terms against query.
Surviving enumerated terms are then looked up in the term- document inverted index.
Fast, space efficient (compared to permuterm).
00:28:25  00:30:00

8. K-gram Index
Larger k-grams would be lesser flexibility in query processing.
Normally bi and tri k-gram proffered.
00:41:40  00:42:00

9. 00:42:55 00:43:05

10. Processing wild-card queries
As before, we must execute a Boolean query for each enumerated, filtered term.
Wild-cards can result in expensive query execution (very large disjunctions…)
pyth* AND prog*
If you encourage “laziness” people will respond!
Which web search engines allow wildcard queries?
Search
00:44:00  00:45:00
Type your search terms, use ‘*’ if you need to.
E.g., Alex* will match Alexander.

11. Soundex

12. Soundex
Class of heuristics to expand a query into phonetic equivalents
Language specific – mainly for names
E.g., chebyshev  tchebycheff
Invented for the U.S. census … in 1918
00:45:50  00:46:20

13. Soundex – typical algorithm
Retain the first letter of the word.
Change all occurrences of the following letters to '0' (zero):   'A', E', 'I', 'O', 'U', 'H', 'W', 'Y'.
Change letters to digits as follows:
B, F, P, V  1
C, G, J, K, Q, S, X, Z  2
D,T  3
L  4
M, N  5
R  6
e.g. Herman
H0rm0n
H0r505
H06505
00:48:00  00:50:50(Layering)

14. Will hermann generate the same code?
Soundex continued
Remove all pairs of consecutive digits.
Remove all zeros from the resulting string.
Pad the resulting string with trailing zeros and return the first four positions, which will be of the form <uppercase letter> <digit> <digit> <digit>.
E.g., Herman becomes H655.
00:50:54  00:52:30(Layering)
Will hermann generate the same code?

15. Soundex
Soundex is the classic algorithm, provided by most databases (Oracle, Microsoft, …)
How useful is soundex?
Generally used for Proper Nouns.
Database systems also provide soundex option.
Get names whose last name is same as soundex(SMITH)
We may get Lindsay and William
It lowers precision.
But increases Recall.
Can be used by Interpol to check names.
Generally such algorithm are tailored according to European names.
There are other variants which work well for IR.
00:53:14  00:54:40

16. Resources Peter Norvig: How to write a spelling corrector
IIR 3, MG 4.2
Efficient spell retrieval:
K. Kukich. Techniques for automatically correcting words in text. ACM Computing Surveys 24(4), Dec 1992.
J. Zobel and P. Dart.  Finding approximate matches in large lexicons.  Software - practice and experience 25(3), March
Mikael Tillenius: Efficient Generation and Ranking of Spelling Error Corrections. Master’s thesis at Sweden’s Royal Institute of Technology.
Nice, easy reading on spell correction:
Peter Norvig: How to write a spelling corrector