Derrick Coetzee, Microsoft Research CC0 waiverCC0 waiver: To the extent possible under law, I waive all copyright and related or neighboring rights to all content in this presentation.

 Consider searching for a subsequence in a collection of genome sequences: …gcaagctttatagtgacaacaataaggtatcactcggtt…  N-gram inverted indexes are the traditional solution, but have times more terms than ordinary word-based inverted indexes  TinyLex indexes achieve similar query performance with 7-17 times less terms  TinyLex provides good worst-case query performance 2 TinyLex: Static N-Gram Index Pruning - Derrick Coetzee

 1. Each wife had seven sacks,  2. Each sack had seven cats,  3. Each cat had seven kits.  4. Kits, cats, sacks, and wives. each: {1, 2, 3} had: {1, 2, 3} seven: {1, 2, 3} wife: {1, 4} sack: {1, 2, 4} cat: {2, 3, 4} kit: {3, 4} 3 TinyLex: Static N-Gram Index Pruning - Derrick Coetzee

 1. Each wife had seven sacks,  2. Each sack had seven cats,  3. Each cat had seven kits.  4. Kits, cats, sacks, and wives. 4 TinyLex: Static N-Gram Index Pruning - Derrick Coetzee Query: sack and cat sack: {1, 2, 4} cat: {2, 3, 4} {1, 2, 4} ∩ {2, 3, 4} = {2, 4}

 Partial word or punctuation queries ◦ Searching a dictionary for all words ending in “ment” ◦ Searching for in HTML files ◦ Searching for "%s" in C source files ◦ Searching for x^2/2 in LaTeX source files  Searching East Asian language text ◦ No spaces, word extraction is complex  Phrase searching 5 TinyLex: Static N-Gram Index Pruning - Derrick Coetzee

Genome sequences:  1. gcaagctttatagtgacaac...  2. aataaggtatcactcggtta...  3. caattacccccacttcccct...  4. cattataaagaaatgatcaa... Example query: Documents containing subsequence “cact” 6 TinyLex: Static N-Gram Index Pruning - Derrick Coetzee

Simplified example: Two-letter alphabet  1. babbbbabab  2. aababaaabb  3. babababaab  4. bbbbaabbbb aaa: {2} aab: {2, 3, 4} aba: {1, 2, 3} abb: {1, 2, 4} baa: {2, 3, 4} bab: {1, 2, 3} bba: {1, 4} bbb: {1, 4} 7 TinyLex: Static N-Gram Index Pruning - Derrick Coetzee

 1. babbbbabab  2. aababaaabb  3. babababaab  4. bbbbaabbbb 8 TinyLex: Static N-Gram Index Pruning - Derrick Coetzee Query: aaba aaba aab and aba

 1. babbbbabab  2. aababaaabb  3. babababaab (false positive)  4. bbbbaabbbb 9 TinyLex: Static N-Gram Index Pruning - Derrick Coetzee Query: aaba aab and aba aab: {2, 3, 4} aba: {1, 2, 3} {2, 3, 4} ∩ {1, 2, 3} = {2, 3}

 1. babbbbabab  2. aababaaabb  3. babababaab  4. bbbbaabbbb a: {1, 2, 3, 4} b: {1, 2, 3, 4} Small number of terms Slow queries Long posting lists Too many false positives length = 1 10 TinyLex: Static N-Gram Index Pruning - Derrick Coetzee

 1. babbbbabab  2. aababaaabb  3. babababaab  4. bbbbaabbbb aababa: {2} aabbbb: {4} abaaab: {2} ababaa: {2,3} ababab: {3} abbbba: {1} baaabb: {2} baabbb: {4} babaaa: {2} babaab: {3} bababa: {3} babbbb: {1} bbaabb: {4} bbabab: {1} bbbaab: {4} bbbaba: {1} bbbbaa: {4} bbbbab: {1} Fast queries Too many terms Queries must be ≥6 characters length = 6 11 TinyLex: Static N-Gram Index Pruning - Derrick Coetzee

 Review of inverted n-gram indexes  Example TinyLex index  TinyLex index construction  Results  Disadvantages  Questions 12 TinyLex: Static N-Gram Index Pruning - Derrick Coetzee

 Goal: less terms without sacrificing query performance  Consider the n-grams “juggl” and “uggle” ◦ Almost exactly the same posting list in a typical English language collection ◦ Just put the n-gram “uggl” in the index, and leave out “juggl” and “uggle” 13 TinyLex: Static N-Gram Index Pruning - Derrick Coetzee juggl: {2, 7, 33} uggle: {2, 7, 33} uggl: {2,7,33}

 Insight: The more false positives a term produces when it is queried for, the more information it adds when it is added to the index.  Choose a false positive threshold t and choose the smallest possible set of index terms that satisfies it.  Allow variable-length n-grams. 14 TinyLex: Static N-Gram Index Pruning - Derrick Coetzee

 1. babbbbabab  2. aababaaabb  3. babababaab  4. bbbbaabbbb aa: {2, 3, 4} bb: {1, 2, 4} aaa: {2} aba: {1, 2, 3} bab: {1, 2, 3} bba: {1, 4} bbb: {1, 4} aaba: {2} baab: {3, 4} babb: {1} 15 TinyLex: Static N-Gram Index Pruning - Derrick Coetzee In this example t = 1. At most 1 false positive is allowed for any query. Only 10 terms!

 1. babbbbabab  2. aababaaabb  3. babababaab  4. bbbbaabbbb 16 TinyLex: Static N-Gram Index Pruning - Derrick Coetzee Query: abaab aba and baab aba: {1, 2, 3} baab: {3, 4} {1, 2, 3} ∩ {3, 4} = {3}

 The construction guarantees that if the query term occurs in the collection, it will have at most t – 1 false positives (zero in this case).  If we observe t false positives, we can halt immediately. 17 TinyLex: Static N-Gram Index Pruning - Derrick Coetzee

18 TinyLex: Static N-Gram Index Pruning - Derrick Coetzee Query: bbbbb bbb and bbb and bbb bbb: {1, 4} {1, 4} ∩ {1, 4} ∩ {1, 4} = {1, 4} 1.babbbbabab (false positive)...can’t happen unless the query result is empty. Halt.

 Achieve similar query performance to classical n-gram indexes with a much larger number of terms  Worst-case bound on number of false positives  Query can be any length 19 TinyLex: Static N-Gram Index Pruning - Derrick Coetzee

 Review of inverted n-gram indexes  Example TinyLex index  TinyLex index construction  Results  Disadvantages  Questions 20 TinyLex: Static N-Gram Index Pruning - Derrick Coetzee

 The problem: ◦ Input: a set of documents, a threshold t ◦ Output: a list of terms such that any query for a term occurring in the collection will have at most t – 1 false positives 21 TinyLex: Static N-Gram Index Pruning - Derrick Coetzee

 Basic construction:  For each n-gram length from 1 to max: ◦ Make a list of all n-grams in the collection and what documents they occur in. ◦ Perform a query on each term using the partially constructed index. ◦ If a term has too many false positives, add it to the index. 22 TinyLex: Static N-Gram Index Pruning - Derrick Coetzee

 1. babbbbabab  2. aababaaabb  3. babababaab  4. bbbbaabbbb (index empty) 23 TinyLex: Static N-Gram Index Pruning - Derrick Coetzee 1-gramsQuery result Actual a{1,2,3,4} b t = 1 If the difference between the query result size and the actual posting list size is at least 1, add it to the index.

 1. babbbbabab  2. aababaaabb  3. babababaab  4. bbbbaabbbb 24 TinyLex: Static N-Gram Index Pruning - Derrick Coetzee 2-gramsQuery result Actual aa{1,2,3,4}{2,3,4} ab{1,2,3,4} ba{1,2,3,4} bb{1,2,3,4}{1,2,4} (index empty)

 1. babbbbabab  2. aababaaabb  3. babababaab  4. bbbbaabbbb 25 TinyLex: Static N-Gram Index Pruning - Derrick Coetzee 2-gramsQuery result Actual aa{1,2,3,4}{2,3,4} ab{1,2,3,4} ba{1,2,3,4} bb{1,2,3,4}{1,2,4} aa: {2,3,4} bb: {1,2,4}

    aa: {2,3,4} bb: {1,2,4} 26 TinyLex: Static N-Gram Index Pruning - Derrick Coetzee 3-gramsQuery result Actual aaa{2,3,4}{2} aab{2,3,4} aba{1,2,3,4}{1,2,3} abb{1,2,4} baa{2,3,4} bab{1,2,3,4}{1,2,3} bba{1,2,4}{1,4} bbb{1,2,4}{1,4}

    aa: {2,3,4} bb: {1,2,4} aaa: {2} aba: {1,2,3} bab: {1,2,3} bba: {1,4} bbb: {1,4} 27 TinyLex: Static N-Gram Index Pruning - Derrick Coetzee 3-gramsQuery result Actual aaa{2,3,4}{2} aab{2,3,4} aba{1,2,3,4}{1,2,3} abb{1,2,4} baa{2,3,4} bab{1,2,3,4}{1,2,3} bba{1,2,4}{1,4} bbb{1,2,4}{1,4}

    TinyLex: Static N-Gram Index Pruning - Derrick Coetzee 4-gramsQuery result Actual aaab{2} aaba{2,3}{2} aabb{2,4} abaa{2,3} abab{1,2,3} abbb{1,4} baaa{2} baab{2,3,4}{3,4} baba{1,2,3} babb{1,2}{1} bbaa{4} bbab{1} bbba{1,4} bbbb{1,4} aa: {2,3,4} bb: {1,2,4} aaa: {2} aba: {1,2,3} bab: {1,2,3} bba: {1,4} bbb: {1,4}

    aa: {2,3,4} bb: {1,2,4} aaa: {2} aba: {1,2,3} bab: {1,2,3} bba: {1,4} bbb: {1,4} aaba: {2} baab: {3,4} babb: {1} 29 TinyLex: Static N-Gram Index Pruning - Derrick Coetzee 4-gramsQuery result Actual aaab{2} aaba{2,3}{2} aabb{2,4} abaa{2,3} abab{1,2,3} abbb{1,4} baaa{2} baab{2,3,4}{3,4} baba{1,2,3} babb{1,2}{1} bbaa{4} bbab{1} bbba{1,4} bbbb{1,4}

 Review of inverted n-gram indexes  Example TinyLex index  TinyLex index construction  Results  Disadvantages  Questions 30 TinyLex: Static N-Gram Index Pruning - Derrick Coetzee

31 TinyLex: Static N-Gram Index Pruning - Derrick Coetzee  Test set: 100MB TREC WSJ collection  documents, English text  Same query performance with 7-17 times less terms

32 TinyLex: Static N-Gram Index Pruning - Derrick Coetzee  Overall compressed index size 2-20% less  TinyLex index has more information per term

33 TinyLex: Static N-Gram Index Pruning - Derrick Coetzee  Dramatic 50x improvement in worst-case query performance for long queries

 Applications to phrase searching using variable-length word n-grams  Making the construction more efficient  Performance on genome sequences  Empirical evaluation of scaling 34 TinyLex: Static N-Gram Index Pruning - Derrick Coetzee

 Suffix arrays (Manber and Myers 1991) ◦ Faster queries, but indexes 3-10 times larger  agrep and GLIMPSE (Wu and Manber 1994) ◦ More general queries, but relies on a word concept  n-Gram/2L (Kim et al 2005) ◦ Orthogonal; examines less document offsets  “Growing an n-gram language model” ◦ (Siivola and Pellom 2005) ◦ Similar idea applied to language modeling 35 TinyLex: Static N-Gram Index Pruning - Derrick Coetzee

 Faster construction time ◦ Currently about 10 times slower to construct than a classical n-gram index.  Queries for nonoccurring terms are more expensive than with classical n-gram indexes (t documents must be read).  Generalize to dynamic collections 36 TinyLex: Static N-Gram Index Pruning - Derrick Coetzee

 N-gram indexes enable practical queries for subsequences  TinyLex indexes achieve similar query performance to classical n-gram indexes with 7-17 times less terms  TinyLex yields good worst-case query performance by placing an upper bound on the number of false positives 37 TinyLex: Static N-Gram Index Pruning - Derrick Coetzee

38 TinyLex: Static N-Gram Index Pruning - Derrick Coetzee