Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3

Augment postings with skip pointers (at indexing time) How do we deploy them ? Where do we place them ? Sec. 2.3

Using skips Suppose we’ve stepped through the lists until we process 8 on each list. We match it and advance. We then have 41 and 11 on the lower. 11 is smaller. But the skip successor of 11 on the lower list is 31, so we can skip ahead past the intervening postings. Sec. 2.3

Placing skips Tradeoff: More skips  shorter spans  more likely to skip. But lots of comparisons to skip pointers. Fewer skips  longer spans  few successful skips. Less pointer comparisons. Sec. 2.3

Placing skips Simple heuristic for postings of length L use  L evenly-spaced skip pointers. This ignores the distribution of query terms. Easy if the index is relatively static. This definitely useful for in-memory index The I/O cost of loading a bigger list can outweigh the gains! Sec. 2.3

Placing skips, contd What if it is known a distribution of access p k to the k-th element of the inverted list? w(i,j) = sum k=i..j p k L^0(i,j) = average cost of accessing an item in the sublist from i to j = sum k=i..j p k * (k-i+1) L^1(1,n) = 1 (first skip cmp) + (cost to access the two lists) min u>1 w(1,u-1) * L^0(1,u-1) + w(u,n) * L^1(u,n) L^0(i,j) can be tabulated in O(n^2) time Computing L^1(i,n) takes O(n), given L^1(j,n), for j>i Computing the total L^1(1,n) takes O(n^2) time Sec. 2.3 You can solve it by Shortest Path

Placing skips, contd What if it is also fixed the number of p skip- pointers that can be allocated? Same as before but we add the parameter p L^1_p(1,n) = 1 + min_{u>1} w(1,u-1) * L^0(1,u-1) + w(u,n) * L^1_{p-1}(u,n) L^1_0(i,j) = L^0(i,j), i.e. no pointers left, so scan L^i(j,n) takes O(n) time [min calculation] if are available the values for L^{i-1}(h,n) with h > j So L^p(1,n) takes O(pn^2) time Sec. 2.3

Faster query = caching? Two opposite approaches: I. Cache the query results (exploits query locality) II. Cache pages of posting lists (exploits term locality)

Query processing: phrase queries and positional indexes Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.4

Phrase queries Want to be able to answer queries such as “stanford university” – as a phrase Thus the sentence “I went at Stanford my university” is not a match. Sec. 2.4

Solution #1: Biword indexes For example the text “Friends, Romans, Countrymen” would generate the biwords friends romans romans countrymen Each of these biwords is now an entry in the dictionary Two-word phrase query-processing is immediate. Sec

Longer phrase queries Longer phrases are processed by reducing them to bi-word queries in AND stanford university palo alto can be broken into the Boolean query on biwords, such as stanford university AND university palo AND palo alto Need the docs to verify + They are combined with other solutions Can have false positives! Index blows up Sec

Solution #2: Positional indexes In the postings, store for each term and document the position(s) in which that term occurs: <term, number of docs containing term; doc1: position1, position2 … ; doc2: position1, position2 … ; etc.> Sec

Processing a phrase query “to be or not to be”. to: 2:1,17,74,222,551; 4:8,16,190,429,433; 7:13,23,191;... be: 1:17,19; 4:17,191,291,430,434; 5:14,19,101;... Same general method for proximity searches Sec

Query term proximity Free text queries: just a set of terms typed into the query box – common on the web Users prefer docs in which query terms occur within close proximity of each other Would like scoring function to take this into account – how? Sec

Positional index size You can compress position values/offsets Nevertheless, a positional index expands postings storage by a factor 2-4 in English Nevertheless, a positional index is now commonly used because of the power and usefulness of phrase and proximity queries … whether used explicitly or implicitly in a ranking retrieval system. Sec

Combination schemes BiWord + Positional index is a profitable combination Biword is particularly useful for particular phrases (“Michael Jackson”, “Britney Spears”) More complicated mixing strategies do exist! Sec

Soft-AND E.g. query rising interest rates Run the query as a phrase query If <K docs contain the phrase rising interest rates, run the two phrase queries rising interest and interest rates If we still have <K docs, run the “vector space query” rising interest rates (…see next…) “Rank” the matching docs (…see next…) Sec