Rank and Select data structures Prof. Paolo Ferragina, Algoritmi per "Information Retrieval" Rank and Select data structures

A basic problem ! How do you retrieve the k-th string? D D B Abaco$Battle$Car$Cold$Cod .... D Array of n string pointers to strings of total length m (n log m) bits = 32 n bits. it depends on the number of strings it is independent of string length Abaco Battle Car Cold Cod .... D 10000 100000 100 1000 100 .... B Spaces are introduced for simplicity You could drop the $ How do you retrieve the k-th string?

Rank/Select Wish to index the bit vector B[1,m] (possibly compressed). 00101001010101011111110000011010101.... Rank1(6) = 2 m = |B| n = #1 Rankb(i) = number of b in B[1,i] Selectb(i) = position of the i-th b in B Two approaches: Takes |B| + o(|B|) bits of space, Aims at achieving n log(m/n) bits, by deplyoing Elias-Fano + point (1)

The Bit-Vector Index: |B| + o(|B|) m = |B| n = #1s The Bit-Vector Index: |B| + o(|B|) Goal. B is read-only, and the additional index takes o(m) bits. Rank B 00101001010101011 1111100010110101 0101010111000.... Z 8 18 block pos #1 z (bucket-relative) Rank1 4 5 8 0000 1 .... ... 1011 2 (absolute) Rank1 Setting Z = poly(log m) and z=(1/2) log m: Extra space is + (m/Z) log m + (m/z) log Z + o(m) + O(m loglog m / log m) = o(m) bits Rank time is O(1) Term o(m) is crucial in practice, B is untouched (not compressed)

The Select operation B Extra space is + o(m), and B is not touched! m = |B| n = #1s The Select operation B 0010100101010101111111000001101010101010111001.... size r is variable until the subarray includes k = (log m)2 1s Sparse case: If r > k2 = (log m)4 , we store explicitly the position of the k = (log m)2 1s, because we have at most (m/r) blocks of this type, each taking (m/r) * k * log m bits = O(m / log m) = o(m) bits Dense case: k ≤ r ≤ k2, recurse by repeating the argument now with k’ = (log log m)2. If r’ including k’ 1s > log m bits, then store the k’ positions explicitly using O(log log m) bits each, thus O(m/log log m) = o(m) bits in total. Otherwise r’ < log m, and thus a precomputed table is enough. Extra space is + o(m), and B is not touched! Select time is O(1)

Via Elias-Fano (|L| + |H| + o(|H|)) Therefore B is not needed Recall that by setting w = log (m/n) and z = log n, where m = |B| and n = #1 then Space = n log (m/n) bits + 2n bits (Build Select1 on H so we need extra |H| + o(|H|) bits = 2n + o(n) bits ) z = 3, w=2 0 1 2 3 4 5 6 7 Select1(i) on B  uses L and (Select1(H,i) – i) in +o(n) space Rank1(i) on B  Needs binary search over B

If you wish to play with Rank and Select Prof. Paolo Ferragina, Algoritmi per "Information Retrieval" If you wish to play with Rank and Select m/10 + n log (m/n) Rank in 0.4 msec, Select in < 1 msec vs 32n bits of explicit pointers