1. Lower Bounds for 2-Dimensional Range Counting Mihai Pătraşcu summer @

2. Range Counting SELECT count(*) FROM employees WHERE salary <= 70000 AND startdate <= 1998 989997 96 70000 69000 68000 71000

3. Some Theory d dimensions range trees (roughly): space S = n lg d -1 n, query t =lg d -1 n space S = n 2d, query t =O(1) geometric extensions: range = disks, half-spaces, polygons… PROBLEM: lower bounds

4. The Semigroup Industry Let (U,+) be a semigroup Points have weights in U Given w 1,…,w n : precompute S sums query: add t precomputed sums w1w1 w2w2 w4w4 w5w5 w6w6 w7w7 w3w3 w 1 +w 2 +w 4 +w 5

5. Why Industry? tight semigroup lower bounds [Fredman JACM81] [Chazelle FOCS86] many, many excellent bounds for many range problems Why are we so good at this? [Caravaggio]

6. Semigroup = Low Dimension say Mem [17]= w 2 + w 6 can Mem[17] help with this query? NO : cannot subtract w 6 Mem [17] described by bounding box can only be used when query dominates bounding box How well do rectangles decompose into rectangles? X (disks, half-planes…) Y w1w1 w2w2 w4w4 w5w5 w6w6 w7w7 w3w3 All these objects are low-dimensional!

7. Computation = High Dimension New concept… weights come from a group (U,+,-) can cancel any weight => no bounding box relevant information about Mem[17] : O(1) -D rectangle linear combination of n variables decomposability in O(1) -D decomposability in n -D n -D means: geometry information theory

8. The Ultimate Frontier The cell-probe model: plain old counting (weights {0,1} ) each cell stores any function of inputs query probes t cells (adaptively), computes any function Theory of the computer in your lap

9. To boldly go where no one has gone before General lack of group lower bounds (never mind cell-probe!) [Chazelle STOC95] (lglg n) [Fredman JACM82 ] [Chazelle FOCS86 ] [Agarwal 9x, 0x ] etc yeah, we have nice semigroup lower bounds but prove something in the group model

10. Our Small Step If S = n lg O(1) n, then t = (lg n / lglg n) group model! …and cell-probe model! tight in 2D Maybe not so giant leap for mankind… does not grow with d => really only relevant for 2D

11. Basic Idea Remember: Separating linear from polynomial space: [P,Thorup STOC06 ] (lglg n) [P,Thorup SODA06 ]---- randomized [P,Thorup FOCS06 ] (lg n / lglg n) (unnatural problems; not tight) NOW (lg n / lglg n) (natural, fundamental problem; tight) space S = n lg d -1 n, query t =lg d -1 n space S = n 2d, query t =O(1) technique framework trick The trick: Consider n / lg n queries, see what happens This paper: what happens for range counting

12. Hard Instance The bit-reversal permutation (see FFT, etc): e.g. π(6) = π(110) = 011 = 3 Well-known hard instance: points at (x, π ( x ) ) random query [0,a] X [0,b] [in fact, many independent random queries] x 01234567 π(x) 04261537

13. The Shuffle View 0000 1111 2222 3333 4444 5555 6666 7777

14. 7777 6666 3333 1111 0000 1111 2222 3333 4444 5555 6666 7777 0000 2222 5555 4444

15. 7777 6666 3333 1111 0000 1111 2222 3333 4444 5555 6666 7777 0000 2222 5555 4444

16. 7777 5555 7777 6666 3333 1111 0000 1111 2222 3333 4444 5555 6666 7777 0000 2222 5555 4444 6666 3333 1111 2222 4444 0000

17. 7777 5555 7777 6666 3333 1111 0000 1111 2222 3333 4444 5555 6666 7777 0000 2222 5555 4444 6666 3333 1111 2222 4444 0000

18. 7777 3333 5555 1111 7777 5555 7777 6666 3333 1111 0000 1111 2222 3333 4444 5555 6666 7777 0000 2222 5555 4444 6666 3333 1111 2222 4444 0000 6666 2222 4444 0000

19. 7777 3333 5555 1111 7777 5555 7777 6666 3333 1111 0000 1111 2222 3333 4444 5555 6666 7777 0000 2222 5555 4444 6666 3333 1111 2222 4444 0000 6666 2222 4444 0000 x 01234567 π(x) 04261537

20. 7777 3333 5555 1111 7777 5555 7777 6666 3333 1111 The Shuffle View 0000 1111 2222 3333 4444 5555 6666 7777 0000 2222 5555 4444 6666 3333 1111 2222 4444 0000 6666 2222 4444 0000 x 01234567 π(x) 04261537 4 1 2 3 5 6 7 8

21. 7777 3333 5555 1111 7777 5555 7777 6666 3333 1111 The Shuffle View 0000 1111 2222 3333 4444 5555 6666 7777 0000 2222 5555 4444 6666 3333 1111 2222 4444 0000 6666 2222 4444 0000 x 01234567 π(x) 04261537 4 1 2 3 5 6 7 8

22. 7777 3333 5555 1111 7777 5555 7777 6666 3333 1111 The Shuffle View 0000 1111 2222 3333 4444 5555 6666 7777 0000 2222 5555 4444 6666 3333 1111 2222 4444 0000 6666 2222 4444 0000 x 01234567 π(x) 04261537 4 1 2 3 5 6 7 8

23. Shuffling the Queries 0000 1111 2222 3333 4444 5555 6666 7777 x 01234567 π(x) 04261537 query [0,4] X [0,5] 4 1 2 3 5 6 7 8

24. Shuffling the Queries 0000 1111 2222 3333 4444 5555 6666 7777 x 01234567 π(x) 04261537 query [0,5] X [0,4] 7777 6666 3333 1111 0000 2222 5555 4444 1 2 3 5 6 7 8 4

25. 7777 6666 3333 1111 Shuffling the Queries 0000 1111 2222 3333 4444 5555 6666 7777 0000 2222 x 01234567 π(x) 04261537 query [0,5] X [0,4] 5555 4444 7777 5555 3333 1111 2222 0000 6666 4444 4 1 2 3 5 6 7 8

26. 4 1 2 3 5 6 7 8 7777 5555 7777 6666 3333 1111 Shuffling the Queries 0000 1111 2222 3333 4444 5555 6666 7777 0000 2222 3333 1111 2222 0000 x 01234567 π(x) 04261537 query [0,5] X [0,4] 5555 4444 6666 4444 7777 3333 5555 1111 6666 2222 4444 0000 Specifying a query = Saying how it shuffles at every level

27. Proof Sketch (I) k queries, space S => one probe revels lg = Θ(k lg (S/k)) bits of information about the queries t probes => Θ( t k lg (S/k)) bits k=n / lg nS= n lg O(1) n t = o (lg n / lglg n) => o(k lg n) bits revealed by the probes about the queries SkSk

28. Proof Sketch (II) I( queries ) I( shuffling at level 1 ) + I( shuffling at level 2 ) + … + I( shuffling at level lg n) o(k lg n) so for one level the data structure learns o(k) bits of information about the queries [i.e. doesnt know where most queries fit] so it cannot precompute useful sums [see Proceedings]

29. D N E T H E

30. 15 13 14 15 13 11 8888 9999 10 11 12 13 14 15 9999 10 12 14 8888 11 9999 10 12 8888 15 13 14 11 9999 10 12 8888