1. Lower bounds for approximate membership dynamic data structures
Shachar Lovett
IAS
Ely Porat
Bar-Ilan University
Synergies in lower bounds, June 2011

2. Information theoretic lower bounds
Information theory is a powerful tool to prove lower bounds, e.g. in data structures
Study size of data structure (unlimited access)
Static d.s.: pure information theory
Dynamic d.s.: communication game

3. Talk overview Approximate set membership problem
Bloom filters (simple near-optimal solution)
Lower bounds – static case
New dynamic lower bounds

4. Talk overview Approximate set membership problem
Bloom filters (simple near-optimal solution)
Lower bounds – static case
New dynamic lower bounds

5. Approximate set membership
Large universe U
Represent subset S  U
Query: is x  S?
Data structure representing S approximately:
If x  S: answer YES always
If x  S: answer NO with high probability
Why approximately? To save space U ~S S

6. Applications
Storage (or communication) is costly, but a small false positive error can be tolerated
Original applications (70’s): dictionaries, databases – Bloom filters
Nowadays: mainly network applications

7. Talk overview Approximate set membership problem
Bloom filters (simple near-optimal solution)
Lower bounds – static case
New dynamic lower bounds

8. Bloom filters S={x1,x2,…,xn} Hash function h:U  {1,…,m}
Bit array of length m

9. Bloom filters S={x1,x2,…,xn} Hash function h:U  {1,…,m}
h(x1)=4 1
Bit array of length m

10. Bloom filters S={x1,x2,…,xn} Hash function h:U  {1,…,m}
h(x2)=1 1
Bit array of length m

11. Bloom filters S={x1,x2,…,xn} Hash function h:U  {1,…,m}
h(x3)=4 1
Bit array of length m

12. Bloom filters S={x1,x2,…,xn} Query: y  S? Hash function h:U  {1,…,m}
Bit array of length m

13. Bloom filters S={x1,x2,…,xn} Query: y  S? Hash function h:U  {1,…,m}
h(y)=3 1
Bit array of length m

14. Bloom filters S={x1,x2,…,xn} Query: y  S? NO Hash function
h:U  {1,…,m}
Query: y  S? NO
h(y)=3 1
Bit array of length m

15. Bloom filters: analysis
hash
S={x1,x2,…,xn}
Query: y  S?
If y  S: returns YES always
If y  S: returns NO with probability
Error ½:
Error : (repetition) 1
Bit array of length m

16. Known bounds Upper bounds (e.g. algorithms) Lower bounds:
Bloom filter:
Improvements:
[Porat-Matthias’03, Arbitman-Naor-Segev’10]
Lower bounds:
information theoretic:
Can be matched by static data structures
[Charles-Chellapilla’08,Dietzfelbinger-Pagh’08,Porat’08]
This work: dynamic d.s.

17. Talk overview Approximate set membership problem
Bloom filters (simple near-optimal solution)
Lower bounds – static case
New dynamic lower bounds

18. Static lower bounds Static settings: insert + query
Yao’s min-max principle: prove lower bound for deterministic data structure, randomized inputs
Insert: x1,…,xn
m bits
Query: y

19. Static lower bounds Deterministic data structure: compression
Insert: x1,…,xn
m bits
Query: y
Static lower bounds
Deterministic data structure: compression
maps all sets to a small family of sets
Input: random set
Accept set:
Properties:
Small memory:
No false negatives:
Few false positives:
Optimal setting:

20. Static lower bounds Set S, Represented by Goal: show #A(S) large U
Insert: x1,…,xn
m bits
Query: y
Static lower bounds U
A(S) S
Set S,
Represented by
Goal: show #A(S) large

21. Static lower bounds Properties: Assume that General case: convexity
Insert: x1,…,xn
m bits
Query: y
Static lower bounds
Properties:
Assume that
If then
General case: convexity

22. Talk overview Approximate set membership problem
Bloom filters (simple near-optimal solution)
Lower bounds – static case
New dynamic lower bounds

23. Dynamic lower bounds Basic dynamic settings: two inserts + query
Break inputs to k, n-k chunks
m bits
m bits
Insert: x1,…,xk
Insert: xk+1,…,xn
Query: y

24. Dynamic lower bounds Accepting sets: Properties:
Insert: x1,…,xk
m bits
Insert: xk+1,…,xn
Query: y
Dynamic lower bounds
Accepting sets:
Properties:
General approach: analyze size of accepting sets
Sets A(x1,…,xk) can’t be too small (covering)
Sets A(A(x1,…,xk),xk+1,…,xn) can’t be too large (error)
These yield the trivial lower bound again… 

25. Dynamic lower bounds Method of typical inputs On a typical input:
Insert: x1,…,xk
m bits
Insert: xk+1,…,xn
Query: y
Dynamic lower bounds
Method of typical inputs
On a typical input:
A(x1,…,xk) not too small
A(A(x1,…,xk),xk+1,…,xn) not too large
Inputs uncorrelated with data structure:
Yields an improved lower bound 
(note: “typical” can be 1% of inputs)

26. Dynamic lower bounds Functional inequality:
Insert: x1,…,xk
m bits
Insert: xk+1,…,xn
Query: y
Dynamic lower bounds
Functional inequality:
Free parameter: k – how to break input
Optimal choice:
Extension: break input into more parts
Doesn’t seem to help much

27. Summary THANK YOU! Approximate membership problem
Static algorithms match static information theoretic lower bound:
This work: new dynamic information theoretic lower bound
THANK YOU!