1. What’s the Difference? Efficient Set Reconciliation without Prior Context Frank Uyeda University of California, San Diego David Eppstein, Michael T. Goodrich & George Varghese 1

2. Motivation Distributed applications often need to compare remote state. 2 R1 R2 Must solve the Set-Difference Problem! Partition Heals

3. What is the Set-Difference problem? What objects are unique to host 1? What objects are unique to host 2? A A Host 1Host 2 C C A A F F E E B B D D F F 3

4. Example 1: Data Synchronization Identify missing data blocks Transfer blocks to synchronize sets A A Host 1Host 2 C C A A F F E E B B D D F F D D C C B B E E 4

5. Example 2: Data De-duplication Identify all unique blocks. Replace duplicate data with pointers A A Host 1Host 2 C C A A F F E E B B D D F F 5

6. Set-Difference Solutions Trade a sorted list of objects. – O(n) communication, O(n log n) computation Approximate Solutions: – Approximate Reconciliation Tree (Byers) O(n) communication, O(n log n) computation Polynomial Encodings (Minsky & Trachtenberg) – Let “d” be the size of the difference – O(d) communication, O(dn+d 3 ) computation Invertible Bloom Filter – O(d) communication, O(n+d) computation 6

7. Difference Digests Efficiently solves the set-difference problem. Consists of two data structures: – Invertible Bloom Filter (IBF) Efficiently computes the set difference. Needs the size of the difference – Strata Estimator Approximates the size of the set difference. Uses IBF’s as a building block. 7

8. Invertible Bloom Filters (IBF) Encode local object identifiers into an IBF. A A Host 1Host 2 C C A A F F E E B B D D F F IBF 2 IBF 1 8

9. IBF Data Structure Array of IBF cells – For a set difference of size, d, require αd cells (α > 1) Each ID is assigned to many IBF cells Each IBF cell contains: 9 idSumXOR of all ID’s in the cell hashSumXOR of hash(ID) for all ID’s in the cell countNumber of ID’s assign to the cell

10. IBF Encode A A idSum ⊕ A hashSum ⊕ H(A) count++ idSum ⊕ A hashSum ⊕ H(A) count++ idSum ⊕ A hashSum ⊕ H(A) count++ idSum ⊕ A hashSum ⊕ H(A) count++ idSum ⊕ A hashSum ⊕ H(A) count++ idSum ⊕ A hashSum ⊕ H(A) count++ Hash1 Hash2 Hash3 B B C C Assign ID to many cells 10 IBF: αdαd “Add” ID to cell Not O(n), like Bloom Filters! All hosts use the same hash functions

11. Invertible Bloom Filters (IBF) Trade IBF’s with remote host A A Host 1Host 2 C C A A F F E E B B D D F F IBF 2 IBF 1 11

12. Invertible Bloom Filters (IBF) “Subtract” IBF structures – Produces a new IBF containing only unique objects A A Host 1Host 2 C C A A F F E E B B D D F F IBF 2 IBF 1 IBF (2 - 1) 12

13. IBF Subtract 13

14. Timeout for Intuition After subtraction, all elements common to both sets have disappeared. Why? – Any common element (e.g W) is assigned to same cells on both hosts (assume same hash functions on both sides) – On subtraction, W XOR W = 0. Thus, W vanishes. While elements in set difference remain, they may be randomly mixed  need a decode procedure. 14

15. Invertible Bloom Filters (IBF) Decode resulting IBF – Recover object identifiers from IBF structure. A A Host 1Host 2 C C A A F F E E B B D D F F IBF (2 - 1) B B E E C C D D Host 1Host 2 15 IBF 2 IBF 1

16. IBF Decode 16 H(V ⊕ X ⊕ Z) ≠ H(V) ⊕ H(X) ⊕ H(Z) H(V ⊕ X ⊕ Z) ≠ H(V) ⊕ H(X) ⊕ H(Z) Test for Purity: H( idSum ) Test for Purity: H( idSum ) H( idSum ) = hashSum H(V) = H(V) H( idSum ) = hashSum H(V) = H(V)

17. IBF Decode 17

18. IBF Decode 18

19. IBF Decode 19

20. 20 Small Diffs: 1.4x – 2.3x Large Differences: 1.25x - 1.4x How many IBF cells? Space Overhead Set Difference Hash Cnt 3 Hash Cnt 4 Overhead to decode at >99%

21. How many hash functions? 1 hash function produces many pure cells initially but nothing to undo when an element is removed. 21 A A B B C C

22. How many hash functions? 1 hash function produces many pure cells initially but nothing to undo when an element is removed. Many (say 10) hash functions: too many collisions. 22 A A A A B B C C B B C C A A A A B B B B C C C C

23. How many hash functions? 1 hash function produces many pure cells initially but nothing to undo when an element is removed. Many (say 10) hash functions: too many collisions. We find by experiment that 3 or 4 hash functions works well. Is there some theoretical reason? 23 A A A A B B C C C C A A B B B B C C

24. Theory Let d = difference size, k = # hash functions. Theorem 1: With (k + 1) d cells, failure probability falls exponentially. – For k = 3, implies a 4x tax on storage, a bit weak. [Goodrich,Mitzenmacher]: Failure is equivalent to finding a 2-core (loop) in a random hypergraph Theorem 2: With c k d, cells, failure probability falls exponentially – c 4 = 1.3x tax, agrees with experiments 24

25. 25 Large Differences: 1.25x - 1.4x How many IBF cells? Space Overhead Set Difference Hash Cnt 3 Hash Cnt 4 Overhead to decode at >99%

26. Connection to Coding Mystery: IBF decode similar to peeling procedure used to decode Tornado codes. Why? Explanation: Set Difference is equivalent to coding with insert-delete channels Intuition: Given a code for set A, send codewords only to B. Think of B’s set as a corrupted form of A’s. Reduction: If code can correct D insertions/deletions, then B can recover A and the set difference. 26 Reed Solomon Polynomial Methods LDPC (Tornado) Difference Digest Reed Solomon Polynomial Methods LDPC (Tornado) Difference Digest

27. Difference Digests Consists of two data structures: – Invertible Bloom Filter (IBF) Efficiently computes the set difference. Needs the size of the difference – Strata Estimator Approximates the size of the set difference. Uses IBF’s as a building block. 27

28. Strata Estimator A A Consistent Partitioning Consistent Partitioning B B C C 28 ~1/2 ~1/4 ~1/8 1/16 IBF 1 IBF 4 IBF 3 IBF 2 Estimator Divide keys into partitions of containing ~1/2 k Encode each partition into an IBF of fixed size – log(n) IBF’s of ~80 cells each

29. 4x Strata Estimator 29 IBF 1 IBF 4 IBF 3 IBF 2 Estimator 1 Attempt to subtract & decode IBF’s at each level. If level k decodes, then return: 2 k x (the number of ID’s recovered) … IBF 1 IBF 4 IBF 3 IBF 2 Estimator 2 … Decode Host 1 Host 2

30. 4x Strata Estimator 30 IBF 1 IBF 4 IBF 3 IBF 2 Estimator 1 Attempt to subtract & decode IBF’s at each level. If level k decodes, then return: 2 k x (the number of ID’s recovered) … IBF 1 IBF 4 IBF 3 IBF 2 Estimator 2 … Decode Host 1 Host 2 What about the other strata?

31. 2x Strata Estimator IBF 1 IBF 4 IBF 3 IBF 2 Estimator 1 … IBF 1 IBF 4 IBF 3 IBF 2 Estimator 2 … Decode Host 1 Host 2 Host 1 31 Observation: Extra partitions hold useful data Sum elements from all decoded strata & return: 2 (k-1) x (the number of ID’s recovered) Decode Host 1 Host 2 …

32. Estimation Accuracy 32 Strata good for small differences. Min-Wise good for large differences. Average Estimation Error (15.3 KBytes) Set Difference Relative Error in Estimation (%)

33. Hybrid Estimator 33 IBF 1 IBF 4 IBF 3 IBF 2 Strata Combine Strata and Min-Wise Estimators. – Use IBF Stratas for small differences. – Use Min-Wise for large differences. … IBF 1 Min-Wise IBF 2 Hybrid IBF 3

34. Hybrid Estimator Accuracy 34 Hybrid matches Strata for small differences. Converges with Min-wise for large differences Set Difference Average Estimation Error (15.3 KBytes) Relative Error in Estimation (%)

35. Application: KeyDiff Service Promising Applications: – File Synchronization – P2P file sharing – Failure Recovery Key Service Application Add( key ) Remove( key ) Diff( host1, host2 ) 35

36. Difference Digests Summary Strata & Hybrid Estimators – Estimate the size of the Set Difference. – For 100K sets, 15KB estimator has <15% error – O(log n) communication, O(log n) computation. Invertible Bloom Filter – Identifies all ID’s in the Set Difference. – 16 to 28 Bytes per ID in Set Difference. – O(d) communication, O(n+d) computation. Implemented in KeyDiff Service 36

37. Conclusions: Got Diffs? New randomized algorithm (difference digests) for set difference or insertion/deletion coding Could it be useful for your system? Need: – Large but roughly equal size sets – Small set differences (less than 10% of set size) 37

38. 38

39. Extra Slides 39

40. Comparison to Logs IBF work with no prior context. Logs work with prior context, BUT – Redundant information when sync’ing with multiple parties. – Logging must be built into system for each write. – Logging add overhead at runtime. – Logging requires non-volatile storage. Often not present in network devices. 40 IBF’s may out-perform logs when: Synchronizing multiple parties Synchronizations happen infrequently IBF’s may out-perform logs when: Synchronizing multiple parties Synchronizations happen infrequently