1. Improving Deduplication by Accelerating Remainder Calculations
Michael Hirsch
Shmuel Tomi Klein
Yair Toaff
ISRAEL

2. Outline Background and motivation Background and motivation
Hierarchical evaluation of remainder
Adapting the hierarchical method
Experimental results
Avoiding overflows
Hierarchical evaluation of remainder
Avoiding overflows
Adapting the hierarchical method
Experimental results

3. Background and motivation
Rabin Karp pattern matching
Rolling hash
Need ?
Only !
But needs

4. Deduplication Deduplication Deduplication Deduplication Deduplication
Background and motivation
Deduplication
Deduplication
Deduplication
Deduplication
Other applications:
Deduplication
CAS: use signatures to find identical chunks
IBM extend to find similar chunks
If chunks are similar
more finegrained comparison

5. Background and motivation
reference
Store reference hashes – search for version hashes
version

6. Background and motivation
To speed up use local parallelism
Hash: remainder modulo P
Use interchangeably:
ASCII
ASCII - Binary
Decimal value
ABC
4,276,803

7. Hierarchical evaluation of remainder
d bits
P prime close to 2r
m processors

8. Hierarchical evaluation of remainder
for to
d bits

9. Hierarchical evaluation of remainder
. . .
Step 0 1 2 3 4 5 6 7
A[7]
A[6]
A[5]
A[4]
A[3]
A[2]
A[1]
A[0]
A[m-1]
m - 1
. . . 1 3 5 7
Step 1
m - 1
. . . 3 7
Step 2
m - 1
. . . 7
Step 3
Step log m
m - 1
steps
parallel steps

10. Hierarchical evaluation of remainder10

11. Avoiding overflows For d=64 need 128-bit arithmetic
d=32 does not help, because R x C
23 bits

12. Avoiding overflows Theorem:
The value of R fits into 56 bits at the end of each iteration

13. Adapting the hierarchical method
more than 64 bits

14. Adapting the hierarchical method

15. Adapting the hierarchical method55 64
110
119
127
BITS:
9+12
55+6 55

16. Experimental results

17. Thank you !