1. Hybrid Indexes Reducing the Storage Overhead of
Main-Memory OLTP Databases with
Hybrid Indexes
Huanchen Zhang
Co-authors:
David G. Andersen
Andrew Pavlo
Michael Kaminsky
Lin Ma
Rui Shen

2. You are running out of memory2

3. You are running out of memory2

4. You are running out of memory?
Buy more
You are running out of memory 2

5. Transactions Executed
TPC-C on
-Store
Memory Limit = 5GB
60K
Throughput
20K 2M 4M 6M 8M
10M
Memory (GB)
Disk tuples
In-memory tuples
Indexes 4 8
Transactions Executed 3

6. 4

7. Use memory more efficiently
The better way:
Use memory more efficiently 5

8. 34% 41% 18% 58% 55% 34% LARGE Indexes are Hybrid Index TPC-C Voter
Benchmark
% space for index
58%
TPC-C
55%
Voter
34%
Articles 6

9. 30 – 70% Our Contributions The hybrid index architecture
The Dual-Stage Transformation
Applied to 4 index structures
B+tree
Masstree
Skip List
Adaptive Radix Tree (ART)
Performance
Space
30 – 70% 7

10. Did we solve this problem?
60K
20K 2M 4M 6M 8M
10M
TPC-C on
-Store
Throughput (txn/s)
Stay tuned
Transactions Executed 8

11. How do hybrid indexes achieve
memory savings ?
Static 9

12. Hybrid Index: a dual-stage architecture
dynamic stage
static stage 10

13. Inserts are batched in the dynamic stage
write
merge
dynamic stage
static stage 11

14. Reads search the stages in order
dynamic stage
static stage 12

15. A Bloom filter improves read performance
dynamic stage
static stage 13

16. ~ ~ ~ ~ Memory-efficient Skew-aware read read write merge
dynamic stage
static stage 14

17. The Dual-Stage Transformation
merge
dynamic stage
static stage 15

18. The Dual-Stage Transformation
merge
dynamic stage
static stage 15

19. The Dynamic-to-Static Rules
Compaction
Reduction
Compression 16

20. The Dynamic-to-Static Rules
Compaction
Reduction
Compression 16

21. 4 2 4 6 8 10 1 2 a b 3 4 c d 5 e f 5 6 g h 7 8 i j 9 10 k l 11 12 m n 17

22. Compaction: minimize # of memory blocks4 2 4 6 8 10 1 2 a b 3 c 4 d 5 5 5 6 h 7 i 8 j 9 k 10 l 11 m 12 n e f g 17

23. Compaction: minimize # of memory blocks1 2 3 a b c 4 5 6 d h 7 8 9 i j k 10 11 12 l m n e f g 17

24. Reduction: minimize structural overhead3 6 9 1 2 3 a b c 4 5 6 d h 7 8 9 i j k 10 11 12 l m n e f g 18

25. Reduction: minimize structural overhead3 6 9 1 2 3 a b c 4 5 6 d h 7 8 9 i j k 10 11 12 l m n e f g 18

26. Reduction: minimize structural overhead2 4 1 a b 6 8 10 3 c d 5 e f g h 7 i j 9 k l 11 12 m n 3 6 9 1 2 3 a b c 4 5 6 d h 7 8 9 i j k 10 11 12 l m n e f g 18

27. The Dual-Stage Transformation
merge
dynamic stage
static stage 19

28. The Dual-Stage Transformation
Merge Questions:
Partial?
When?
Blocking? ?
merge
dynamic stage
static stage 19

29. Did we solve this problem?
B+tree
60K
20K 2M 4M 6M 8M
10M
TPC-C on
-Store
Throughput (txn/s)
Transactions Executed 20

30. Yes, we improved the DBMS’s capacity!
B+tree
60K
20K 2M 4M 6M 8M
10M
TPC-C on
-Store
Throughput (txn/s)
60K
20K
Hybrid
Transactions Executed 20

31. Transactions Executed
B+tree
60K
20K
Throughput (txn/s)
Hybrid
60K
20K
TPC-C on
-Store 2M 4M 6M 8M
10M 8
B+tree
Disk tuples 4
In-memory tuples
Indexes
Memory (GB) 8
Hybrid 4
Transactions Executed 21

32. Transactions Executed
B+tree
60K
20K
Throughput (txn/s)
Hybrid
60K
20K
TPC-C on
-Store 2M 4M 6M 8M
10M 8
B+tree
Disk tuples 4
In-memory tuples
Indexes
Memory (GB) 8
Hybrid 4
Transactions Executed 21

33. Transactions Executed
B+tree
60K
20K
Throughput (txn/s)
Hybrid
60K
20K
TPC-C on
-Store 2M 4M 6M 8M
10M 8
B+tree
Disk tuples 4
In-memory tuples
Indexes
Memory (GB) 8
Hybrid 4
Transactions Executed 21

34. Transactions Executed
B+tree
60K
20K
Throughput (txn/s)
Hybrid
60K
20K
TPC-C on
-Store 2M 4M 6M 8M
10M 8
B+tree
Disk tuples 4
In-memory tuples
Indexes
Memory (GB) 8
Hybrid 4
Transactions Executed 21

35. Transactions Executed
B+tree
60K
20K
Throughput (txn/s)
Hybrid
60K
20K
TPC-C on
-Store 2M 4M 6M 8M
10M 8
B+tree
Disk tuples 4
In-memory tuples
Indexes
Memory (GB) 8
Hybrid 4
Transactions Executed 21

36. Transactions Executed
Take Away:
Larger working set in memory
Higher throughput
Memory saved by indexes
B+tree
60K
20K
Throughput (txn/s)
Hybrid
60K
20K
TPC-C on
-Store 2M 4M 6M 8M
10M 8
B+tree
Disk tuples 4
In-memory tuples
Indexes
Memory (GB) 8
Hybrid 4
Transactions Executed 21

37. BEGINNING
This is just the 22

38. The hybrid index architecture GENERAL
Conclusions
The hybrid index architecture
GENERAL
The Dual-Stage Transformation
PRACTICAL
Applied to 4 index structures
USEFUL
B+tree
Masstree
Skip List
Adaptive Radix Tree (ART) 23

39. Toll-Free Hotline:
CMUDB