1. Fast Accesses to Big Data in Memory and Storage Systems
Xiaodong Zhang
Ohio State University

2. Fast Data Accesses in In-Memory Key-Value Stores

3. Information processing without tables: Key Value Stores
A simple but effective method in data processing where a data record (or a value) is stored and retrieved with its associated key
variable types and lengths of record (value)
simple or no schema
easy software development for many applications
Key-value stores have been widely used in production systems:

4. Simple and Easy Interfaces of Key-Value Stores
value = get (key)
put (key, value)
get (key) 38
John_age …
Variable-length
keys & values
Index
Client

5. Key-Value Store: A Data Model Supporting Scale-out
Command
Operation
GET(key)
Read value
SET(key, value)
Write a KV item
DEL(key)
Delete a KV item
Hash (Key)  Server ID
 finer control over availability.
, such as relational databases and file systems can be , , demand infrastructures for data collecting, organizing, and management
It is horizontally scalable,
It removes scalability bottleneck constraining distributed databases and file systems .
It can be potentially of (very) high performance.
Functionality-richer systems can be build above it.
It is horizontally scalable,
It can be potentially of (very) high performance.
Data Servers 5

6. Workflow of an in-memory Key-Value Store
Network Processing
Memory Management
Index Operations
Access Value

7. Workflow of a Typical Key-Value Store
DELETE
SET
GET
Extract Keys
TCP/IP Processing
Request Parsing
Extract Key&Value
Network Processing
Delete from Index
Insert into Index
Search in Index
Index Operations
Memory
Full
Not Full
Evict
Allocate
Memory Management
Read & Send Value
Access Value

8. Where does time go in KV-Store MICA [NSDI’14]
Network processing &
Memory Management
Index Operations
Access Value
Index operation becomes one of the major bottlenecks

9. Data Workflow of Key-Value Stores
Query
Network Processing &
Memory Management
Index Operation
Hash Table
Random
Memory Accesses
Access Value

10. Random Memory Accesses of Indexing are Expensive
Sequential memory access
Random memory access 16 3 7 2
CPU: Intel Xeon E5-2650v2
Memory: 1600 MHz DDR3

11. Reason: long latency comes from row buffer misses
Processor
Bus bandwidth time
Row Buffer
Col. Access (cache line)
Row Access
Precharge
DRAM
Latency
DRAM Core
Row buffer hits: repeated accesses in the same page
Row buffer misses: random accesses in different pages

12. Inabilities for CPUs to accelerate random accesses
Do something for Index Operations (random memory accesses)
Caching
Working set is large (100 GB), CPU cache is small (10 MB)
Prefetching
Hard to predict next memory address
Multithreading
Limited number of hardware supported threads
Limited by # of Miss Status Holding Registers (MSHRs) for large volumes

13. High Throughput is Desirable for Processing Big Data
It measures the capability of a system to process a growing amount requests on an increasingly large data set
Existing KV-Store throughput (MOPS), aiming for low latency
MICA (CMU research prototype): 71
MassTree (MIT research prototype): 14
RamCloud (Stanford research prototype): 6
Memcached: (open source software): 1.3
Key-Value Store latency (millisecond)
< 1ms is acceptable, e.g. facebook, Amazon, …
Our goal: to maximize throughput subject to acceptable latency

14. To accelerate it by GPUs
Mega-KV addresses two issues: large number of requests and random access delay
Network Processing
Memory Management
Access Value
Index Operation
User space I/O, Multiget, UDP
Bitmap, Optimistic concurrent access
To accelerate it by GPUs
Prefetch

15. CPU vs. GPU Massive ALUs Control Cache Intel Xeon E5-2650v2:
2.3 billion Transistors
8 Cores
59.7 GB/s memory bandwidth
Nvidia GTX 780:
7 billion Transistors
2,304 Cores
288.4 GB/s memory bandwidth

16. Two Advantages of GPUs for Key-Value Stores
Massive Parallel Processing Units in GPU
To process a large number of simple and independent memory access operations
Massively Hiding Random Memory Access Latency
GPU can effectively hide memory latency with massive hardware threads and zero-overhead thread scheduling by hardware support
We find that GPUs have several advantages for kv stores. First, it has massive processing units. The operations in kv stores are simple independent memory accesses, while the thousands of cores in GPUs are ideal for massive parallel processing. Second, it has an ability of massively hiding memory access latency. As we have shown previously, lots of random memory accesses are involved in index operations, and GPUs can effectively hide them with …

17. Two Unique Advantages of GPUs for Key-Value Stores
Massive GPU Processing Units
To process a large number of simple independent memory access operations
Massively Hiding Random Memory Access Latency
GPUs can effectively hide memory latency with massive hardware threads and zero-overhead thread scheduling by hardware support
memory request issued,
switch to another thread
memory request issued,
switch to another thread
cache
miss
cache
miss …
Thread A …
Thread B …
Thread C
GPU Core
Instruction Buffer

18. Mega-KV System Framework
Pre-Processing
GPU Processing
Post-Processing

19. Mega-KV System Framework
Network processing,
Memory management
Index Operations
Read & Send Value
Pre-Processing
GPU Processing
Post-Processing
Request TX

20. Mega-KV System Framework
Network processing,
Memory management
Pre-Processing

21. Mega-KV System Framework
Network processing,
Memory management
Pre-Processing
Parallel Processing
in GPU

22. Mega-KV System Framework
Network processing,
Memory management
Pre-Processing
Post-Processing
Parallel Processing
in GPU
Read & Send Value

23. Challenges of Offloading Index Operations to GPUs
GPU’s memory capacity is small: ~10 GB
Keys may take hundreds of GBs
Low PCIe bandwidth
PCIe is generally the bottleneck of GPU if large bulk of data needs to be transferred
Handling variable-length input is inefficient for GPUs
Search along the input buffer or transfer another offset buffer
Variable-length string comparison
Linked list in GPUs is inefficient
The locks to insert/delete linked items would hinder parallelism
more random memory accesses

24. Our Solutions Input data (Keys) Index Compress key C C
Compressed fixed-length signatures
key
GPU optimized cuckoo hash table that
stores key signatures and value locations
Address challenges 2, 3 (PCIe bandwidth and variable length data)
Address challenges 1, 3, and 4
(GPU memory capacity and variable length data)

25. High Throughput Comes from large Batch Size subject to acceptable latencies

26. High Throughput Comes from large Batch Size subject to acceptable latencies

27. High Throughput Comes from large Batch Size subject to acceptable latencies

28. 系统的可扩展性是一个延迟和吞吐量平衡的结果
最佳点 延迟
吞吐量
并发进程的数量

29. A Significant Execution Time Reduction by Fast Indexing
Access Value
Index Operations
Network Processing &
Memory Management
Execution time of the query
in CPU-based KV
Execution time of the query
in Mega-KV

30. Reaching a Record High Throughput
2.1x
1.9x
2.8x
We measure the throughput with 95% GET and 5% SET queries under uniform and skewed distribution. With 8B key and 8B value, we can get up to 160MOPS throughput. For all data sets, we are about 2 times as fast as the fastest cpu-based kv stores.

31. Low and Acceptable Latency
Compared with
Facebook
1,200 (95th)
300 (50th)
95th: 390
50th: 256
This graph shows the distribution of MegaKV’s round trip latency under its maximum throughput 160 MOPS. The 50th and 95th percentile latency are 256 and 390 microseconds respectively. As that in Facebook is reported to be 300 and 1200 ms. Therefore, our latency is far below the requirement of production systems.

32. The Mega-KV software is open to the public (VLDB ‘15)
Mega-KV is open source at

33. Summary Restructuring basic data structures to respond big data
Reserving buffer cache locality in fast writes: LSbM-tree
Balancing network transfer and I/O bandwidth: RCFile/ORC
Accelerating indexing operations by GPU: Mega-KV