NVWAL: Exploiting NVRAM in Write-Ahead Logging ASPLOS’16, NVWAL: Exploiting NVRAM in Write-Ahead Logging Wook-Hee Kim1, Jinwoong Kim1, Woongki Baek1, Beomseok Nam1, Youjip Won2 1 UNIST (Ulsan National Institute of Science and Technology) 2 Hanyang University

Outline Motivation IO Bottleneck in SQLite NVWAL: Write-Ahead-Logging on NVRAM Byte-granularity differential logging User-level NVRAM management for WAL Transaction-aware lazy synchronization Evaluation Conclusion SQLite can be insensitive to NVRAM write latency. ASPLOS '16, Atlanta, GA.

SQLite data ASPLOS '16, Atlanta, GA.

I/O Bottleneck in Mobile device Small I/O (a few bytes) SQLite Database Journaling & Database page granularity I/O File System Block granularity I/O & File system Journaling Slow storage ASPLOS '16, Atlanta, GA.

How? NVRAM Storage Memory NVRAM Fast Persistence Volatile Cheap Byte-addressable How? ASPLOS '16, Atlanta, GA.

NVWAL (NVRAM Write-Ahead Logging) Byte-granularity differential logging User-level NVRAM management for WAL Transaction-aware lazy synchronization NVRAM Heap Manager DB H/W OS ASPLOS '16, Atlanta, GA.

Differential Logging ASPLOS '16, Atlanta, GA.

Write-Ahead Logging DRAM Buffer Volatile Commit! Non-Volatile ^ WAL File Flash memory Database File ASPLOS '16, Atlanta, GA.

Write-Ahead Logging in NVRAM Commit v v DRAM Buffer NVRAM Log Flash memory Database File ASPLOS '16, Atlanta, GA.

Differential Logging Differential Logging reduces WAL Header Wal frame Wal frame header Unused Wal frame Wal frame header Unused Commit! Wal frame Wal frame header Unused DRAM Buffer Wal frame Wal frame header Unused Differential Logging reduces up to 84% of unnecessary I/O. NVRAM Log ASPLOS '16, Atlanta, GA.

User-level Heap management ASPLOS '16, Atlanta, GA.

NVRAM Block Management DRAM Buffer User level Commit Kernel level nv_malloc() overhead Wal frame Wal frame header nv_malloc() overhead Wal frame Wal frame header nv_malloc() overhead Wal frame Wal frame header nv_malloc() overhead Wal frame Wal frame header NVRAM Log ASPLOS '16, Atlanta, GA.

NVRAM User-level management : in-use p : pending f : free Metadata u f DRAM buffer Kernel space Metadata block WAL Header Wal frame header Commit! Wal frame (Page 3) Wal frame header Wal frame (Page 7) User Space NVRAM Heap Unused next ASPLOS '16, Atlanta, GA.

NVRAM User-level management : in-use p : pending f : free Metadata u p f DRAM buffer Kernel space Metadata block WAL Header Unused Wal frame header Commit! Wal frame (Page 3) b = nv_pre_malloc() is called ! Wal frame header Wal frame (Page 7) User Space NVRAM Heap Unused next next ASPLOS '16, Atlanta, GA.

NVRAM User-level management : in-use p : pending f : free Metadata u f DRAM buffer Kernel space Metadata block WAL Header Unused Wal frame header Wal frame (Page 3) Wal frame header Wal frame (Page 5) Wal frame header Commit! Wal frame (Page 3) Wal frame (Page 8) Wal frame header Unused nv_set_flag(b,in-use) is called ! Wal frame header Wal frame (Page 7) User Space NVRAM Heap Unused next next ASPLOS '16, Atlanta, GA.

Recovery in NVWAL (1) – free in-use free WAL Header Unused Next frame Wal frame header Wal frame (Page 1) Wal frame header Wal frame (Page 7) Unused next ASPLOS '16, Atlanta, GA.

Recovery in NVWAL (2) – pending in-use free pending WAL Header Unused Next frame Wal frame header Wal frame (Page 1) Wal frame header Wal frame (Page 7) Recovery process reverts ‘pending’ to ‘free’ Unused next ASPLOS '16, Atlanta, GA.

Recovery in NVWAL (3) – pending in-use free pending WAL Header Unused Wal frame header Wal frame (Page 1) Wal frame header Wal frame (Page 7) Recovery process deletes pointers to pending blocks Unused next ASPLOS '16, Atlanta, GA.

Recovery in NVWAL (4) in-use pending WAL Header Wal frame header Wal frame (Page 3) Unused Wal frame header Wal frame (Page 1) Wal frame header Wal frame (Page 7) Normal WAL recovery Unused next ASPLOS '16, Atlanta, GA.

Persistency guarantee

Persistency guarantee in Flash W write() F fsync() Insert data W F W F ASPLOS '16, Atlanta, GA.

Eager Synchronization in NVRAM memcpy mb memory barrier CL flush cache line flush Insert data pb persist barrier CL flush mb Enforce ordering CL flush mb Enforce ordering CL flush mb Enforce ordering m m pb C pb ASPLOS '16, Atlanta, GA.

Transaction-Aware Persistency Guarantee in NVRAM memcpy mb memory barrier CL flush cache line flush Insert data pb persist barrier Logging phases Commit phases m mb CL flush mb m mb CL flush mb pb C mb CL flush mb pb ASPLOS '16, Atlanta, GA.

Asynchronous Commit in NVRAM pb memcpy persist barrier mb Chk sum checksum memory barrier CL flush cache line flush Insert data m m mb CL flush CL flush mb Chk sum pb C mb CL flush mb pb ASPLOS '16, Atlanta, GA.

Implement NVWAL in SQLite 3.7.11 Evaluation Implement NVWAL in SQLite 3.7.11 Used in Android 4.4 Tuna NVRAM emulation board with ARM Cortex-A9 DDR3-SDRAM DRAM DDR3-SDRAM NVRAM (Xilinx Zynq SOC) Nexus5 2.26 GHz Snapdragon 800 processor DDR memory we assume that specific address range of DRAM is NVRAM NVRAM latency is emulated by nop operations. Performance Analysis Tools Mobibench ASPLOS '16, Atlanta, GA.

Overhead of Ordering Constraints Memory fence Cache line flush Cache line flush + mfence memcpy Lazy synchronization eliminates up to 23% of persistency overhead. ASPLOS '16, Atlanta, GA.

Differential Logging and I/O Up to 84% of I/O reduced Differential logging eliminates up to 84% of the unnecessary I/O. ASPLOS '16, Atlanta, GA.

Transaction Throughput and NVRAM Latency insensitive 37% improvement 6% improvement 28% improvement User-Level Heap improves 6% of performance. Differential logging yields up to 28% higher throughput. Combining all, we can get up to 37% higher performance. ASPLOS '16, Atlanta, GA.

Transaction Throughput of NVWAL on Nexus5 10x faster Combining all of optimization performs at least 10 times faster when NVRAM latency is smaller than 3us. With our optimization, NVWAL shows performance similar to that of WAL on flash memory when the write latency is set to 230 usec. ASPLOS '16, Atlanta, GA.

Conclusion Strict ordering of memory writes causes unnecessary overhead. Transaction-aware lazy synchronization Leveraging byte-addressability of NVRAM Byte-granularity differential logging User-level NVRAM heap manager Via the optimizations, we make application performance insensitive to the NVRAM write latency. 400 nsec → 1900 nsec NVRAM latency results in only 4% performance degradation ASPLOS '16, Atlanta, GA.

Thank You ASPLOS '16, Atlanta, GA.