Download presentation
Presentation is loading. Please wait.
Published byFrida Graf Modified over 6 years ago
1
PASTE: A Networking API for Non-Volatile Main Memory
Michio Honda (NEC Laboratories Europe) Lars Eggert (NetApp) Douglas Santry (NetApp) 99, Prague May 22th 2017 More details at our HotNets paper: © 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only
2
What are implications for networking?
Motivation Non-Volatile Main Memories (NVMMs) Persistent Byte-addressable Low latency 10s-1000s of ns A lot of work in databases and file systems NVHeap [ASPLOS'11], NOVA [FAST'16], NVWal [ASPLOS'16], HiKV [ATC'17] etc What are implications for networking? © 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only
3
Review: Today's Stack with NVMM
© 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only
4
Review: Today's Stack with NVMM – How data move from the network to the storage
© 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only
5
Case Study: Careful Data Transfer
Server persists client’s request prior to acknowledgment e.g., 1KB commit:
6
Case Study: Careful Data Transfer
Server persists client’s request prior to acknowledgment e.g., 1KB commit: (4) write()/fsync() or memcpy()/msync() (3) read() App (5) Network stack (2) Storage stack DRAM client NIC SSD/ DIsk (1) 2030 us Networking (w/o step (4)) takes 40 us © 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only
7
Case Study: Careful Data Transfer
Server persists client’s request prior to acknowledgment e.g., 1KB commit: write()/fsync() or memcpy()/msync() (3) read() App (4) Network stack Storage stack (2) DRAM NVMM (5) client NIC Emulated using a reserved region of DRAM (1) us Networking takes 40 us This 2 us is not small
8
Case Study: Careful Data Transfer
Parallel requests are serialized on each core 33 % throughput decrease, 50 % latency increase
9
We must avoid data copy! Data Copies Matter Cache Misses
Persisting data (e.g., to a log) always happens to a different destination app buffer kernel buffer read() log file (mmap()-ed) memcpy() Overall cache misses Major Contributor Networking only % net_rx_action() (84%) Networking + NVMM (read() + memcpy() + msync()) % memcpy() (98%) Reported by Linux perf We must avoid data copy! © 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only
10
Packet Store (PASTE) Overview
Packet buffers on a named NVMM region DMA to NVMM Zero-copy APIs (4) metadata only (e.g., buffer index) App (3) Network stack Storage stack NVMM /mnt/nvmm/pktbufs /mnt/pmem/appmd (2) client NIC (1)
11
Fast Persistent Write with PASTE
Application (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) User mmap() netmap API Kernel metadata header TCP/IP input and output /mnt/nvmm/pktbufs buf_ofs: 123 metadata entries buf_idx off len 100 1135 1 932 3 1024 NIC ring packet buffers (static) /mnt/nvmm/myapp_metadata /mnt/nvmm/pktbufs
12
Fast Persistent Write with PASTE
Unread Application Read or written (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) User mmap() netmap API Kernel metadata header TCP/IP input and output /mnt/nvmm/pktbufs buf_ofs: 123 metadata entries buf_idx off len 100 1135 1 932 3 1024 NIC ring packet buffers (static) /mnt/nvmm/myapp_metadata /mnt/nvmm/pktbufs
13
Fast Persistent Write with PASTE
Unread Application Read or written (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) User mmap() netmap API Kernel metadata header TCP/IP input and output /mnt/nvmm/pktbufs buf_ofs: 123 metadata entries buf_idx off len 100 1135 1 932 3 1024 NIC ring packet buffers (static) /mnt/nvmm/myapp_metadata /mnt/nvmm/pktbufs
14
Fast Persistent Write with PASTE
Unread Application Read or written (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) User mmap() netmap API Kernel metadata header TCP/IP input and output /mnt/nvmm/pktbufs buf_ofs: 123 metadata entries buf_idx off len 100 1135 1 932 3 1024 NIC ring packet buffers (static) /mnt/nvmm/myapp_metadata /mnt/nvmm/pktbufs
15
Fast Persistent Write with PASTE
Unread Application Read or written (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) Flushed User mmap() netmap API Kernel metadata header TCP/IP input and output /mnt/nvmm/pktbufs buf_ofs: 123 metadata entries buf_idx off len 100 1135 1 932 3 1024 NIC ring packet buffers (static) /mnt/nvmm/myapp_metadata /mnt/nvmm/pktbufs DMA is performed to L3 cache (DDIO)
16
Fast Persistent Write with PASTE
Unread Application Read or written (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) Flushed User mmap() netmap API Kernel metadata header TCP/IP input and output /mnt/nvmm/pktbufs buf_ofs: 123 metadata entries buf_idx off len 100 1135 1 932 3 1024 NIC ring packet buffers (static) /mnt/nvmm/myapp_metadata /mnt/nvmm/pktbufs DMA is performed to L3 cache (DDIO)
17
Fast Persistent Write with PASTE
Unread Application Read or written (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) Flushed User mmap() netmap API Kernel metadata header TCP/IP input and output /mnt/nvmm/pktbufs buf_ofs: 123 metadata entries buf_idx off len 100 1135 1 932 3 1024 NIC ring packet buffers (static) /mnt/nvmm/myapp_metadata /mnt/nvmm/pktbufs DMA is performed to L3 cache (DDIO)
18
Fast Persistent Write with PASTE
Unread Application Read or written (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) Flushed User mmap() netmap API Kernel metadata header TCP/IP input and output /mnt/nvmm/pktbufs buf_ofs: 123 metadata entries buf_idx off len 100 1135 1 932 3 1024 NIC ring packet buffers (static) /mnt/nvmm/myapp_metadata /mnt/nvmm/pktbufs DMA is performed to L3 cache (DDIO)
19
Fast Persistent Write with PASTE
Unread Application Read or written (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) Flushed User mmap() netmap API Kernel metadata header TCP/IP input and output /mnt/nvmm/pktbufs buf_ofs: 123 metadata entries buf_idx off len 100 1135 1 932 3 1024 NIC ring packet buffers (static) /mnt/nvmm/myapp_metadata /mnt/nvmm/pktbufs DMA is performed to L3 cache (DDIO) © 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only
20
Unnecessary data is not flushed to DIMM
Fast and Selective Persistent Write with PASTE Idempotent request Unread Application Read or written (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) Flushed User mmap() netmap API Kernel metadata header TCP/IP input and output /mnt/nvmm/pktbufs buf_ofs: 123 metadata entries buf_idx off len 100 1135 1 932 3 1024 NIC ring packet buffers (static) /mnt/nvmm/myapp_metadata /mnt/nvmm/pktbufs DMA is performed to L3 cache (DDIO) Unnecessary data is not flushed to DIMM © 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only
21
Unnecessary data is not flushed to DIMM
Fast and Selective Persistent Write with PASTE Unread Application Read or written (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) Flushed User mmap() netmap API Kernel metadata header TCP/IP input and output /mnt/nvmm/pktbufs buf_ofs: 123 metadata entries buf_idx off len 100 1135 1 932 3 1024 NIC ring packet buffers (static) /mnt/nvmm/myapp_metadata /mnt/nvmm/pktbufs DMA is performed to L3 cache (DDIO) Unnecessary data is not flushed to DIMM © 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only
22
Unnecessary data is not flushed to DIMM
Fast and Selective Persistent Write with PASTE Unread Application Read or written (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) Flushed User mmap() netmap API Kernel metadata header TCP/IP input and output /mnt/nvmm/pktbufs buf_ofs: 123 metadata entries buf_idx off len 100 1135 1 932 3 1024 NIC ring packet buffers (static) /mnt/nvmm/myapp_metadata /mnt/nvmm/pktbufs DMA is performed to L3 cache (DDIO) Unnecessary data is not flushed to DIMM © 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only
23
Unnecessary data is not flushed to DIMM
Fast and Selective Persistent Write with PASTE Unread Application Read or written (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) Flushed User mmap() netmap API Kernel metadata header TCP/IP input and output /mnt/nvmm/pktbufs buf_ofs: 123 metadata entries buf_idx off len 100 1135 1 932 3 1024 NIC ring packet buffers (static) /mnt/nvmm/myapp_metadata /mnt/nvmm/pktbufs DMA is performed to L3 cache (DDIO) Unnecessary data is not flushed to DIMM © 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only
24
Extend the netmap framework
Preliminary Results Implementation Extend the netmap framework TCP/IP is also supported 10-88 % throughput increase, 9-46 % latency reduction © 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only
25
Related Work Enhanced network stacks MegaPipe (OSDI’12), Stackmap (ATC’16), Fastsocket (ASPLOS’16) IX and Arrakis (OSDI’14), mTCP (NSDI’13), Sandstorm (SIGCOMM’14), MICA (NSDI’14) NVMM filesystems BPFS (SOSP’09), NOVA (FAST’15) NVMM databases NVWAL (ASPLOS’15), REWIND (VLDB’15), NV-Tree (FAST’15) No NVMM aware No networking aware
26
Networking APIs are now a bottleneck for durably storing data
Conclusion Networking APIs are now a bottleneck for durably storing data Ongoing work Brushing up APIs Smooth integration with netmap API Forming a filesystem with packet buffers Fast data movement between files Use case applications App Netmap API Network Stack Filesystem NVMM NIC
Similar presentations
© 2024 SlidePlayer.com. Inc.
All rights reserved.