Presentation is loading. Please wait.

Presentation is loading. Please wait.

PASTE: A Networking API for Non-Volatile Main Memory

Published byFrida Graf Modified over 6 years ago

Similar presentations

Presentation on theme: "PASTE: A Networking API for Non-Volatile Main Memory"— Presentation transcript:

1 PASTE: A Networking API for Non-Volatile Main Memory
Michio Honda (NEC Laboratories Europe) Lars Eggert (NetApp) Douglas Santry (NetApp) 99, Prague May 22th 2017 More details at our HotNets paper:  © 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

2 What are implications for networking?
Motivation Non-Volatile Main Memories (NVMMs) Persistent Byte-addressable Low latency 10s-1000s of ns A lot of work in databases and file systems NVHeap [ASPLOS'11], NOVA [FAST'16], NVWal [ASPLOS'16], HiKV [ATC'17] etc What are implications for networking? © 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

3 Review: Today's Stack with NVMM
© 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

4 Review: Today's Stack with NVMM – How data move from the network to the storage
© 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

5 Case Study: Careful Data Transfer
Server persists client’s request prior to acknowledgment e.g., 1KB commit:

6 Case Study: Careful Data Transfer
Server persists client’s request prior to acknowledgment e.g., 1KB commit: (4) write()/fsync() or memcpy()/msync() (3) read() App (5) Network stack (2) Storage stack DRAM client NIC SSD/ DIsk (1) 2030 us Networking (w/o step (4)) takes 40 us © 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

7 Case Study: Careful Data Transfer
Server persists client’s request prior to acknowledgment e.g., 1KB commit: write()/fsync() or memcpy()/msync() (3) read() App (4) Network stack Storage stack (2) DRAM NVMM (5) client NIC Emulated using a reserved region of DRAM (1) us Networking takes 40 us This 2 us is not small

8 Case Study: Careful Data Transfer
Parallel requests are serialized on each core 33 % throughput decrease, 50 % latency increase

9 We must avoid data copy! Data Copies Matter Cache Misses
Persisting data (e.g., to a log) always happens to a different destination app buffer kernel buffer read() log file (mmap()-ed) memcpy() Overall cache misses Major Contributor Networking only % net_rx_action() (84%) Networking + NVMM (read() + memcpy() + msync()) % memcpy() (98%) Reported by Linux perf We must avoid data copy! © 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

10 Packet Store (PASTE) Overview
Packet buffers on a named NVMM region DMA to NVMM Zero-copy APIs (4) metadata only (e.g., buffer index) App (3) Network stack Storage stack NVMM /mnt/nvmm/pktbufs /mnt/pmem/appmd (2) client NIC (1)

11 Fast Persistent Write with PASTE
Application (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) User mmap() netmap API Kernel metadata header TCP/IP input and output /mnt/nvmm/pktbufs buf_ofs: 123 metadata entries buf_idx off len 100 1135 1 932 3 1024 NIC ring packet buffers (static) /mnt/nvmm/myapp_metadata /mnt/nvmm/pktbufs

12 Fast Persistent Write with PASTE
Unread Application Read or written (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) User mmap() netmap API Kernel metadata header TCP/IP input and output /mnt/nvmm/pktbufs buf_ofs: 123 metadata entries buf_idx off len 100 1135 1 932 3 1024 NIC ring packet buffers (static) /mnt/nvmm/myapp_metadata /mnt/nvmm/pktbufs

13 Fast Persistent Write with PASTE
Unread Application Read or written (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) User mmap() netmap API Kernel metadata header TCP/IP input and output /mnt/nvmm/pktbufs buf_ofs: 123 metadata entries buf_idx off len 100 1135 1 932 3 1024 NIC ring packet buffers (static) /mnt/nvmm/myapp_metadata /mnt/nvmm/pktbufs

14 Fast Persistent Write with PASTE
Unread Application Read or written (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) User mmap() netmap API Kernel metadata header TCP/IP input and output /mnt/nvmm/pktbufs buf_ofs: 123 metadata entries buf_idx off len 100 1135 1 932 3 1024 NIC ring packet buffers (static) /mnt/nvmm/myapp_metadata /mnt/nvmm/pktbufs

15 Fast Persistent Write with PASTE
Unread Application Read or written (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) Flushed User mmap() netmap API Kernel metadata header TCP/IP input and output /mnt/nvmm/pktbufs buf_ofs: 123 metadata entries buf_idx off len 100 1135 1 932 3 1024 NIC ring packet buffers (static) /mnt/nvmm/myapp_metadata /mnt/nvmm/pktbufs DMA is performed to L3 cache (DDIO)

16 Fast Persistent Write with PASTE
Unread Application Read or written (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) Flushed User mmap() netmap API Kernel metadata header TCP/IP input and output /mnt/nvmm/pktbufs buf_ofs: 123 metadata entries buf_idx off len 100 1135 1 932 3 1024 NIC ring packet buffers (static) /mnt/nvmm/myapp_metadata /mnt/nvmm/pktbufs DMA is performed to L3 cache (DDIO)

17 Fast Persistent Write with PASTE
Unread Application Read or written (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) Flushed User mmap() netmap API Kernel metadata header TCP/IP input and output /mnt/nvmm/pktbufs buf_ofs: 123 metadata entries buf_idx off len 100 1135 1 932 3 1024 NIC ring packet buffers (static) /mnt/nvmm/myapp_metadata /mnt/nvmm/pktbufs DMA is performed to L3 cache (DDIO)

18 Fast Persistent Write with PASTE
Unread Application Read or written (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) Flushed User mmap() netmap API Kernel metadata header TCP/IP input and output /mnt/nvmm/pktbufs buf_ofs: 123 metadata entries buf_idx off len 100 1135 1 932 3 1024 NIC ring packet buffers (static) /mnt/nvmm/myapp_metadata /mnt/nvmm/pktbufs DMA is performed to L3 cache (DDIO)

19 Fast Persistent Write with PASTE
Unread Application Read or written (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) Flushed User mmap() netmap API Kernel metadata header TCP/IP input and output /mnt/nvmm/pktbufs buf_ofs: 123 metadata entries buf_idx off len 100 1135 1 932 3 1024 NIC ring packet buffers (static) /mnt/nvmm/myapp_metadata /mnt/nvmm/pktbufs DMA is performed to L3 cache (DDIO) © 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

20 Unnecessary data is not flushed to DIMM
Fast and Selective Persistent Write with PASTE Idempotent request Unread Application Read or written (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) Flushed User mmap() netmap API Kernel metadata header TCP/IP input and output /mnt/nvmm/pktbufs buf_ofs: 123 metadata entries buf_idx off len 100 1135 1 932 3 1024 NIC ring packet buffers (static) /mnt/nvmm/myapp_metadata /mnt/nvmm/pktbufs DMA is performed to L3 cache (DDIO) Unnecessary data is not flushed to DIMM © 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

21 Unnecessary data is not flushed to DIMM
Fast and Selective Persistent Write with PASTE Unread Application Read or written (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) Flushed User mmap() netmap API Kernel metadata header TCP/IP input and output /mnt/nvmm/pktbufs buf_ofs: 123 metadata entries buf_idx off len 100 1135 1 932 3 1024 NIC ring packet buffers (static) /mnt/nvmm/myapp_metadata /mnt/nvmm/pktbufs DMA is performed to L3 cache (DDIO) Unnecessary data is not flushed to DIMM © 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

22 Unnecessary data is not flushed to DIMM
Fast and Selective Persistent Write with PASTE Unread Application Read or written (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) Flushed User mmap() netmap API Kernel metadata header TCP/IP input and output /mnt/nvmm/pktbufs buf_ofs: 123 metadata entries buf_idx off len 100 1135 1 932 3 1024 NIC ring packet buffers (static) /mnt/nvmm/myapp_metadata /mnt/nvmm/pktbufs DMA is performed to L3 cache (DDIO) Unnecessary data is not flushed to DIMM © 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

23 Unnecessary data is not flushed to DIMM
Fast and Selective Persistent Write with PASTE Unread Application Read or written (1) Read data (zero copy) (2) Write metadata entry (3) Flush (buffer and metadata) Flushed User mmap() netmap API Kernel metadata header TCP/IP input and output /mnt/nvmm/pktbufs buf_ofs: 123 metadata entries buf_idx off len 100 1135 1 932 3 1024 NIC ring packet buffers (static) /mnt/nvmm/myapp_metadata /mnt/nvmm/pktbufs DMA is performed to L3 cache (DDIO) Unnecessary data is not flushed to DIMM © 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

24 Extend the netmap framework
Preliminary Results Implementation Extend the netmap framework TCP/IP is also supported 10-88 % throughput increase, 9-46 % latency reduction © 2015 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only

25 Related Work Enhanced network stacks MegaPipe (OSDI’12), Stackmap (ATC’16), Fastsocket (ASPLOS’16) IX and Arrakis (OSDI’14), mTCP (NSDI’13), Sandstorm (SIGCOMM’14), MICA (NSDI’14) NVMM filesystems BPFS (SOSP’09), NOVA (FAST’15) NVMM databases NVWAL (ASPLOS’15), REWIND (VLDB’15), NV-Tree (FAST’15) No NVMM aware No networking aware

26 Networking APIs are now a bottleneck for durably storing data
Conclusion Networking APIs are now a bottleneck for durably storing data Ongoing work Brushing up APIs Smooth integration with netmap API  Forming a filesystem with packet buffers Fast data movement between files Use case applications App Netmap API Network Stack Filesystem NVMM NIC

Download ppt "PASTE: A Networking API for Non-Volatile Main Memory"

Similar presentations

Assessment of Data Path Implementations for Download and Streaming Pål Halvorsen 1,2, Tom Anders Dalseng 1 and Carsten Griwodz 1,2 1 Department of Informatics,

Assessment of Data Path Implementations for Download and Streaming Pål Halvorsen 1,2, Tom Anders Dalseng 1 and Carsten Griwodz 1,2 1 Department of Informatics,
Better I/O Through Byte-Addressable, Persistent Memory

Better I/O Through Byte-Addressable, Persistent Memory
Rethink the Sync Ed Nightingale Kaushik Veeraraghavan Peter Chen Jason Flinn University of Michigan.

Rethink the Sync Ed Nightingale Kaushik Veeraraghavan Peter Chen Jason Flinn University of Michigan.
1 Design and Implementation of A Content-aware Switch using A Network Processor Li Zhao, Yan Luo, Laxmi Bhuyan University of California, Riverside Ravi.

1 Design and Implementation of A Content-aware Switch using A Network Processor Li Zhao, Yan Luo, Laxmi Bhuyan University of California, Riverside Ravi.
RDMA ENABLED WEB SERVER Rajat Sharma. Objective  To implement a Web Server serving HTTP client requests through RDMA replacing the traditional TCP/IP.

RDMA ENABLED WEB SERVER Rajat Sharma. Objective  To implement a Web Server serving HTTP client requests through RDMA replacing the traditional TCP/IP.
COM S 614 Advanced Systems Novel Communications U-Net and Active Messages.

COM S 614 Advanced Systems Novel Communications U-Net and Active Messages.
Memory-Mapped Files & Unified VM System Vivek Pai.

Memory-Mapped Files & Unified VM System Vivek Pai.
Joel Coburn. , Trevor Bunker

Joel Coburn. , Trevor Bunker
I/O Systems ◦ Operating Systems ◦ CS550. Note:  Based on Operating Systems Concepts by Silberschatz, Galvin, and Gagne  Strongly recommended to read.

I/O Systems ◦ Operating Systems ◦ CS550. Note:  Based on Operating Systems Concepts by Silberschatz, Galvin, and Gagne  Strongly recommended to read.
Performance Tradeoffs for Static Allocation of Zero-Copy Buffers Pål Halvorsen, Espen Jorde, Karl-André Skevik, Vera Goebel, and Thomas Plagemann Institute.

Performance Tradeoffs for Static Allocation of Zero-Copy Buffers Pål Halvorsen, Espen Jorde, Karl-André Skevik, Vera Goebel, and Thomas Plagemann Institute.
Network Stack Specialization for Performance

Network Stack Specialization for Performance
RAMCloud: A Low-Latency Datacenter Storage System Ankita Kejriwal Stanford University (Joint work with Diego Ongaro, Ryan Stutsman, Steve Rumble, Mendel.

RAMCloud: A Low-Latency Datacenter Storage System Ankita Kejriwal Stanford University (Joint work with Diego Ongaro, Ryan Stutsman, Steve Rumble, Mendel.
Hardware Definitions –Port: Point of connection –Bus: Interface Daisy Chain (A=>B=>…=>X) Shared Direct Device Access –Controller: Device Electronics –Registers:

Hardware Definitions –Port: Point of connection –Bus: Interface Daisy Chain (A=>B=>…=>X) Shared Direct Device Access –Controller: Device Electronics –Registers:
RAMCloud: Low-latency DRAM-based storage Jonathan Ellithorpe, Arjun Gopalan, Ashish Gupta, Ankita Kejriwal, Collin Lee, Behnam Montazeri, Diego Ongaro,

RAMCloud: Low-latency DRAM-based storage Jonathan Ellithorpe, Arjun Gopalan, Ashish Gupta, Ankita Kejriwal, Collin Lee, Behnam Montazeri, Diego Ongaro,
Srihari Makineni & Ravi Iyer Communications Technology Lab

Srihari Makineni & Ravi Iyer Communications Technology Lab
1 File Systems: Consistency Issues. 2 File Systems: Consistency Issues File systems maintains many data structures  Free list/bit vector  Directories.

1 File Systems: Consistency Issues. 2 File Systems: Consistency Issues File systems maintains many data structures  Free list/bit vector  Directories.
Increasing Web Server Throughput with Network Interface Data Caching October 9, 2002 Hyong-youb Kim, Vijay S. Pai, and Scott Rixner Rice Computer Architecture.

Increasing Web Server Throughput with Network Interface Data Caching October 9, 2002 Hyong-youb Kim, Vijay S. Pai, and Scott Rixner Rice Computer Architecture.
1 Public DAFS Storage for High Performance Computing using MPI-I/O: Design and Experience Arkady Kanevsky & Peter Corbett Network Appliance Vijay Velusamy.

1 Public DAFS Storage for High Performance Computing using MPI-I/O: Design and Experience Arkady Kanevsky & Peter Corbett Network Appliance Vijay Velusamy.
Fast Crash Recovery in RAMCloud. Motivation The role of DRAM has been increasing – Facebook used 150TB of DRAM For 200TB of disk storage However, there.

Fast Crash Recovery in RAMCloud. Motivation The role of DRAM has been increasing – Facebook used 150TB of DRAM For 200TB of disk storage However, there.
Eduardo Gutarra Velez. Outline Distributed Filesystems Motivation Google Filesystem Architecture The Metadata Consistency Model File Mutation.

Eduardo Gutarra Velez. Outline Distributed Filesystems Motivation Google Filesystem Architecture The Metadata Consistency Model File Mutation.

Similar presentations

Ads by Google