Integrating DPDK/SPDK with storage application @ Vishnu Itta @ Mayank Patel Containerized Storage for Containers

Agenda Problem Statement Current state of legacy applications and its issues Possible solutions Integrating DPDK related libraries with legacy applications Approach of client/server model to use SPDK Future work

Problem Statement

State of legacy storage applications Highly depends on locks for synchronization Heavy system calls, context switches Dependency on kernel for scheduling Frequent memory allocations Dependency on locks and heavy page faults due to huge memory requirement Copy of data multiple times between userspace and device Leads to Doesn’t scale with current hardware trends Context switches IO latencies

Possible solutions lockless data structures Reactor pattern Usage of HUGE pages Integrating with userspace NVM driver from SPDK Leads to Scalability with CPUs Lesser context switches Better performance

Integrating DPDK related libraries with storage applications

Linking DPDK with application Shared library CONFIG_RTE_BUILD_SHARED_LIB=y Static library CONFIG_RTE_BUILD_SHARED_LIB=n Using rte.ext or rte. Makefiles from dpdk/mk For building application : rte.extapp.mk For shared libraries : rte.extlib.mk, rte.extshared.mk Fo building object files : rte.extobj.mk

Integration of DPDK libraries with Application libraries Usage of constructor in libraries Force linking of unused libraries with shared library LDFLAGS += --no-as-needed -lrte_mempool --as-needed rte_eal_init () - to initialize EAL - from legacy application

Memory allocation with rte_eal library Memory allocation from HUGE pages No overhead of address transformation at kernel Issues with this approach in legacy multi-threaded applications: Spinlock with multiple threads Dedicated core to threads

Cache base allocation with mempool library Allocator of fixed-size object Ring base handler for managing object block Each get/put request does CAS operations

Ring library to synchronize threads operations One thread to process operations Message passing between core thread to/from other threads through RING buffer Single CAS operation Issue Dedicated CPU core to thread But, is there a way to integrate SPDK with current state of legacy applications without much redesign?

Approach of client/server model to use SPDK

Table with data pointers vhost-user + virtio Fast movement of data between processes (zero copy) Based on shared memory with lockless ring buffers Processes exchange data without involving kernel Used ring buf Legacy App Vhost-user client Vhost-user server SPDK Table with data pointers Avail ring buf

vhost-user client impl Minimalistic lib has been implemented for prototyping: https://github.com/openebs/vhost-user It is needed for embedding to legacy application replacing read/write calls Provides simple API for read and write ops, sync and async variants Storage device to open is UNIX domain socket with listening SPDK vhost server instead of traditional block device

Results Our baseline is SPDK with NVMe userspace driver Vhost-user library can achieve around 90% of native SPDK’s performance if tuned properly Said differently: overhead of virtio library can be as low as 10% For comparison, using SPDK with libaio which simulates what could have been achieved with traditional IO interface, achieves 80%

Future work While the results may not be appealing enough, it is not the end SPDK scales nicely with number of CPUs. If vhost-user can scale too, that could make the gap between traditional IO interfaces and vhost-user bigger Requires more work on vhost-user library (adding support for multiple vrings in order to be able to issue IOs to SPDK in parallel) The concern is still there if legacy apps could fully utilize this potential without major rewrite remains

QUESTIONS?