1. DPDK overview / APIs
Keith Wiles
July 11, 2016

2. Legal Notices and Disclaimers
Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer. No computer system can be absolutely secure. Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance and benchmark results, visit Intel, the Intel logo and others are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. © 2016 Intel Corporation.
Place at the back of the deck

3. DPDK Overview and Core APIs

4. Agenda DPDK design overview DPDK in an NFV VPP FD.io with DPDK
What is TLDK and use cases
Transport Layer Development Kit

5. DPDK Overview
20 pt Intel Clear Bold Subhead, Date, etc.

6. DPDK Overview DPDK Framework DPDK API Legacy DPDK
Traffic Gens
Pktgen, T-Rex, Moongen, …
vSwitch
OVS, Lagopus, …
DPDK example apps
AES-NI
VPP
Applications
Event based program models
Threading Models
lthreads, …
Video
Apps
EAL
MALLOC
MBUF
MEMPOOL
RING
TIMER
Core Libraries
KNI
POWER
IVSHMEM
Platform
LPM
Classification
ACL
Classify
e1000
ixgbe
bonding
af_pkt
i40e
fm10k
Packet Access PMD
ETHDEV
xenvirt
enic
ring
METER
SCHED
QoS
cxgbe
vmxnet3
virtio
PIPELINE
mlx4
memnic
others
HASH
Utilities
IP Frag
CMDLINE
JOBSTAT
KVARGS
REORDER
TABLE
Legacy DPDK
Future accelerators
Crypto
Programmable Classifier/Parser HW
3rd Party
GPU/FPGA
SoC PMD
External mempool manager
SoC HW
SOC model
VNF Apps
DPDK Acceleration Enhancements
DPDK Framework
Network Stacks
libUNS, mTCP, SeaStar, libuinet, TLDK, …
Compression
HW/SW
IPSec
DPI Hyperscan
Proxy Apps, …

7. DPDK – Data Plane development Kit
DPDK Overview
Core X
DPDK
Application
Core Y
DPDK
Application
Core Y
DPDK
Application
Core N
DPDK
Application
Runs in Linux user space in the Host and/or Guest
Grabs cores and places a single pThread on each core
Each core then becomes a run-to-completion thread
DPDK uses SR-IOV to bypass the Linux Kernel to have direct access to the ports
The PMD or (Poll Mode Driver) handles the access to the port hardware using SR-IOV Virtual Function interfaces one per PMD
The applications use DPDK generic APIs to access and control the ports along with many utilities
DPDK Libraries and utilities
PMD 0
PMD 1
PMD 2
PMD 3
SR-IOV
SR-IOV
SR-IOV
SR-IOV
Linux Kernel
Port 0
Port 1
Port 2
Port 3

8. DPDK – Data Plane development Kit
DPDK Overview
Core X
DPDK
Application
Core Y
DPDK
Application
Core Y
DPDK
Application
Core N
DPDK
Application
Each application runs on a single core for best performance
If applications need to share data then the application must provide the means to communicate
DPDK does contain many types of ways to allow communication between cores lockless rings, semaphore, …
The application needs to poll the ports it wishes to receive packets of data called mbufs, RTE_MBUF
The Rx/Tx routines in DPDK manage multiple mbufs at a time for high performance
The applications also process multiple mbufs at a time
DPDK Libraries and utilities
PMD 0
PMD 1
PMD 2
PMD 3
SR-IOV
SR-IOV
SR-IOV
SR-IOV
Linux Kernel
Port 0
Port 1
Port 2
Port 3

9. DPDK – Data Plane development Kit
DPDK Features
DPDK contains a large number of tools and features for application developers to use
Things like:
Lockless rings
High performance Hash table design
Contains multi-hash support
LPM Longest Prefix Match (IPv4/v6)
vHost support
Packet Framework (pipeline, metering, ...)
IP Fragmentation
Hierarchical Scheduler using Service Level Agreements
Enhanced Memory Copy with AVX instructions
Many different enhances per CPU arch as well
KNI Kernel Network Interface
Power management support
Many more features all highly tuned code
Core X
DPDK
Application
Core Y
DPDK
Application
Core Y
DPDK
Application
Core N
DPDK
Application
DPDK Libraries and utilities
PMD 0
PMD 1
PMD 2
PMD 3
SR-IOV
SR-IOV
SR-IOV
SR-IOV
Linux Kernel
Port 0
Port 1
Port 2
Port 3

10. DPDK – Data Plane development Kit
DPDK Overview
Core X
DPDK
Application
Core Y
DPDK
Application
Core Y
DPDK
Application
Core N
DPDK
Application
The primary Rx/Tx APIs is:
static inline uint16_t
rte_eth_rx_burst(uint8_t port_id, uint16_t queue_id,
struct rte_mbuf **rx_pkts,
const uint16_t nb_pkts);
rte_eth_tx_burst(uint8_t port_id, uint16_t queue_id,
struct rte_mbuf **tx_pkts, uint16_t nb_pkts);
DPDK Libraries and utilities
PMD 0
PMD 1
PMD 2
PMD 3
SR-IOV
SR-IOV
SR-IOV
SR-IOV
Linux Kernel
Port 0
Port 1
Port 2
Port 3

11. DPDK – Data Plane development Kit
DPDK simple startup
int main(int argc, char ** argv) { n = rte_eal_init( argc, argv); argc -= n; argv += n; application_args( argc, argv); nb_ports = rte_eth_dev_count(); /* initialize and allocate port resources */ … rte_eal_mp_remote_launch( lcore_main, NULL, CALL_MASTER); RTE_LCORE_FOREACH_SLAVE( lcore_id) { if ( rte_eal_wait_lcore( lcore_id) < 0) return -1; } return 0;
Core X
DPDK
Application
Core Y
DPDK
Application
Core Y
DPDK
Application
Core N
DPDK
Application
DPDK Libraries and utilities
PMD 0
PMD 1
PMD 2
PMD 3
SR-IOV
SR-IOV
SR-IOV
SR-IOV
Linux Kernel
Port 0
Port 1
Port 2
Port 3

12. DPDK – Data Plane development Kit
DPDK Overview Simple processing loop
Core X
DPDK
Application
Core Y
DPDK
Application
Core Y
DPDK
Application
Core N
DPDK
Application
struct rte_mbuf *pkts[32]; int pid = 0, qid = 0; void lcore_main(void) { /* Setup code for loop */ …. while( core_is_running ) { n = rte_eth_rx_burst(pid, qid, pkts, 32); if ( n ) { n = process_pkts(pkts, n); rte_eth_tx_burst(pid, qid, pkts, n); }
DPDK Libraries and utilities
PMD 0
PMD 1
PMD 2
PMD 3
SR-IOV
SR-IOV
SR-IOV
SR-IOV
Linux Kernel
Port 0
Port 1
Port 2
Port 3

13. DPDK – Data Plane development Kit
DPDK Overview
Core X
DPDK
Application
Core Y
DPDK
Application
Core Y
DPDK
Application
Core N
DPDK
Application
DPDK also runs in Virtual Machines (VM) or Guests running Linux
These VM can access ports directly vi SR-IOV or using VirtIO (virtual I/O)
DPDK does support multiple threads per core, but the thread must corperate using the core
Many example applications are provided to show how to use most of the features in DPDK
Using DPDK with your application in a run-to-completion design can increase your application performance
DPDK Libraries and utilities
PMD 0
PMD 1
PMD 2
PMD 3
SR-IOV
SR-IOV
SR-IOV
SR-IOV
Linux Kernel
Port 0
Port 1
Port 2
Port 3

14. DPDK – Data Plane development Kit
DPDK Overview
Core X
DPDK
Application
Core Y
DPDK
Application
Core Y
DPDK
Application
Core N
DPDK
Application
All source code is provided and MIT licensed
~98% of the code is written in portable ANSI ‘C’
Runs on IA32, IA64, MIPS, PowerPC, ARM, ...
Supports NIC devices from many different vendors including Intel, Broadcom, EZChip, Chelsio, Mellanox, ENA, Cisco ENIC, Cavium, ThunderX, QLOGIC, NXP, ...
DPDK is completely Open Source anyone can contribute to the project
No fees of any kind to contribute or use DPDK
DPDK:
git clone
DPDK Libraries and utilities
PMD 0
PMD 1
PMD 2
PMD 3
SR-IOV
SR-IOV
SR-IOV
SR-IOV
Linux Kernel
Port 0
Port 1
Port 2
Port 3

15. DPDK for NFV Solutions
How to accelerate your NFV with DPDK

16. virtio-backend + vHost-user
NFV DPDK design
Focus on one design as a basic high level view:
Software Acceleration Layer is the software to hardware abstraction layer
Software Acceleration Layer makes possible additional services which can be controlled by the orchestration layer
sio-backend + vHost-user is normally in the SAL or SRL layer, but shown here to illustrate vHost in the host.
A SAL in the guest allows for the best performance selection
Direct access to hardware acceleration via SR-IOV, SOC-specific interface or other pass-though
Able to do software acceleration in the guest
SRL or HW vSwitch adds VM2VM routing or switching of packets
A SAL in the host gives scalability for non-accelerated VMs and/or native applications
Application
SAL (DPDK)
guest
Crypto
virtio
SR-IOV
virtio-backend + vHost-user
SRL(VPP)
SAL(DPDK)
host
Crypto
SR-IOV
Accelerator
HW vSwitch/Crypto
device

17. Acceleration Layer for Host
SAL: Software Acceleration Layer
Provides an abstraction between SW and HW, plus able to support multiple devices at the same time
sio-backend: backend of paravirtualized drivers
vHost-user: User space based VirtIO interface
optional for VM   host access
s-API: APIs for utilizing an AC (APIs from the AC)
g-drivers: General driver for each device type
Implemented in software or the frontend to the hardware (may be different for different acceleration functions)
hio: Hardware I/O interface
Non-virtualized, accessed only by host SAL
AC: Software/Hardware Acceleration Core (DPDK)
SRL: Software Routing Layer (OVS)
AML: Acceleration Management Layer
VM 0
VM 1
VM 2
VM 3
VM 4
sio-backend (optional vHost-user)
Software Routing Layer (SRL)
vSwitch –OVS/VPP
Acceleration Core (DPDK)
DPDK API(s-api)
Software Acceleration Layer (SAL)
Acceleration Management Layer
Buffer and memory mgnt, rings/queues, ingress/egress scheduling, tasks, pipeline, …
g-drivers
SW-crypto or drivers for HW-crypto
SR-IOV(hio)

18. DPDK and VPP
Using VPP to extend DPDK functionality

19. VPP a project within FD.io
FD.io (Fido) is an open source group within the Linux Foundation with a number of sub-projects
The VPP platform is an extensible framework that provides out-of-the-box production quality switch/router functionality
It is the open source version of Cisco's Vector Packet Processing (VPP) technology: a high performance, packet-processing stack that can run on commodity CPUs.
The benefits of this implementation of VPP are its high performance, proven technology, its modularity and flexibility, and rich feature set.
The VPP technology is based on proven technology that has helped ship over $1 Billion of Cisco products. It is a modular design.
The framework allows anyone to "plug in" new graph nodes without the need to change core/kernel code.

20. virtio-backend + vHost-user
NFV DPDK/VPP design
DPDK Application
Software Routing Layer can be VPP (Vector Packet Processing) code for a FastPath design of L2/L3 forwarding
VPP can also be in the guess for location fastpath support
VPP can replace the vSwitch at the routing layer and provide added functionality to the guess or host
Using VPP and DPDK one can create any number of systems for high packet processing performance
DPDK
guest
Crypto
virtio
SR-IOV
virtio-backend + vHost-user
VPP FastPath or OVS
DPDK
host
Crypto
SR-IOV
Accelerators
HW vSwitch/Crypto
device

21. TLDK Network protocol
Transport Layer Development Kit

22. Life of a Packet RX Packet path:
VNF
VNF
VNF
RX Packet path:
RX Packet from port to DPKD via SR-IOV
Packet moves to the application layer and includes:
Native applications
vSwitch
VPP Fast path to VPP applications
User space Network stack to Network applications
vSwitch to VNF is one possible path for packets
TX Packet path:
VNF application sending traffic back to vSwitch
vSwitch and/or other applications sending back to DPDK
DPDK sending the packets via SR-IOV to the network device
Linux native stack path in Green using Sockets
Orange line is the standard VirtIO path to a NFV
VPP Apps
NW Apps
VirtIO 3 1
VPP
Network Stack
DPDK Apps
vSwitch
User Space 2 2
Linux Apps
DPDK 3
Linux Kernel
Native Stack
SR-IOV
Kernel Space 1
Hardware
Ports
Version: 0.2.0

23. TLDK project Transport Layer Development Kit – TLDK
TLDK is a project housed under the FD.io Linux Foundation group
FD.io (pronounced Fido) contains many projects the primary is VPP
VPP is the Cisco contributed Vector Packet Processing code base for L2/L3 fast path used in Cisco routers today
TLDK and other projects in FD.io are open source projects
Anyone can contribute to the project without having to pay any fees
Need a free Linux Foundation login ID to contribute code
The code is free to clone via Git or tarball from:
TLDK Wiki page

24. TLDK (Transport Layer Development Kit)
TLDK is to provide a clean set of ‘C’ libraries to enable network protocol handling at the application layer
TLDK will provide IPv4/v6 and TCP/UDP protocols along with others as required for normal network operation
Goal is to provide a very high performance network stack with termination support for applications using VPP and DPDK
TLDK will provide a set of libraries to allow for applications to build a complete network stack support
Including a high performance non-socket type application interface
Including a socket layer for applications linked with the application
Including a LD_PRELOAD socket layer to run native Linux applications

25. TLDK (Transport Layer Development Kit)
TLDK is not a normal network designed stack!
TLDK has turned the network stack upside down for better performance
Network protocols are driven by the application needing the data
Normal network stack designs drive packet into the protocols, then to the application
In TLDK the packets are per-filtered to a given DPDK core/thread first
The application then drives the packets into the stack when it needs the data not before
The design attempts to keep the CPU cache warm to reduce wasted cycles
The goal is to move multiple packets thru the stack at a time, using the vector style packet processing
Multiple packets at a time allows us to amortize packet processing overhead for higher throughput

26. TLDK Uses case with VPP Application Layer TLDK Control Plane VPP DPDK
Handles packet I/O and protocol processing of packets
Application sets up the UDP/TCP protocol contexts and then calls I/O routines in TLDK to start processing packets
VPP Fastpath:
Using VPP as the first layer for packet processing before packets are sent to the application layer
DPDK:
DPDK provides the I/O abstraction to the physical layer for the network devices. The DPDK could be optional here only if some other I/O layer is used.
Physical Layer:
Ports and other devices like crypto, compression, …
Control Plane:
Not fully defined yet, but will need support in the future
Application Layer
TLDK
VPP
Control Plane
DPDK
Physical Layer
Control or
non-TLDK pkts
TLDK pkts

27. TLDK Application Layer break down
Purpose built TLDK Application
Socket Application
Application Layer:
The application layer utilizes the TLDK library to process packets for UDP and TCP
Purpose Built Application:
A purpose built application is one that uses TLDK APIs directly and is built to use these APIs
Highest performance is expected with this design
BSD Socket Layer:
A standard BSD socket layer for applications using sockets in its design
A lower performance is expected, but allows for current socket type applications to be ported to the system
LD_PRELOAD Socket Layer:
LD_PRELOAD is used to allow a ‘native binary Linux’ application to use the accelerated path of VPP/DPDK
The performance should be a bit better, but does allow these native binary applications to work without any change
LD_PRELOAD
Socket layer
Native Linux Application
BSD Socket layer
TLDK
VPP
optional
DPDK
Physical Layer

28. TLDK Developer View UDP Echo Application TLDK DPDK 40Gbit port IXIA
struct tle_ctx *tle_ctx_create(struct tle_ctx_param *ctx_param);
struct void tle_ctx_destroy(struct tle_ctx *ctx);
struct tle_dev *tle_add_dev(struct tle_ctx *ctx, struct tle_dev_param *dev);
void tle_del_dev(struct tle_dev *dev);
uint16_t tle_udp_rx_bulk(struct tle_dev *dev, struct rte_mbuf *pkts[], struct rte_mbuf *rp[], int32_t rc[], uint16_t num);
uint16_t tle_udp_tx_bulk(struct tle_dev *dev, struct rte_mbuf *pkts[], uint16_t num);
struct tle_stream *tle_udp_stream_open(struct tle_ctx *ctx,
struct tle_udp_stream_param *udp_parm);
int tle_udp_stream_close(struct tle_stream *udp);
/* Global variables and structures */
struct tle_ctx *ctx;
struct tle_stream *udp;
struct tle_ctx_param ctx_param;
struct tle_dev_param dev_param;
struct tle_udp_stream_param udp_param;
struct rte_mbuf *pkts[MAX_PKTS], *not_processed_pkts[MAX_PKTS];
int32_t return_codes[MAX_PKTS];
UDP Echo
Application
TLDK
DPDK
40Gbit port
IXIA

29. TLDK Developer View int main() { uint16_t n, r, running = 1; UDP Echo
ctx = tle_ctx_create(&ctx_param); /* Fill in the ctx_param structure */
tle_add_dev(ctx, &dev_param); /* Fill in the dev_param structure */
/* fill in udp_param here */
udp = tle_udp_stream_open(&udp_param);
while(running) {
n = tle_udp_rx_bulk(dev, pkts, not_processed_pkts, rc, MAX_PKTS);
if (n && ((r = tle_udp_tx_bulk(dev, pkts, n)) != n))
handle_extra_pkts(dev, pkts, r); /* Free or resend? */ }
tle_udp_stream_close(udp);
UDP Echo
Application
TLDK
DPDK
40Gbit port
IXIA

30. TLDK Performance Numbers (non-optimized code)
CPU: Intel(R) Xeon(R) CPU E GHz
64G Ram, Dual socket system, 2x400GB SSD, 2x1TB drives
NIC: Ethernet Controller XL710 for 40GbE QSFP+
Firmware: x
DPDK: 16.07
Linux: Ubuntu (GNU/Linux generic x86_64)
TLDK: Current release ( )
UDP Packet size used is 64 bytes, 5 cores we max out the NIC
UDP Echo
Application
TLDK
DPDK
#Physical Cores
#Queues
Frame Rate Mpps 1
7.4 2
14.8 3
22.2 4
29.5 5
36.4 (max for NIC)
40Gbit port
IXIA

31. More Information on TLDK
The project is under the FD.io a Linux Foundation project
FD.io Wiki Page (
TLDK is located at:
Source Code at: git clone 
Current code base includes an optimized UDP implementation
Currently working on TCP implementation
Each Wednesday10am CST is the TLDK community meeting
Meeting info:
Additional ideas and contributions welcomed!

32. TLDK (Transport Layer Development Kit)
TLDK is an open source FD.IO project (
The TLDK project will implement a set of libraries for L4 protocol processing (UDP, TCP etc.) and VPP graph nodes, plugins, etc Using those libraries to implement a host stack
TLDK works on standalone DPDK or integrated into VPP to provide termination support
Committers are:
Konstantin Ananyev
Keith Wiles
Ed Warnicke (IRC nick: edwarnicke)

33. Further Information
Open Source Website: Documentation: or or Includes Getting Started Guides, Release Notes, Programmer’s Guide, NIC Guides, Sample App Guide, API Guide etc. Mailing Lists: The most commonly used are “dev” for development discussions and patches, and “users” for usage discussions. Webinars: search for “DPDK” Examples include “DPDK 101: Introduction to Data Plane Development Kit” and “Data Plane Development Kit – Sample Applications and New Features Deep Dive”. DPDK Events: Provides information on past and future DPDK events. Videos of presentations at previous events are available on the Past Events page FD.io (Fido) : VPP: TLDK: