1. Zero-copy Receive Path in Virtio
Kalman Meth, Mike Rapoport, Joel Nider
This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No

2. Virtio Architecture p Host VM Guest user space kernel space
User buffer
Guest
kernel buffer
………….
network
DMA
KVM Hypervisor
Ethernet adapter
virtio/vnet
Ethernet ring buffer(s)
MAC1
MAC4
MAC3
MAC2
NIC
Socket interface
Copy data between guest buffers and host buffers

3. Problem
In the KVM hypervisor, incoming packets from the network must pass through several objects in the Linux kernel before being delivered to the guest VM.
To keep the implementation simple, both the hypervisor and the guest keep their own sets of buffers on the receive path.
For large packets, the overall processing time is dominated by the copying of data from hypervisor buffers to guest buffers.

4. Solution: Zero-copy receive
Maintain separate ring buffer for each guest.
Guest allocates buffers and posts to appropriate host adapter ring buffer.
When data arrives, the NIC performs DMA directly into guest- allocated buffers.

5. Zero-Copy Rx Architecture
Host
VM Guest
user space
kernel space
User buffer
Guest
kernel buffer
………….
network
DMA
KVM Hypervisor
Ethernet adapter
macvlan
macvtap
virtio/vnet
Per-MAC ring buffer
MAC1
MAC4
MAC3
MAC2
NIC
Socket interface
Pass the buffers down through the kernel layers

6. Technical Issues
Need to match guest interface with adapter device that consumes its buffers.
Need support in ethernet adapter to assign a particular queue to a specified MAC address.
What if not enough user-space buffers have been posted?
Drop packets?
Ethernet adapter driver expects 4K page aligned buffers (not what virtio-net currently provides).
Add flag in virtio/vhost to negotiate and use 4K aligned pages
Need 2 values returned for each buffer:
For each buffer filled by Ethernet adapter, need to match it back to index in virtio-net ring.
Need to provide virtio with number of bytes actually written into buffer.
Special handling for broadcast and multicast packets?
Do we need to turn off GRO, and have the guest do all the re-assembly?
Allow different virtio/vhost devices to use different schemes (4K vs meargeable vs big buffers, etc), depending on capabilities of underlying physical device.

7. Proposed Interfaces
int (*ndo_set_zero_copy_rx)(struct net_device *dev, struct net_device *base_dev);
Pass base_dev down the stack to the ethernet adapter so it knows which MAC address is using zero-copy rx.
int (*ndo_post_rx_buffer)(struct net_device *dev, struct sk_buff *skb);
Passes a single (4K aligned) buffer to the ethernet adapter
skb contains pointer to upper level device, so ethernet adapter knows to which MAC address the buffer is associated.
Ethernet adapter should save the skb to return the buffer up the stack (via the sock interface). Can use the skb auxiliary fields to hold the index of the buffer in the virtio-net ring.
May need a callback to handle some cases; perhaps similar to zero- copy transmit mechanism. Need to provide both number of bytes and index into virtio-net ring buffer.