1. Enabling Fast, Dynamic Network Processing with ClickOS
Joao Martins*, Mohamed Ahmed*, Costin Raiciu§, Felipe Huici*
* NEC Europe, Heidelberg, Germany
§University Politehnica of Bucharest 1

2. The Idealized Network Physical Datalink Network Transport Application
The view of the network taught in schools looks like this: end systems with a network in between containing switches and routers. Pretty simple.
Physical
Datalink
Page 2 2

3. A Middlebox World transcoder WAN accelerator ad insertion BRAS
session border
controller
carrier-grade NAT
DDoS protection
firewall
IDS
load balancer
But as we all know the reality is quite different. Middleboxes are commonplace in today’s networks, as many middleboxes as routers/switches. They are useful for many reasons: security (firewalls, IDSes), monitoring (DPI, QoE monitors), traffic shaping (load balancers, rate limiters), address exhaustion (NATs), performance optimization (WAN accelerators, transcoders, caches, proxies) and others of a more dubious nature like this ad insertion box.
DPI
QoE monitor
Page 3 3

4. Hardware Middleboxes - Drawbacks
Middleboxes are useful, but…
Expensive
Difficult to add new features, lock-in
Difficult to manage
Cannot be scaled with demand
Cannot share a device among different tenants
Hard for new players to enter market
Clearly shifting middlebox processing to a software-based, multi- tenant platform would address these issues
But can it be built using commodity hardware while still achieving high performance?
ClickOS: tiny Xen-based virtual machine that runs Click
Obviously middleboxes are very useful, but they come with a number of problems: the most dedicated st clear one is price but it certainly not the only one; another issue is that is difficult to add new features; once you buy a cisco box for example you are stuck with them; they are difficult to manage, often requiring specialized personnel for each type of middlebox. Hardware middleboxes cannot be scaled on demand nor can they be shared among different tenants, and it is hard for new players to enter the market.
I'm here to show you a prototype of such a platform called ClickOS...
Page 4

5. Xen Background - Overview
dom0
paravirt
apps
guest OS
domU
paravirt
apps
guest OS
domU
paravirt
apps
guest OS
domU
paravirt
apps
guest OS
domU
paravirt
apps
guest OS
Let me give you some background on XEN...
We have our hardware, and on top of that we run an hypervisor that manages things like: memory, CPU, scheduling. On top of the hypervisor we have the actual virtual machines. Under XEN, VMs are paravirtualized meaning that their OS is slightly modified to interact with the hypervisor. On top of that, there is the guest OS for instance Linux or FreeBSD, and applications running on top of that. And of course we can have multiple virtual machines.
In addition, there is one special VM called Domain 0 (or Dom 0) that has special access to hypervisor in order to do things like create/destroy virtual machines.
hypervisor
dom0
interface
hardware
Page 5 5

6. Xen Background – Split Driver Model
Let me tell you now how networking works under XEN...
XEN has a split driver model, meaning that the Dom0 runs device drivers which connect something called a netback driver. The virtual machines, instead implement a network driver called netfront which talks to the netback driver. All the packet buffers are transferred through shared memory between Domain-0 and the guests.
Page 6 6

7. ClickOS - Contributions
domU
paravirt
apps
guest OS
ClickOS
paravirt
Click
mini OS
Work consisted of
Build system to create ClickOS images (5 MB in size)
Emulating a Click control plane over MiniOS/Xen
Optimizations to reduce boot times (30 miliseconds)
Optimizations to the data plane (10 Gb/s for larger pkt sizes)
As you saw a XEN virtual machine has a PV layer, a guest OS and applications. In the case of ClickOS we use MiniOS as the guest OS. MiniOS as the name implies is a minimalistic operating system distributed with the XEN sources. It has a single memory space (no system calls) and a cooperative scheduler (no context switches). On top of MiniOS, we run the Click modular router software and we call the whole thing ClickOS.
Our contributions include a generic build system in order to generate ClickOS XEN virtual machines. Emulating a Click control plane for MiniOS (there is not a command line in MiniOS that can be used to control Click). Doing optimizations to reduce ClickOS boot times down to 30 miliseconds and other optimizations to the data plane to achieve line-rate for almost all packet sizes.
Because this presentation is short I only have time to speak about the last contribution.
Page 7 7

8. Xen I/O Subsystem and Bottlenecks
pkt size (bytes)
10Gb rate 64
14.8 Mp/s
128
8.4 Mp/s
256
4.5 Mp/s
512
2.3 Mp/s
1024
1.2 Mp/s
1500
810 Kp/s
Driver Domain (e.g., dom0)
ClickOS Domain
netback
netfront
Xen ring API
(data)
Xen bus/store
Click
FromNetfront
ToNetfront
NW driver
Linux/OVS bridge
Event channel
vif
So let me talk to you in a little bit more detail about how Xen’s I/O subsystem works. First we have a NIC, in our case a 10Gb intel NIC. Then the Domain-0 has a network driver for it, for instance the ixgbe driver. This network driver is attached to a software switch, say the Linux bridge or openvswitch. To the switch we attach a virtual interface called VIF, which is managed by the netback driver I mentioned to you earlier. The netback driver connects to the netfront driver over a XEN ring API. We also have an event channel which emulates interrupts and the xenbus used for control messages.\
Finally, we created two new clck elements called FromNetfront and ToNetfront to interact with the netfront driver.
So the goal is to reach 10Gb/s, meaning 800,000 packets per second for maximum sized packets all the way up to 14.8 Mp/s for min-sized ones, as the table shows. 
The problem is that when we first put all of this together there were several important performance bottlenecks. For instance, openvswitch could only 300 Kpps; the netback driver and VIF 350 Kpps and 225 Kpps for the MiniOS netfront driver.
So how do we improve this?
300 Kp/s
350 Kp/s
225 Kp/s
Page 8 8

9. Optimized Xen I/O vif Driver Domain (e.g., dom0) ClickOS Domain
netback
netback
netfront
Click
FromNetfront
ToNetfront
NW driver
Linux/OVS bridge
VALE
Xen bus/store
Event channel
Xen bus/store
Event channel
Xen ring API
(data)
vif
Netmap API
(data)
The first thing we did was replace openvswitch with the High speed VALE software switch, which can switch packets at rates of 18 Mpps. We also got rid of the VIF and any overheads associated with skbufs in the netback driver. So we ended up with a much thinner netback. As a result of these changes we are now using the netmap API, so we have to introduce a new netfront that implements this API. The other thing I should mention is that we map the memory for the ring on the VALE port all the way to the netfront driver, thus avoid memory copies and greatly improving performance.
Page 9 9

10. Throughput – One CPU Core
So now let me show you some performance results.
For this experiment we have two machines: one running ClickOS and the other one just counting packets; both of these have a 10Gb intel card and are connected with a direct cable. We instantiate an increasing number of VMs and we have each send packets to the same interface. And we measure the accumulative throughput and the destination machine.
What we can see from the graph is that we are able to achieve line-rate for larger packet sizes and rates of several million packets per second for smaller packet sizes. Also the graph shows that these rates are sustained even when we create a large number of VMs.
ClickOS
rate meter
10Gb/s direct cable
Page 10 10

11. Boot times 220 milliseconds 30 milliseconds
I havent had a chance to talk about boot time optimizations but let me show you some results from them.
This graph shows how long it takes for a ClickOS VM to boot as we add an increasing number of VMs. As you can see ClickOS can boot in a rather quick 30msecs and even when creating a large number of VMs this figure only goes up to 200msecs.
30 milliseconds
Page 11

12. Conclusions Presented ClickOS Future work
Tiny (5MB) Xen VM tailored at network processing
Can be booted in 30 milliseconds
Can run a large number of ClickOS vm concurrently (128)
Can achieve 10Gb/s throughput using only a single core.
Future work
Implementation and performance evaluation of ClickOS middleboxes (e.g., firewalls, IDSes, carrier-grade NATs, software BRASes)
Work to adapt Linux netfront to netmap API
Service chaining
Introducing Netmap API for virtual machines, breaks the XEN guests such as the Linux one. We are modifying the Linux netfront so that it benefits from our performance improvements.
We are also investigating ways of chaining middleboxes together on the same server in an efficient manner.
Page 12

13. NEC's brand statement, "Empowered by Innovation," expresses our commitment to empowering people and society through continuous innovation in every area of our businesses, fueled by our infinite passion for innovation and our customer-focused spirit of collaboration. 13