Openstack Summit November 2017 Warp-speed Open vSwitch: Turbo Charge VNFs to 100Gbps in NextGen SDN/NFV Datacenter Openstack Summit November 2017 Anita Tragler Product Manager Networking/NFV Mark Iskra Technical Marketing Ash Bhalgat Sr. Director, Cloud Marketing

Agenda Requirements for Next-Gen SDN/NFV Data Centers Boosting NFV performance Datapath options - OVS, SR-IOV, OVS-DPDK OVS Offload options – Full and Partial Offloads Why Full OVS Offload ?, NIC architecture and Packet Flow Open Source Community Contributions Benchmark Testing Setup Performance Results References

Next-Gen Data Center Needs : NFV, 5G and IoT Scale-up Services Need High Bandwidth Millions of Mobile Flows (voice, video, data) 100 Billion 1-10Gbps virtual connections  High Performance: 32-64 Cores/Socket, PCIe Gen4 DDR4, All Flash Arrays (NVMe) 25G to 100G per Server NIC Port Low Latency (1ms RTT)   Optimized Resource Usage (Save CapEx)  Multi-site: Efficient Multi-tenancy w/ SDN Overlay Integrated End-to-End Solution w/ No Vendor Lock-in Which Openstack Network Driver Is Popular? Openstack Survey April 2017

Datapath Options Today - 10G Server DPDK VNF with SR-IOV Single-Root IO Virtualization DPDK VNF with Open vSwitch + DPDK VNF with Open vswitch (kernel datapath) Kernel space User space VF0 VF1 Default for Openstack switching, bonding, overlay, live migration Hardware Dependent to the NIC line rate, no CPU overhead ToR for switching DPDK - Direct IO to NIC or vNIC switching, bonding, overlay

OVS Offload – SR-IOV w/ OVS Control Plane Neutron Nova Openstack Controller Support for offload of includes OVS rule match/classification & action QoS marking, Overlay tunneling (VXLAN, GRE, QinQ) Most usecases need security groups Basic - Firewall (stateless), Filtering Advanced - Connection tracking, NAT Fall back to OVS on host (slow path) [Work In Progress Upstream] VNF VM VNF VM SDN Controller OVS Control ovsdb-server ovs-vswitchd VF VF ovsdb-server User hypervisor KVM ovs-vswitchd Kernel OVS-bridge (kernel datapath) SR-IOV VFs TC/flower offload NIC VF VF PF OVS eswitch 5 NIC NIC

OVS Offload - Virtio options OVS-DPDK Partial Offload QoS, Security groups, Conntrack, overlay Openstack Controller Neutron Nova OVS embedded in NIC No Host OVS Tighter integration testing Openstack Controller VNF VM VNF VM Nova VNF VM VNF VM Neutron SDN Controller VF VF SDN Controller ODL, OVN OVS Control ovsdb-server ovs-vswitchd User vNIC vNIC ovsdb-server OVS-DPDK bridge Partial offload dpdk VF1 VF2 ovs-vswitchd Kernel User NIC OVS Control ovsdb-server ovs-vswitchd ovsdb-server Kernel ovs-vswitchd NIC VF1 VF2 OVS eswitch OVS eswitch NIC NIC NIC NIC

ASAP2 Direct: Full OVS Offload Full OVS Offload: Best of Both Worlds !! ASAP2 Direct: Full OVS Offload Server NICs : 100G is the new 40G, 25G is the new 10G Accelerate OVS data path with standard OVS control plane (Mellanox ASAP2) - In other words, enable support for most SDN controllers with SR-IOV data plane OVS Offload better than OVS DPDK: Up to 10x PPS performance with ZERO CPU consumption (Mellanox Lab Results) SR-IOV VF SR-IOV VF PF

Open Source Community Upstream Contributions Linux Kernel Community Representor port TC (traffic control) and flower offload hooks to NIC Conntrack Offload Openstack Community OVS ML2 driver to bind to new VIF/port type (SR-IOV + OVS) Nova new VIF support Disable OVS in host TripleO Installer OVS Userspace Flow offload via TC or DPDK Policy mechanism Conntrack Offload OVN flow Offload DPDK community Flow offload from DPDK (RTE_Flow) DPDK conntrack offload Currently Available Future

Benchmark Testing Configuration VxLAN Tunnels between back to back hypervisors Server Specs: E5-2667 v3 @ 3.20GHz Mellanox ConnectX-5 NIC (100Gbps) RHEL 7.4 Network Configuration: Switch: Mellanox SN2100(100Gbps) NIC: Mellanox ConnectX-5 100G MTU:9k (Underlay), 1500 (Overlay) SDN: Nuage Virtualized Cloud Services v5.1u1

Line Rate Throughput w/ Zero CPU Utilization !! OVS ASAP2 Achieves ~ Line Rate (94Gbps) for Large Packets VXLAN Tunnels OVS Virtio Can’t Scale beyond 30Gbps OVS ASAP2 CPU Utilization is ~50% lower than OVS Virtio (can’t compare beyond 30Gbps) OVS ASAP2 CPU Utilization at @94G is less than OVS VirtIo @30G Lower is better @ each data rate Testing methodology: iperfv2 load generation 12 cpu cores dedicated to testing Measure difference in CPU utilization CPU Utilization numbers include iPerf Ping latency with 20Gbps background load OVS Kernel-Virtio -- 0.110 ms ASAP2 -- 0.06 ms

Highest PPS Performance w/ Zero CPU Utilization !! OVS ASAP2 Achieves ~60MPPS for Small Packets VXLAN Tunnels CPU Utilization – Entire CPU consumption from test bed only 18% ZERO CPU Utilization for OVS ASAP2 packet processing Flat CPU Consumption Shows: 18%@56MPPS from test bench 0% from OVS ASAP2 Packet Processing Testing methodology: TRex load generation 6 cpu cores dedicated to TestPMD Ping latency with 20Gbps background load Virtio -- 0.110 ms ASAP2 Direct OVS Offload -- 0.06 ms

References Nuage: www.nuagenetworks.net - Nuage Developer Experience: http://nuagex.io - Nuage Networks ML2 Community: https://github.com/nuagenetworks/nuage-openstack-neutron - Nuage for OSPD: https://github.com/nuagenetworks/nuage-ospdirector Mellanox: www.mellanox.com - Using SR-IOV offloads with Open-vSwitch and similar applications: https://netdevconf.org/1.2/papers/efraim-gerlitz-sriov-ovs-final.pdf - OVS patch series for all changes including new offloading API: : https://mail.openvswitch.org/pipermail/ovs-dev/2017-April/330606.html               - DPDK RTE_Flow API: https://rawgit.com/6WIND/rte_flow/master/rte_flow.pdf Redhat: www.redhat.com OpenStack blueprints, specs and reviews OVS Offload SmartNIC enablement Openstack blueprint + Spec [Pike] https://review.openstack.org/#/c/275616/ - neutron https://review.openstack.org/#/c/398265/ - nova ​​https://review.openstack.org/#/c/398277/ - os-vif

Thank-you