1. Maryam Tahhan, Al Morton, Jack Morgan. VSPERF Deep Dive: Virtual Switch performance In OPNFV 1

2. 2 Why are vSwitches important and do we need to characterize their performance? Virtual switches are a key component of Network Function virtualization (NFV) based telecoms networks and Software Defined Networking (SDN) based data centres. –They provide similar functionality to their physical counterparts. –One key difference between a virtual switch and a physical switch is that the vSwitch runs on the same server/platform where the VNFs are running –Its performance can directly affect the number of subscribers/VNFs that can be supported on a single blade.

3. VSPERF Overview Define, implement and execute a test suite to characterize the performance of a virtual switch in the NFVI Drive standardization Promote a defined platform and reuse Establish best practice 30+ committers and contributors

4. VSPERF Standardization and Open Source Projects Feedback Driving the standard platform – by doing

5. VSPERF Deliverables Consumable by: Traffic Gen DUT vSwitch VNF(s) Traffic Gen Client VSPERF Modular Test Framework Consumable by: Test Specification IETF Draft

6. Test specification approach and Test coverage

7. VSPERF test specification approach Start with existing RFCs/Specifications that describe the testing of physical switches, and are applicable to vSwitches. Define additional tests applicable to virtual switches, such as: noisy neighbour tests, datapath and control path coupling, CPU and memory utilization…

8. BMWG and IPPM specs referenced by VSPERF –[RFC2544] Benchmarking Methodology for Network Interconnect Devices.RFC2544 –[RFC2889] Benchmarking Methodology for LAN SwitchingRFC2889 –[RFC6201] Device Reset CharacterizationRFC6201 –[RFC3393] IP Packet Delay Variation Metric for IPPMRFC3393 –[RFC5481] Packet Delay Variation Applicability StatementRFC5481 –[RFC2285] Benchmarking Terminology for LAN Switching DevicesRFC2285 –[RFC1242] Benchmarking Terminology for Network Interconnection Devices.RFC1242

9. VSPERF Test Coverage Matrix SpeedAccuracyReliabilityScalability Activation RFC2889. AddressLearningRate RFC2889. AddressCachingCapacity InitialPacketProcessingLatency LatencyAndLatencyVariation CPDP.Coupling.Flow. Addition RFC2544.SystemRecovery Time RFC2544.ResetTime RFC2889.AddressCachin gCapacity Operation RFC2544.PacketLossRatio RFC2544.PacketLossRateFrmMod RFC2544.BackToBackFrames RFC2889.MaxForwardingRate RFC2889.ForwardPressure RFC2889.BroadcastFrameForwarding RFC2889 Broadcast Frame Latency test Stress.RFC2544.0PacketLoss RFC2544.WorstN-BestN InterPacketDelayVariation.RFC5481 Overlay.Network..RFC2544.Pack etLossRatio RFC2889.ErrorFrames Filtering RFC2544.Profile RFC2889.ErrorFramesFilter ing RFC2544.Profile Scalability.Flows.RFC254 4.0PacketLoss MemoryBandwidth.RFC2 544.0PacketLoss.Scalabi lity Scalability.VNF.RFC2544.PacketLossProfile Scalability.VNF.RFC2544.PacketLossRatio Deactivation 9

10. Key considerations when benchmarking vSwitches

11. 11 Key considerations when benchmarking vSwitch performance Three fundamental considerations: –vSwitches should be benchmarked in a similar way to physical switches – when possible – to allow for a direct comparison. –Try to ensure accuracy, consistency, stability and repeatability of the results between test runs. –It’s essential to limit and if possible eliminate any noise that may interfere with the accuracy of the metrics collected by the test.

12. 12 Consistency and Stability of Test Results Can be achieved through appropriate configuration of: –BIOS –The OS/Kernel. –The system hardware. –The system software.

13. 13 Consistency and Stability Configuration Parameters BIOS Configuration –BIOS should be configured for performance where an explicit option exists –sleep states should be disabled –any virtualization optimization technologies should be enabled –hyper threading should also be disabled –turbo boost and overclocking should also be disabled.

14. 14 Consistency and Stability Configuration Parameters for the OS/Kernel Boot/Grub Parameters: –maxcpus = n where n >= 0; limits the kernel to using 'n' processors. Only use exactly what you need. –isolcpus: Isolate CPUs from the general scheduler. Isolate all CPUs except one which will be used by the OS. –Taskset/affinitize the vSwitch and the VNFs onto isolated cores. –Note: VNFs and the vSwitch must not share the same cores. vCPUs for the VNF should be affinitized to individual cores also.

15. 15 Consistency and Stability Configuration for System HW and SW –Limit the amount of background applications that are running. –Set OS to boot to runlevel 3. Make sure to kill any unnecessary system processes/daemons. –Only enable hardware that you need to use for your test – to ensure there are no other interrupts on the system. –Configure NIC interrupts to only use the cores that are not allocated to any other process (VNF/vSwitch).

16. 16 Repeatability of Test Results In addition to the previous system configuration recommendations it is critical to document all aspects of the SUT to foster repeatability, including: –Hardware parameters –Software parameters –Traffic profile/parameters

17. 17 Hardware Parameters to document for repeatability –Platform details –Processor details –Memory information –Number of enabled cores –Number of cores used for the test –Number of physical NICs. –NIC interrupt configuration – BIOS version, release date and any configurations that were modified – Memory DIMM configurations (quad rank performance may not be the same as dual rank) in size, freq and slot locations – PCI configuration parameters (payload size, early ack option...) – Power management at all levels (ACPI sleep states, processor package, OS...) 17

18. 18 Software Parameters to document for repeatability –OS version –Kernel version –GRUB boot parameters –Hypervisor details –Selected vSwitch, version number –vSwitch launch command line –Memory allocation to the vSwitch –Libraries or any other SW dependency versions. – Memory allocation to a VM – VM launch commandline – VM storage type – Number of VMs – Number of Virtual NICs (vNICs), versions, type, driver and interrupt configuration. – Number of virtual CPUs and their core affinity on the host 18

19. 19 Software Parameters to document for repeatability cont. –Thread affinitization for the applications (including the vSwitch itself) on the host and VNF. –Details of Resource isolation, such as CPUs designated to Host Kernel (isolcpus) and CPUs designated for specific processes (taskset).

20. 20 Traffic Parameters to document for repeatability –Traffic type - UDP, TCP, IMIX / Other –Transmission duration. –Packet Sizes –Deployment Scenario –Number of flows.

21. Benchmarking Methodology

22. VSPERF Typical System Under Test (SUT) 22

23. 23 VSPERF Benchmarking Methodology  Benchmark platform forwarding capability.  Benchmark VNF forwarding capability.  Benchmarking the vSwitch: With isolated resources alone, leaving some resources unused. With isolated resources alone, with other resources (both HW&SW) disabled. With Isolated resources and all resources occupied.

24. Baseline Platform forwarding capability 24 –Run RFC 2889 Maximum forwarding rate test –Transmit traffic at line rate/max forwarding rate (whichever is higher) for at least 72 hours, measure the forwarding rate (fps) and latency. –Traffic should be bidirectional. –Alternative approach is to run an RFC2544 0% frame loss throughput test.

25. Benchmarking the VNF forwarding capability 25 –Run RFC 2889 Maximum forwarding rate test –Transmit traffic at line rate/max forwarding rate (whichever is higher) for at least 72 hours, measure the forwarding rate (fps) and latency. –Traffic should be bidirectional. –Alternative is to run an RFC2544 0% frame loss throughput test. Note: The VNF configuration used for this test, is the configuration that should be used for all subsequent tests.

26. Benchmarking the vSwitch 26 Essentially executing the full test specification on a Typical SUT. The baseline deployment scenarios to be tested by VSPERF to date, include: –Physical to physical (phy2phy). –Single VM loopback framework (PVP). –Two VM loopback framework (PVVP). –VM to VM (VM2VM)..

27. VSPERF LTD Supported Deployment Scenarios PVP VM Test Device (Send&Rcv) vSw Physical port Physical port Test Device (Send&Rcv) vSw Physical port Physical port Logical port Logical port VM Test Device (Send&Rcv) vSw Physical port Physical port Logical port Logical port Logical port Logical port PVVP Phy2Phy

28. Overlay(VM2VM) VSPERF LTD Supported Deployment Scenarios cont. VM Test Device (Send&Rcv) vSw Physical port Physical port Logical port Logical port Logical port Logical port vSw Overlay net VM Test Device (Send&Rcv) vSwitch bypass VM2VM vNIC

29. Physical to Physical (Phy2Phy) 29 Physical port  vSwitch  Physical port. Note: the vSwitch runs on the host

30. VM Loopback (PVP) 30 Physical port  vSwitch  VNF  vSwitch  Physical Port Note: the vSwitch runs on the host

31. Two VM Loopback (PVVP) 31 Physical port  vSwitch  VNF  vSwitch  VNF  vSwitch  Physical Port Note: the vSwitch runs on the host

32. 32 VM2VM in Practice Hasn’t been implemented yet Concerns around time synchronization between VMs and clock accuracy. Recommendation under consideration: Test must include an external HW traffic generator to act as the tester/traffic source and sink.

33. VM2VM Steps Determine: –Forwarding capability and latency through the virtual interface –Forwarding capability and latency through the VNF/hypervisor –Forwarding capability and latency for VM2VM 33

34. 34 Forwarding capability and latency through the virtual interface First determine the forwarding capability and latency through the virtual interface (shown in green). Loopback application should run on the host (in user mode). Loopback application might need to be developed. The same loopback application should be used in all subsequent steps in the methodology. One possible app is DPDK testpmd. To measure the maximum achievable performance the app must be one that is capable of stressing the system/passing traffic at line rate The same driver must be used for both vNICs (igb_uio/vfio). Traffic can be uni/bi-directional. Loopback App Test Device (Traffic Gen) Bypass vNIC Bypass vNIC Host User space

35. 35 Forwarding capability and latency through the VNF/hypervisor Then determine the forwarding capability and latency through the VNF/hypervisor by directly connecting the NIC to the VM (shown in blue). Delay of Bypass paths with vNICs can be subtracted from measured delay (if constant) to get “VM with VNF Delay” alone The same driver must be used for both vNICs as that which determines the forwarding capability and latency through the virtual interface (for example vfio). Traffic can be uni/bi-directional. The same loopback application must be used as that which determines the forwarding capability and latency through the virtual interface VM Test Device (Send&Rcv) bypass vNIC Loopback App User space

36. 36 Forwarding capability and latency for VM2VM Determine the performance for VM2VM in the following configuration (shown in orange). Subtract the latency of the 2x (VM with VNF Delay) + (Bypass paths with vNICs) to eliminate the impairments of the green and blue paths. NOTE: the VNFs used for this methodology must be the same. Traffic can be uni/bi-directional. The same loopback application must be used as that which determines the forwarding capability and latency through the virtual interface VM Test Device (Send&Rcv) vSwitch bypass Logical port Logical port vNIC Loopback App

37. 37 Alternate VM2VM Configuration Emphasize Delay measurement of path through Logical ports of vSwitch with low-rate stream from dedicated and isolated VM-PktGen. Confirm VM-PktGen Isolation in loopback under full vSwitch load Max Forwarding Rate is the sum of the Test Device stream and the low-rate stream from VM NOTE: the VNFs used for this methodology must be the same. Traffic can be uni/bi-directional. The same loopback application must be used as that which determines the forwarding capability and latency through the virtual interface VM-VNF Test Device (Send&Rcv) vSwitch bypass VM2VM VM-PktRcv (Isolated) VM-PktGen (Isolated) Loop back vNIC

38. VSPERF Test Framework

39. A Python based test framework for characterizing the performance of virtual switches. Used to prove out and refine the tests and the methodologies for VSPERF. As of today, capable of conducting the following tests on Vanilla OVS and OVS with DPDK: Supported deployment scenarios to date: no vswitch (Platform baseline), Phy2Phy, PVP and PVVP. Supported modes of operation: normal, trafficgen, trafficgen-off and trafficgen- pause Loopback application in the guest can be DPDK testpmd, Linux bridge or L2FWD kernel Module. Supported trafficgens: Dummy, IXIA, Spirent, Xena and Moongen (In progress).

40. 40 Modular Test Framework VSPERF in Normal mode looks after: Setting up the vSwitch. Setting up the VNF. Setting up the Traffic gen and running the test with it. Collecting the results from the Traffic gen. 40 Traffic Gen DUT vSwitch VNF(s) Traffic Gen Client VSPERF Normal Mode

41. 41 VSPERF (non Normal) Modes of Operation 41 Traffic Gen DUT Traffic Gen Client VSPERF Traffic Gen setup and controlled separately DUT vSwitch VNF(s) VSPERF Traffic Gen DUT vSwitch VNF(s) Traffic Gen Client VSPERF Trafficgen Forwarding on the DUT is setup separately Trafficgen-offTrafficgen-pause

42. 42 VSPERF Modes of Operation for Yardstick 42 Traffic Gen DUT Traffic Gen Client VSPERF Traffic Gen setup and controlled separately DUT vSwitch VNF(s) VSPERF Traffic Gen DUT vSwitch VNF(s) Traffic Gen Client VSPERF Trafficgen Forwarding on the DUT is setup separately Trafficgen-offTrafficgen-pause

43. 43 Trafficgen-only configuration 'traffic_type' - One of the supported traffic types. E.g. rfc2544, back2back or continuous. Default value is "rfc2544". 'bidirectional' - Specifies if generated traffic will be full-duplex (true) or half-duplex (false). Default value is "false". 'iload' - Defines desired percentage of frame rate used during continuous stream tests. Default value is 100. 'multistream' - Defines number of flows simulated by traffic generator. Value 0 disables MultiStream feature. Default value is 0. 'stream_type' - Stream Type is an extension of the "MultiStream" feature. If MultiStream is disabled, then Stream Type will be ignored. Stream Type defines ISO OSI network layer used for simulation of multiple streams. Default value is "L4". SWPC COLLABORATE. INNOVATE. ENRICH.

44. VSPERF Framework Supported Deployment Scenarios PVP VM Test Device (Send&Rcv) vSw Physical port Physical port Test Device (Send&Rcv) vSw Physical port Physical port Logical port Logical port VM Test Device (Send&Rcv) vSw Physical port Physical port Logical port Logical port Logical port Logical port PVVP Phy2Phy Test Device (Send&Rcv) Testpmd Physical port Physical port vSwitch Non

45. VSPERF Framework Supported Deployment Scenarios (In Progress) Test Device (Send&Rcv) VM bypass VM SR-IOV/Bypass logical port logical port

46. NIC SUT (vSwitch + VNF) typical configuration 0 7 Cache System Memory Node 0 CPUs 0 7 Cache System Memory Node 1 CPUs NIC VNF vswitch Traffic Traffic Gen One Free Core for the OS NUMA – Ideally the vSwitch and the VNF reside on the same NUMA node CPU pinning – accomplished using a combination of isolcpus and taskset.

47. Selection of Loopback applications in the Guest DPDK testpmd: set to forward traffic. L2FWD: A Kernel Module that provides OSI Layer 2 Ipv4 termination or forwarding with support for Destination Network Address Translation (DNAT) for both the MAC and IP addresses. Linux Bridge

48. VNF VSPERF uses a VM called vloop_vnf for looping traffic in the PVP and PVVP deployment scenarios. –The image can be found @ http://artifacts.opnfv.org/vswitchperf/vloop-vnf-ubuntu- 14.04_20151216.qcow2 http://artifacts.opnfv.org/vswitchperf/vloop-vnf-ubuntu- 14.04_20151216.qcow2 –Alternatively you can use your own QEMU image.

49. VSPERF for Integration Testing VSPERF includes a set of integration tests defined in conf/integration. These tests can be run by specifying –integration as a parameter to vsperf. VSPERF supports VXLAN, GRE and GENEVE tunneling protocols. Testing of these protocols is limited to unidirectional traffic and P2P (Physical to Physical scenarios). NOTE: The configuration for overlay tests provided in this guide is for unidirectional traffic only. 49

50. Integration Tests * overlay_p2p_tput: Overlay Encapsulation Throughput RFC2544 Test * overlay_p2p_cont: Overlay Encapsulation Continuous Stream * overlay_p2p_decap_tput: Overlay Decapsulation Throughput RFC2544 Test * overlay_p2p_decap_cont: Overlay Decapsulation Continuous Stream * vswitch_add_del_bridge: vSwitch - add and delete bridge * vswitch_add_del_bridges: vSwitch - add and delete bridges * vswitch_add_del_phy_port: vSwitch - add and delete physical port * vswitch_add_del_phy_ports: vSwitch - add and delete physical ports * vswitch_add_del_vport: vSwitch - add and delete virtual port * vswitch_add_del_vports: vSwitch - add and delete virtual ports * vswitch_add_del_flow: vSwitch - add and delete flow * vswitch_add_del_flows: vSwitch - add and delete flows * vswitch_p2p_tput: vSwitch - configure switch and execute RFC2544 throughput test 50

51. Integration Tests cont. * vswitch_p2p_back2back: vSwitch - configure switch and execute RFC2544 back2back test * vswitch_p2p_cont: vSwitch - configure switch and execute continuous stream test * vswitch_pvp: vSwitch - configure switch and one vnf * vswitch_pvp_tput: vSwitch - configure switch, vnf and execute RFC2544 throughput test * vswitch_pvp_back2back: vSwitch - configure switch, vnf and execute RFC2544 back2back test * vswitch_pvp_cont: vSwitch - configure switch, vnf and execute continuous stream test * vswitch_pvp_all: vSwitch - configure switch, vnf and execute all test types * vswitch_pvvp: vSwitch - configure switch and two vnfs * vswitch_pvvp_tput: vSwitch - configure switch, two chained vnfs and execute RFC2544 throughput test * vswitch_pvvp_back2back: vSwitch - configure switch, two chained vnfs and execute RFC2544 back2back test * vswitch_pvvp_cont: vSwitch - configure switch, two chained vnfs and execute continuous stream test * vswitch_pvvp_all: vSwitch - configure switch, two chained vnfs and execute all test types 51

52. Integration Testing in the User Guide http://artifacts.opnfv.org/vswitchperf/docs/userguide/integratio n.html?highlight=integrationhttp://artifacts.opnfv.org/vswitchperf/docs/userguide/integratio n.html?highlight=integration 52

53. Directories and Documentation Overview

54. 54 VSPERF Directories conf: contains the main configuration for testcases, vSwitches, traffic generators and VNFs core: contains the controllers for vSwitches, traffic generators and VNFs. These controllers look after the configuration and launching of deployment scenarios for vSwitches, traffic generators and VNFs. Implementation details of vSwitches, traffic generators and VNFs are abstracted from the controllers. Systems: contains the script that look after installing system dependencies and setting up the python environment for vsperf. src: looks after cloning and building the libraries/utilities we depend on including OVS, DPDK and QEMU (v2.3 for CentOS 7). testcases: contains the base implementation of a test case. tools: contains the implementation of packet generators, collectors and load generators. vnfs: contain the implementation that allow us to launch a VM with a particular hypervisor vswitches: contains the implementation of virtual switches.

55. VSPERF Documentation http://artifacts.opnfv.org/vswitchperf/docs/index.html Includes: User guides Design guides Test Specification Release Information

56. Running Tests Most of the instructions to run a test can be found in the documentation and an example @ https://wiki.opnfv.org/get_started/pod_3_- _characterize_vswitch_performance#running_tests https://wiki.opnfv.org/get_started/pod_3_- _characterize_vswitch_performance#running_tests A summary of steps: –Clone vswitchperf –cd systems and run build_base_machine.sh to install all dependencies –cd../conf, copy and modify the 10_custom.conf with appropriate settings. –cd.. and./vsperf –list –./vsperf –conf-file=$HOME/10_custom.conf phy2phy_cont

57. Continuous Integration

58. 58 Where do VSPERF CI jobs run? VSPERF daily CI jobs run on pod3-wcp-node3 and pod3-wcp- node4 pod3-wcp-node4 is reserved for Moongen and the NIC interface connecting it to pod3-wcp-node3 is a Niantic NIC. Verify and merge jobs run on OPNFV PODs allocated to RELENG.

59. JOBs: The VSPERF CI jobs are broken down into: –Daily job: Runs everyday takes about 10 hours to complete. TESTCASES_DAILY='phy2phy_tput back2back phy2phy_tput_mod_vlan phy2phy_scalability pvp_tput pvp_back2back pvvp_tput pvvp_back2back' TESTPARAM_DAILY='--test-params pkt_sizes=64,128,512,1024,1518' –Merge job: Runs whenever patches are merged to master. Runs a basic Sanity test. –Verify job Runs every time a patch is pushed to gerrit Builds documentation 59

60. VSPERF CI – Scripts and Jenkins Build Results Uses ci/build-vsperf.sh (vsperf) and Jenkins Job builder (jjb) + job description vswitchperf.yml file (releng)build-vsperf.shvswitchperf.yml For more info on writing and using jjbs: https://wiki.opnfv.org/octopus/jenkins_wow?s[]=jjb https://wiki.opnfv.org/octopus/jenkins_wow?s[]=jjb Jenkins Build Results: https://build.opnfv.org/ci/view/vswitchperf/ https://build.opnfv.org/ci/view/vswitchperf/

61. VSPERF CI – OPNFV Output

62. Additional Info Gerrit : https://gerrit.opnfv.org/gerrit/#/q/vswitchperfhttps://gerrit.opnfv.org/gerrit/#/q/vswitchperf List of Candidate Work Items –List of candidate work items for "vSwitch Performance"List of candidate work items for "vSwitch Performance" –vSwitchPerf New Test EtherpadvSwitchPerf New Test Etherpad Demo: https://prezi.com/tlriipoafn5_/vsperf-demohttps://prezi.com/tlriipoafn5_/vsperf-demo

63. SUMMARY 63

64. 64 Summary  It’s important to understand the performance of vSwitches as they are a key component in the NFV infrastructure.  vSwitches should be benchmarked in a similar way to physical switches – when possible  Consistency, stability and repeatability can be achieved through appropriate configuration and documentation of the SUT.  Today, VSPERF describes the baseline scenarios to test the maximum achievable performance of the vSwitch. More realistic scenarios are WIP.  VSPERF provides a python framework that implements a subset of the VSPERF test specification.

65. 65 VSPERF Project Future Work  Integrating multiple traffic gens with the Test Framework: Moongen and Xena  VM Baseline performance using PCI Passthrough/SR-IOV  Methodology extensions: Iterations for the short trial tests  Prove out and refine methodology and tests through the framework  Add more tests to the Test Spec and the framework, an initial list: Scalability Tests Overlay Networking Tests Match action performance testing Classifying L2, L3 and L4 traffic Profile Tests.