Flow Processing for Fast Path & Inline Acceleration Srinivasa Addepalli – Intel Rajesh Madabhushi- Freescale Denis Crasta - Freescale

Background : Performance requirements & Solutions VPN VM SGW High PPS (Packets/sec) Low Latency Low Jitter High single flow performance Packet Order maintenance in a flow IKE S1-CP IPSec Control SGW-DP IPSec Data Path L2TP NAT Router L2TP-CP Normal Path Control L2TP-DP CGNAT-FP Forwarding - DP PGW FW SLB Control Normal Path Normal Path PGW-DP FW-FP SLB-FP Data Path for many network services is simple and can be separated from Control-path/Normal-Path Data Path in physical appliances is normally implemented using specialized packet processors Specialized Packet Processors. Some examples include Network processors (eg. PCIe based, Network based) Inbuilt (to SoC) specialized network processing FPGA Few examples of network functions

Background : Clouds - Virtual Machines with both Control and Data Planes together MME HSS PCRF AAA Web Server DB Server App Server Anti-Malware ADC WAF API Gateway/Security Scanner Monitoring tools Hadoop Example Virtual Machines L2TP NAT Router L2TP-CP Normal Path Control L2TP-DP CGNAT-FP Forwarding - DP VPN VM SGW PGW FW SLB IKE S1-CP Control Normal Path Normal Path IPSec Control SGW-DP IPSec Data Path PGW-DP FW-FP SLB-FP Host Linux Compute nodes Full inspection Applications No data packet inspection Apps Partial inspection apps CP-DP separation is possible for partial and no-data inspection services/applications

Background – Continued… Virtualization Challenges (Over Physical appliances) Performance Challenges - vSwitch overhead Unable to utilize the specialized packet processors Virtualization Considerations Any vNF should run on compute nodes. Supply Chain – Multiple suppliers in creating cloud environment Compute node hardware vendors VMM vendors Multiple vNFs & Multiple virtual appliance vendors. Multiple accelerator vendors No/Minimal dependencies among the suppliers Virtual appliances must continue to work with newer hardware (Accelerator and compute node) vendors without operator having to go back to virtual appliance vendors for image upgrades New virtual appliances should be able to take advantage of performance features. Standardization and programmability is the key. 10/25/2019

Background : vSwitch Overheads – Mitigations Traditional Accelerated vSwitch compute nodes vSwitch Overheads are minimized Direct connectivity with no changes to vNF is achieved using virtio interface 10/25/2019

Requirement : vNF Data Paths in Packet processors Local apps Local apps Network Function Control Network Function Control Mitigate performance challenges Bring performance closer to the physical appliances DP Control DP Control User space Kernel space Nova compute Neutron agent TCP/IP ovs-vsctl ovs-ofctl IPtables OVSDB Ovs-vswitchd Compute IPSec Control vSwitch DP Control Smart-NIC (FPGA, NP, Inbuilt NP) / DPDK based vSwitch acceleration vHost Eth Network Function Data Path for vNF1 Network Function Data Path for vNF2 But… Cloud challenges…. vSwitch DP

Challenge – Proliferation of Data Path functions L2TP NAT Router L2TP-CP Normal Path Control VPN VM SGW PGW FW SLB IKE S1-CP Control Normal Path Normal Path IPSec Control Host Linux Smart NIC/DPDK vhost - Ethernet Forwarding - DP IPSec Data Path SGW-DP PGW-DP FW-FP SLB-FP CGNAT-FP L2TP-DP vSwitch DP Many Data Path Functions now and in future  Could be a challenge for smart-NIC vendors  Challenge for operators in coordinating between virtual appliance vendors and smart-NIC vendors. Virtual appliance vendors might be satisfied with DP functions provided by smart-NIC vendors. vNF vendors might want to add their own extensions Choices : Provide a way for uploading of new DP modules in smart-NIC Flow programmable DPs (Flow processor)

Challenges with Dynamic DP Modules Many smart-NIC vendors and Smart-NIC technologies Virtual appliance vendors supporting many of them could be a challenge Reliability Concerns Smart-NIC may not provide good isolation among DPs -> Malicious and poorly coded DP could make entire smart-NIC unstable. Security Concerns Performance Isolation concerns Coordination & Code Bloating Concerns Multiple versions of VMs of same type and multiple DP versions -> Code bloating. Version mismatch Dependencies among suppliers etc… 10/25/2019

One Logic (One code piece) Flow Processing One Logic (One code piece) Run time programmable using messages from the control logic (No code upgrade). Multiple Instances of Data Paths. Each instance can be programmable independently with tables, flows in tables and action in flows. Pipeline of flows across the tables. Known best flow processing today Openflow Standardized by ONF. Ability to dynamically program the flows and actions. Ability to create multiple instances. Ability to assign the ownership of control to instances. Ability to control memory usage Possible to implement processing isolation across instances Proven/WIP to create DPs for Forwarding, IPSec, Firewall/NAT, SGW, PGW etc.. Possible to extend or create complementary specifications to meet our requirements. 10/25/2019

Mitigation of proliferation of DP logic modules using Openflow VPN VM SGW PGW FW SLB IKE S1-CP Control Normal Path Normal Path IPSec Control Virtio-flow Virtio-flow Virtio-flow Virtio-flow Virtio-flow Host Linux Smart NIC/DPDK vhost - Ethernet vhost - flow Openflow DP Instance 1 DP Instance 2 DP Instance 3 DP Instance 4 DP Instance 5 As many DP instances as number of vNFs in the compute node. Smart-NICs to have only one logic module (Openflow). Each vNF controls its own DP instance by programming tables, flows and actions at run time. May require upgrade to smart-NIC software only for bug fixes & to implement OF extensions.

Identify the components to enable Openflow based Data Paths DPACC Proposal Identify the components to enable Openflow based Data Paths Openstack Controller Openstack agents QEMU/KVM side Define the API and messages to communicate from Control to Data Planes For standardization. Binary compatibility and source compatibility. Identify the gaps & Document them. Security and reliability aspects Functional limitations Any other… Create a PoC Define the 1st functional PoC Identify the existing Open Source Identify the gaps. 10/25/2019

Openflow based Network Function Data Path 10/25/2019

Without VNF Accelerated Data Path With VNF Accelerated Data Path Local apps Network Function Network Function Control DP Control User space Kernel space Nova compute Neutron agent TCP/IP ovs-vsctl ovs-ofctl IPtables OVSDB Ovs-vswitchd Compute IPSec Control vSwitch DP Control Smart-NIC (FPGA, NP, Inbuilt NP) / DPDK based vSwitch acceleration vHost Eth Network Function Data Path for vNF2 Low Latency, Jitter High throughput vSwitch DP October 25, 2019

OF Data Path OFLS – Data Path Flexibility to configuration entity : Parse & Validation Lookup Instruction Parse & Validation Lookup Instruction Parse & Validation Reassembly Lookup Instruction Pase & Validation Traffic Management PKT-IN I1 I2 I3 I4 I5 I1 I2 I3 I4 I5 A W M Action Write Meter A Action A1 A2 G1 A3 A1 A2 A3 A4 PKT-Out O A11 A12 A13 A14 Output Port Flexibility to configuration entity : Data Path Instances can be created and deleted. Ports can be attached/Detached from data path instances. Flexibility to Control entity : Ability to create multiple pipeline stages (Each stage consisting of a table, set of flows in a table). Ability to create/delete flows in tables. Ability to add instructions/actions to the flows for packet transformation. Control the packet flow using go-to instruction.

Advantages of OF based inline/fast-path One common interface from vNFs irrespective of fast-path type. Multiple fast-path types can be realized using Openflow. Many Newer enhancements to fast-path does not require software upgrade of fast-path. Can be realized using flow programming from the vNFs. Simpler interoperability – One time, but not for each fast-path. No/minimal dependencies between smart-NIC vendors and vNF vendors. October 25, 2019

OF based Data Path realization – Compute node Components OF Control Protocol layer using vrings A vring for messages from control layer. A vring to receive responses from OF DP. A vring for notifications from OF DP. NF DP-CP Glue Layer A glue layer to get hold of DP objects from network function. Example : Netlink socket based module to receive Linux TCP/IP (firewall, IPSec, routing, ARP) objects that would eventually need to be sent to OF DP as OF flows. NF DP Object translation Layer: A layer that implements table patterns in OF DP and translated Network function objects into OF flows with right instructions/actions. Virtio-flow vring backend : A module that receives the flows, instructions/actions from the front-end and programs them in OF DP. Virtio-flow setup backend : A module that informs vring addresses to virtio-flow backend. NOVA-compute/libvirtd/QEMU: Integration with Openstack Controller to create OF DP instances and expose OF DP instances to vNFs. Local apps Network Function NF DP to CP Glue Layer NF DP Object translation Layer Virtio-flow front-end driver VNF NOVA Compute QEMU Libvirtd Virtio-flow setup backend Smart-NIC (FPGA, NP, Inbuilt NP) / DPDK based vSwitch acceleration vHost Eth Virtio-flow vring backend Openflow Switch for vNF DPs vSwitch DP 10/25/2019

Backup 10/25/2019

Network Function Control Network Function Control vNF1 vNF 2 Local apps Network Function Control Local apps Network Function Control Network Function Data Path Network Function Data Path User space Kernel space TCP/IP Nova compute Neutron agent Vhost-net ovs-vsctl ovs-ofctl IPtables OVSDB OVS vSwitchD OVS DP VxLAN + GPE IPSec dataplane

Firewall FP OVS DP VxLAN + GPE Network Function Control vNF1 vNF 2 Local apps Network Function Control Local apps Network Function Control Network Function Data Path Network Function Data Path User space Kernel space Nova compute Neutron agent TCP/IP ovs-vsctl ovs-ofctl IPtables OVSDB Ovs-vswitchd Compute IPSec Control vSwitch DP Control Smart-NIC (FPGA, NP, Inbuilt NP) / DPDK based vSwitch acceleration vHost Eth vSwitch DP Firewall FP OVS DP VxLAN + GPE Compute IPSec FP