1. SDN Architect, Nov 8 2013 Vinay Bannai NEUTRON HYBRID MODE

2. Confidential and Proprietary2 PayPal offers flexible and innovative payment solutions for consumers and merchants of all sizes. 137 Million Active Users $300,000 Payments processed by PayPal each minute 193 markets / 26 currencies PayPal is the World’s Most Widely Used Digital Wallet ABOUT PAYPAL

3. Confidential and Proprietary3 Data Center Architecture Neutron Basics Overlays vs Physical Networks Use Cases Problem Definition Hybrid Solution Performance Data Analysis Q&A INTRODUCTION

4. Confidential and Proprietary4 DATA CENTER ARCHITECTURE Internet Racks Layer-3 switch Access Data Center Layer-3 switch Aggregation Layer-3 router Core Bisection BW

5. Confidential and Proprietary5 NEW DATACENTER ARCHITECTURE Internet vswitches Layer-3 switch Access Data Center Layer-3 switch Aggregation Layer-3 router Core Bisection BW VM Edge Layer

6. Confidential and Proprietary6 DATACENTER WITH VSWITCHES Layer-3 switch Access Data Center VM Racks VM

7. Confidential and Proprietary7 NEUTRON BASICS

8. Confidential and Proprietary8 Overlays provide connectivity between VMs and Network Devices using tunnels The physical core network does not need to be re-provisioned constantly The tunneling encap/decap is done at the edge in the virtual switch Decouples the tenant network address from the physical Data Center network address Easy to support overlapping address Tunneling techniques in vogue −VXLAN −STT −NVGRE OVERLAY NETWORKS

9. Confidential and Proprietary9 Physical Networks connect VM’s and Network Devices using provider network VM’s are first class citizens with the hypervisor and the networking devices No tunneling protocols used Tenant separation is achieved by using VLANs or IP subnetting Hard to achieve overlapping address spaces Underlying network needs to be provisioned with VLANs PHYSICAL NETWORKS

10. Network Virtualization Layer L2 VM L2 L3 VM Tenant on Overlay Network Tenant on Physical Network PHYSICAL VS OVERLAY

11. Confidential and Proprietary11 PROS & CONS FunctionHypervisorBridged VMs (VLAN) Tunneled VM’s ThroughputBestBetterWorse LatencyBestBetterWorse FlexibilityWorseBetterBest Overlapping IP addresses Worse Best Operational Dependency WorseBetterBest

12. Confidential and Proprietary12 Production Environment −Production website across multiple data centers −Low latency and high throughput −Bridged Mode Mergers & Acquisitions Private Community Cloud −Private Community Cloud −Needs address isolation and overlapping −Address isolation, Flexibility, low latency and high throughput −Overlay Mode Development & QA Environment −Production development, QA & Staging −Flexibility, high throughput but can tolerate higher latency −Bridged and Overlay Mode USE CASES

13. Confidential and Proprietary13 Support flexibility, low latency, high throughput and overlapping address space all at the same time Support both bridged and overlay networks VM’s on a hypervisor should be able to choose networks Need a consistent deployment pattern Configurable by automation tools (puppet, chef, salt etc) PROBLEM STATEMENT

14. Confidential and Proprietary14 TYPICAL VSWITCH br-int br-bond VM Ta VM Tb VM Tc br-tun Mgmt Interface Prod Interface VLAN 200 Overlay Traffic Bridged Traffic Bond Intf IP Interface  HYBRID VSWITCH Hypervisor

15. Confidential and Proprietary15 Create the neutron networks −Flat Network −neutron net-create bridged-flat --provider:network_type=flat --provider: physical_network= −neutron subnet-create --allocation-pool start=10.x.x.100, end=10.x.x.200 bridged-flat --gateway 10.x.x.1 10.0.0.0/23 --name bridged-flat-subnet -- enable_dhcp=False −VLAN Network −neutron net-create bridged-vlan --provider:network_type=vlan --provider: physical_network= --provider:segmentation_id= −neutron subnet-create --allocation-pool start=10.x.x.100, end=10.x.x.200 bridged-vlan 10.x.x.1 10.0.0.0/23 --name bridged-vlan-subnet CONFIGURATION OF HYBRID MODE

16. Confidential and Proprietary16 Neutron networks (contd.) −Overlay Network −neutron net-create overylay-net −neutron subnet-create --allocation-pool start=10.x.x.100, end=10.x.x.200 overlay-net --gateway 10.x.x.1 10.0.0.0/23 --name overlay-net-subnet On the compute node −Configure the bond −ovs-vsctl add-br br-bond0 −Configure the OVS −ovs-vsctl br-set-external-id br-bond0 bridgeid br-bond0 −ovs-vsctl set Bridge br-bond0 fail-mode=standalone −ovs-vsctl add-port br-bond0 eth0 eth1 CONTD.

17. Confidential and Proprietary17 To measure latency and throughput, we ran following tests Within a rack (L2 switching) −Bare metal to Bare metal −Bridged VM to Bridged VM −Tunneled VM to Tunneled VM Across racks (L3 switching) −Bare metal to Bare metal −Bridged VM to Bridged VM −tunneled VM to tunneled VM Across the Network Gateway −Bare metal to Bare metal (outside the cloud) −Bridged VM to Bare metal (outside the cloud) −tunneled VM to Bare metal (outside the cloud) PERFORMANCE DATA

18. Confidential and Proprietary18 Compute Hypervisors −2 sockets, 16 cores/socket SandyBridge @ 2.6GHz (32 Hyper Threaded) −2 x 10G ports (Intel PCIe) −RAM : 256GB −Disk: 4 x 600GB in RAID-10 −RHEL 6.4 running OVS VM −vCPUs: 2 −RAM: 8GB −Disk: 20GB −RHEL 6.4 HYPERVISOR, VM AND OS DETAILS

19. Confidential and Proprietary19 TEST SETUP X.X.X.X/23Y.Y.Y.Y/23 Half rack with Two Fault Zones L3 Gateways For Overlays X.X.X.X/23Y.Y.Y.Y/23X.X.X.X/23Y.Y.Y.Y/23

20. Confidential and Proprietary20 Tunneling VM uses STT (OVS) Bridged VM uses Flat Network (OVS) Used nttcp 1.47 for throughput Bi-directional TCP with varying buffer size Buffer size in bytes : [64,… 65536] MTU size : 1500 Bytes (on both bare metal and VM’s) Used ping for latency measurement (60 samples) Used python scripts and paramiko to run the tests Tests done with other traffic (Dev/QA) −Around 470+ active VM’s −Around 100 Hypervisors −Multiple half racks TESTING METHODOLOGY

21. Confidential and Proprietary21 TEST SETUP FOR SAME RACK

22. Confidential and Proprietary22 WITHIN A RACK (L2 SWITCHING) THROUGHPUT

23. Confidential and Proprietary23 WITHIN A RACK (L2 SWITCHING) PING LATENCY

24. Confidential and Proprietary24 Observations Results for buffer size < MTU size −Tunneled VM’s tend to have best overall throughput −Bridged VM’s tend to better than bare metal −OVS and tunnel optimizations at play Results for buffer size > MTU size −Tunneled VM’s and bare metal performance about the same −Bridged VM’s bests both bare-metal and tunneled VMs (??) OVS and tunnel optimizations apply for buffer sizes smaller than MTU OVS optimization apply for buffer sizes greater than MTU Tunneled and Bridged VM’s have a slightly higher latency than bare metal ANALYSIS

25. Confidential and Proprietary25 TEST SETUP ACROSS RACKS

26. Confidential and Proprietary26 ACROSS RACKS (L3 SWITCHING) THROUGHPUT

27. Confidential and Proprietary27 ACROSS R3ACKS (L SWITCHING) PING LATENCY

28. Confidential and Proprietary28 No bridged VM’s in the tests (setup problem) Results for buffer size < MTU size −tunneled VM’s tend to have best overall throughput −OVS and tunnel optimizations at play Results for buffer size > MTU size −tunneled VM’s and bare metal performance about the same OVS and tunnel optimizations apply for buffer sizes smaller than MTU tunneled and Bridged VM’s have a slightly higher latency than bare metal ANALYSIS

29. Confidential and Proprietary29 TEST SETUP ACROSS L3 GATEWAY

30. Confidential and Proprietary30 ACROSS NETWORK GATEWAY THROUGHPUT

31. Confidential and Proprietary31 ACROSS NETWORK GATEWAY PING LATENCY

32. Confidential and Proprietary32 tunneled VM’s tend to have similar if not better throughput as bare metal or bridged VM tunneled VM’s have a slightly higher latency Bridged VM’s tend to have same overall throughput as the hypervisor Bridged VM’s tend to have same latency as the hypervisor Latency from a tunneled VM across L3 gateway is higher than Physical VMs due to extra hops, but need to re-run the tests ANALYSIS

33. Confidential and Proprietary33 Understand your network requirements −Latency, bandwidth throughput, flexibility Overlay Vs Physical Hybrid Mode Performance Analysis Make your deployment patterns simple and repeatable Future work −Additional performance tests −VXLAN, NVGRE −Varying MTU size −Setup without background traffic Let me know if you are interested to collaborate CONCLUSION & FUTURE WORK

34. THANK YOU vbannai@paypal.com