Heitor Moraes, Marcos Vieira, Italo Cunha, Dorgival Guedes Efficient Virtual Network Isolation in Multi-Tenant Data Centers on Commodity Ethernet Switches Heitor Moraes, Marcos Vieira, Italo Cunha, Dorgival Guedes

Problem I want IP 192.168.0.1 I want IP 192.168.0.1 commodity switch

Introduction Ideally, except for specific interconnection agreements, traffic from one tenant‘s VMs should never be visible to other tenants‘ VMs Only that tenant‘s traffic should be able to reach his VMs IaaS providers must provision network resources to garantee isolation between customer networks

Virtualization Server Virtualization Server Data center set up VM 1A VM 2A VM 3A Open vSwitch VM 1B VM 2B VM3B Tenant 1 Tenant 2 Logical plane Tenant 3 Users Physical plane commodity switch Virtualization Server Host B Virtualization Server Host A

Network isolation approaches Extra header Packet tagging with additional packet headers VLANs, QinQ Tunneling Packets transported inside other packets Total control over transport’s packet header Fragmentation Packet rewriting

LANES We propose LANES, a system that: Provides arbitrary virtual network topologies Ensures isolation between tenants Uses commodity switches Is free of encapsulation overheads

LANES set up Follows SDN paradigm Requires no physical changes VM 1A Open vSwitch VM 1B VM 2B VM3B Open vSwitch ETHERNET NETWORK Follows SDN paradigm Requires no physical changes

How LANES works LANES associates a flow identifier to the traffic between each pair of communicating VMs Flow IDs are generated on demand when VMs start communicating and need to be unique only between pairs of servers As Ethernet switches forward packets based on MAC addresses alone, LANES uses the source and destination IP addresses to store flow identifiers.

Packet rewriting Traffic of VMs go through the physical network are rewritten to hide source and destination MAC and IP addresses Each VM interface has a unique IP assigned to it that is associated with its virtual switch It is used in the rewriting SDN rules Openstack and its tenants are unaware of this IP because packets are modified when they leave a host and when they arrive at another

How LANES works Source MAC: Ms Dest MAC: Mr Source IP: 10.0.0.1 Dest IP: 10.0.0.1 MAC Addresses of the OvS Switches LANES Flow ID Rewriting flow rules applied by OvS switch on source virtualization host Source MAC: Mu Dest MAC: Mw Source IP: 192.168.0.1 Dest IP: 192.168.0.2 Packet sent through physical network Source MAC: Mu Dest MAC: Mw Source IP: 192.168.0.1 Dest IP: 192.168.0.2 Rewriting flow rules applied by OvS switch on destination virtualization host Source MAC: Ms Dest MAC: Mr Source IP: 10.0.0.1 Dest IP: 10.0.0.1

How LANES works

Types of traffic Traffic within a host Authorized traffic between VMs located on the same host is forwarded without modification and no packet rewriting is performed

Types of traffic Between hosts When LANES identifies that a packet‘s destination VM runs on a different host, LANES allocates any unused Flow ID to the pair of communicating VMs Flow rules rewrite packet headers before they are transmited through the physical network Flow rules in the destination server recover packet‘s original headers

Types of traffic ARP Queries LANES has all information about the interfaces used by Openstack VMs, including IP, MAC and Network of each one ARP Requests are intercepted by LANES, MAC addresses requested are looked up by the controller in its database and the response is returned only to the requester

Types of traffic IP broadcast Broadcasts packets need to be delivered to all ports allocated on the network of a tenant These networks may span across multiple hosts LANES delivers broadcasts messages to all ports of the network inside the host and rewrites them as unicast packets to be sent to other hosts which also have VMs on the same network Other hosts rewrite back the received packets into broadcast and deliver them to the local ports

Types of traffic External networks LANES generates external flow identifiers for packets between VMs and external IP addresses To avoid generating one flow identifier whenever a VM connects to a different external IP address, external flow identifiers overwrite the source IP address of outbound packets and the destination IP address of inbound packets LANES keeps the external IP address untouched when rewriting inbound and outbound packets

Implementation LANES prototype works on top of OpenStack using the POX SDN controller The virtual network topologies are created using OpenStack‘s Neutron module

Implementation Changes in the virtual networks topology are propagated by Neutron to LANES which can reconfigure Open vSwitches as necessary When a virtualization host boots, its Open vSwitch instance contacts the LANES controller, which configures that instance and adds it to its database

And does it work? What was tested? Network isolation Latency Physical address resolution (ARP) Configuration latency Communication latency after configuration Bandwidth Broadcast latency Controller load under heavy load of new flows

System Evaluation We considered three different software stacks for the evaluation: LANES with POX module L2 switch from POX, which is offered as a reference, indicated as POX+L2 OvS switch as a simple L2 switch, without isolation or an OpenFlow controller

System Evaluation Testing environment A physical infrastructure corresponding to part of the infrastructure of an IaaS provider was build to validate LANE‘s operation One switch for OpenStack control communications, to access the datacenter network, to communicate with the POX controller, and to exchange traffic with the Internet One switch for traffic between virtual machines

Testing network isolation Are the networks protected? ICMP packets were sent to all IPs of the local area network Bandwidth tests were executed while the network was under attack

Testing network isolation LANES OvS and POX+L2

Testing network isolation

Testing MAC address resolution latency How much time does it take to resolve the MAC address of a VM? And during an attack?

Testing MAC address resolution latency LANES OvS POX+L2

Testing flow configuration latency How long does it take for the first packet to leave a VM, be received by the destination and the response return?

Testing flow configuration latency Same server Between servers

Testing estabilished flows latency What is the delay after the forwarding rules are installed into the switches? This is the state where the communication will effectively occur

Packet latency in milliseconds Testing estabilished flows latency Packet latency in milliseconds

Testing bandwidth What is the maximum bandwidth available between VMs? Inside the same virtualization server When the traffic flows through the physical network

Available bandwidth in Gbps Testing bandwidth Available bandwidth in Gbps

Testing scalability Evaluate the controller capacity in dealing with heavy bursts of new flows Measurement of multiple parameters CPU, latency, bandwidth and number of new flows

Testing scalability Bandwidth capacity between VM1 and VM3 Latency between VM1 and VM2 New flows bursts originated on VM4

Testing scalability

Conclusions LANES ensures isolation between virtual networks Packet rewrite hides tenants’ traffic from the physical network LANES can be effective in protecting the network from DoS attacks within the datacenter network

Conclusions LANES does not require advanced features and works on top of commodity Ethernet switches LANES requires no modification to hosted VMs Puts no restrictions on VM IP addresses Does not incur encapsulation overhead

Thank you.

OpenStack Architecture http://docs.openstack.org/security-guide/content/ch031_neutron-architecture.html

OpenFlow versions Ren, Tiantian, and Yanwei Xu. "Analysis of the New Features of OpenFlow 1.4." 2nd International Conference on Information, Electronics and Computer. Atlantis Press, 2014.

Packet rewriting example

Packet rewriting example

Related Work

Tenant’s demands Efficiency Flexibility Freedom Isolation of other tenants It is easy to isolate CPU, memory and storage Network is not