1. NetLord: A Scalable Multi-tenant Network Architecture for Virtualized Cloud Datacenters
Jayaram Mudigonda
Praveen Yalagandula, Jeff Mogul, Bryan Stiekes, Yanick Pouffary
Hewlett-Packard Company

2. The Goal Build the right network for a cloud datacenter?
12 November November 2018

3. The Goal Build the right network for a cloud datacenter?
12 November November 2018

4. Cloud Datacenter Provides Infrastructure as a Service Virtualized
Shared across multiple tenants
Virtualized
Tenants run Virtual Machines (VMs)
12 November November 2018

5. The Right Network Virtualization + Multi-tenancy Scale Inexpensive
A Virtual Network to each tenant
Flexible abstraction: No restrictions on addressing or protocols
Scale
Tenants, Servers: 10s of 1000s, VMs: 100s of 1000s
Adequate bandwidth
Inexpensive
Capital: Cheap COTS components
Operational: Ease of management, High bisection BW
12 November November 2018

6. The Challenge Basic COTS switching gear 
Limited functionality
Limited resources:
Not enough Forwarding Information Base (FIB) space
COTS Switches + Bisection BW  Multipathing
Multipathig and/or Scale  More switch resources
12 November November 2018

7. The Challenge
Assume no switch support and conserve switch resources (FIB) to Simultaneously achieve: Scale, Multi-tenancy, Bisection BW, and Ease of config
12 November November 2018

8. State of the Art – Scale Most prior work is limited by one or more of:
New protocols
Modified control and/or data planes
Preferred topologies
Resources (such as table space) on switches
12 November November 2018

9. State of the Art – Multi-tenancy
Traditional VLANs
Single tenant
No L3 abstraction
Cannot scale beyond 4K
Commercial: Amazon EC2 and Microsoft Azure
Most of recent work is on segregation not virtualization
Diverter [WREN 2010]
Carves a SINGLE IP address space among tenants
Restricts: VM placements and IP address assignment
Can be limited by the FIB space in switches
12 November November 2018

10. NetLord An encapsulation scheme and
A complementary switch configuration 
Scalable multi-tenancy
Ease of configuration
Order of magnitude reduction in FIB requirements
High bisection BW
12 November November 2018

11. Why Encapsulate? Unmodified VM packets onto the network
Excessive FIB pressure and no L3 abstraction
Alternative: Rewrite headers
Cannot assume L3 header  Rewrite only MAC hdr
Rewrite with server MACs  Somewhat reduced FIB
Cannot identify the right VM on destination Server
12 November November 2018

12. NetLord Encapsulation
Two headers: a MAC and an IP
Reduced FIB pressure
Outer Src MAC = MAC of the Src edge switch
Outer Dst MAC = MAC of the Dst edge switch
Correct delivery
Right edge switch: The outer MAC header
Right server: Right port # in the outer dest IP addr
Right VM: Tenant-ID frm outer dest IP + Inner dest MAC
Clean abstraction
No assumptions about VM protocols and/or addressing
12 November November 2018

13. NetLord Encapsulation
VM1
Tenant 15
Hypervisor
Server 1
ES-MAC1 P1
NetLord
Agent
Hypervisor
Server 2
ES-MAC2 P2
NetLord
Agent
VM2
Tenant 15
Port#
TenantID
7 bits
24 bits
IP Address Format
IP-Range-2
COTS Ethernet
ES-MAC2
ES-MAC1
VM1  VM2
P  P
ES-MAC1 
ES-MAC2
12 November November 2018

14. Switch Configuration Outer MAC hdr takes pkt to egress edge switch
A switch on MAC Pkt addressed to itself
Strips MAC hdr and forwards based on IP hdr inside
Standard behavior
Correct forwarding 
Configure the L3 forwarding tables right
Make sure to match the server configs
12 November November 2018

15. … … Switch Configuration 1.*.*.* 1.0.0.1 2.*.*.* 2.0.0.1 48.*.*.*
Hypervisor
Server 1
1.*.*.*
NLA
Hypervisor
Server 2
NLA
Hypervisor
Server 48
NLA …
2.*.*.*
48.*.*.*
Prefix
Gateway, port
1.*.*.*
, 1
2.*.*.*
, 2 …
48.*.*.*
, 48 … 2 1 48
12 November November 2018

16. NL-ARP Builds and maintains the mapping
<VM-MAC,Subnet,Tenant-ID>
 <SWITCH-MAC,PORT>
Query – response. Similar to regular ARP
NL-Agent broadcasts on VM boot-up and migration
12 November November 2018
HP Confidential
12/11/18

17. Goals Achieved: Abstraction
No assumptions about VM-to-VM pkt contents 
No restrictions on addressing schemes
No restrictions on L3 protocols
12 November November 2018

18. Goals Achieved: Scale 24-bit tenant ID  Millions of tenants
Order of magnitude reduction in FIB requirements
Non-edge switches see only edge-switch MACs
Edge switches see their local server MACs in addition
Small number of L3 table entries in edge switches
One per downlink
12 November November 2018

19. Goals Achieved: Ease of Config
Automatic
Local
Same across all switches  Easy to debug
Does not change  Easy to debug
Easy per-tenant traffic engineering in the network
Tenant ID readily available in standard IP header
12 November November 2018

20. Bisection Bandwidth Reduced FIB pressure  Utilize SPAIN effectively
J. Mudigonda, P. Yalagandula, M. Al-Fares, and J. C. Mogul. SPAIN: COTS Data-Center Ethernet for Multipathing over Arbitrary Topologies. In Proc. USENIX NSDI, 2010.
12 November November 2018

21. SPAIN in One Slide Multipath forwarding in container/pod datacenters
Few thousand servers (or unique MAC addresses)
COTS Ethernet (with VLAN support)
Arbitrary topologies
A central, off-line manager:
Discover topology
Compute paths; exploit redundancy in the topology
Merge paths into a minimal set of trees
Install trees as VLANs into switches
End-host (NIC driver):
Download VLAN maps from the manager
Spread flows over VLANs (i.e., paths)
12 November November 2018

22. SPAIN & Extreme Scalability
Critical resource: Forwarding table entries (FIB)
Each MAC address appears on multiple VLANs
Worst case FIB = #VLANS × #MACs
Overflowing FIB  Floods and poor goodput
64K Entries  only a few 1000 unique MACs
NetLord’s ability to hide MACs helps greatly
12 November November 2018

23. Putting It All Together
VM1
Hypervisor
Server 1
P1.*.*.*
NetLord
Agent
VM2
Hypervisor
Server 2
P2.*.*.*
NetLord
Agent
VM1VM2
VM1VM2
IP1IP2
VM1VM2
IP1IP2
macAmacB
VM1VM2
IP1IP2
macAmacB
macB
macA
COTS Ethernet
12 November November 2018

24. Evaluation Prototype + VM emulation Analytical arguments:
Max VMs : ~650 K using 48-port 64K FIB switches
NL-ARP overheads:
< 1% of link BW, ~0.5% of Server memory
Prototype + VM emulation
NLA, encaping overheads are ignorable
Substantial goodput improvements
12 November November 2018

25. Experimental Setup: Topology
74 Servers in a 2-level fat-tree topology
12 November November 2018

26. Prototype, VM Emulation
NLA Kernel module
VM Emulator
A thin module above NLA
Exports a virtual device
Re-writes MAC addresses
Unique pair for each TCP cxn
Up to 3K VMs / Servers
200K VMs in all
User-level
Process …
Protocol Stack
VM Emulator
NetLord Agent
NIC Driver
Physical NIC
Network
12 November November 2018

27. Experimental Setup: Workload
Parallel shuffles
Emulating shuffle-phase of Map-Reduce jobs
Each shuffle: 74 mappers & 74 reducers, one per server
Each mapper transfers 10MB of data to all reducers
12 November November 2018

28. Goodput
Traditional
SPAIN
NetLord
Goodput (in Gbps)
Number of VMs

29. Flooded Packets (in Millions)
Floods
Flooded Packets (in Millions)
Number of VMs
HP Confidential
12/11/18

30. Summary NetLord combines simple existing primitives in a
novel fashion to achieve several out-sized benefits
of practical importance:
Scale
Multi-tenancy
Ease-of-use
Bisection BW
12 November November 2018

31. BACKUP
12 November November 2018

32. Putting It All Together
VM1
Hypervisor
Server 1
P1.*.*.*
NetLord
Agent
VM2
Hypervisor
Server 2
P2.*.*.*
NetLord
Agent
VM1VM2
VM1VM2
IP1IP2
VM1VM2
IP1IP2
macAmacB
VM1VM2
IP1IP2
macAmacB
MAC-A
COTS Ethernet
12 November November 2018