1. Scaling Data Center Networks
Ignas Bagdonas

2. Scaling Data Center Networks
DC players: the Application and the Network.
Different interdependent entities and layers
Scalability is a broad topic.
Different scaling problems for 10 hosts and 10M hosts
The topic is rapidly evolving and may be controversial
There are multiple concepts in different stages of development in various SDOs.
Assuming some foundation knowledge about data center technologies
This is not a technology tutorial and does not cover deep technology details
It does not cover every possible scalability aspect related to DC networking

3. DC Network Architecture Context
A very abstracted view from high above
End System
Hypervisor
Top of Rack
End of Row
Edge Gateway
Have we been here before? H H VM VS VM
ToR
ToR VM VS VM VM
EoR H VM VM VS VM VM GW

4. Scalability Requirements
It has to work.
L2 and L3 identifier scope and rate of growth
L3 to L2 mapping
Identifiers required for data plane isolation
Amount of logical infrastructure entities
OAM scalability
External connectivity separation and isolation

5. Address Resolution and Mapping
Guest VMs cannot be easily modified
Edge gateway needs to have all L3/L2 bindings
EoR and ToR nodes build dataplane driven forwarding topology
Host addressing is stable
Possible solution - L2/L3 mapping
Possible solution - shim interceptor layer and lookup directory
Both options lead to definition of Network Controller entity
Both options interwork with orchestration mechanisms closely
Security aspect – any to any reachability is not required

6. Mobility
L2/L3 mapping works for static topology. VMs can move and the pattern is hardly predictable from network point of view.
Routing system scale limitations
VM to hypervisor to network signalling
ToR node state estimation
This leads to definition of Application Controller entity
Application and Network Controllers must interwork

7. Overlays and Underlays
Partition the network by building another layer and isolate as close to VM as possible, yet not requiring changes to VM.
VN-ID: another layer requires another identifier namespace, abstracted away from data plane
Overlay Gateway function (External, Intra-DC, Inter-DC)
Capsulation points: Hypervisor or ToR?
Host signalling
Edge signalling and external connectivity interworking

8. Overlay ArchitecturesVM VS OV
ToR
Overlay capsulation points can be on hypervisor or ToR or a standalone function H VM VS OV
ToR
EoR H VM VS
ToR
OGW GW

9. Topology Control Overlay tends to hide underlay capabilities
Multidestination forwarding constructs do not help scaling
Distributed control plane is required but not enough
Feedback and interworking with application controller allows for more effective network topology control

10. Overlay Technology Options
Multiple options having different data plane realization, addressing namespace, and control plane options
PBB, SPB(M), TRILL
VXLAN, NVGRE, OTV
EVPN
LISP
STT
MPLS family
Pick any type of data plane representation, L2/L3 address namespace, control plane complexity, ease of OAM integration, end system interworking, and architectural brokenness.

11. Operational Requirements
Interwork with orchestration
Identifier manageability and scope
Service and network discovery
Mapping/directory system
OAM
Security

12. Scalability Revisited
Overlay function in hypervisor
First hop data plane
ToR state
Gateway state
Edge state

13. Summary Adding another layer always solves the problem?
Controllers and dynamic control plane need to interwork.
Many aspects of scalability have been studied and implemented in SP networks – DC network architecture is starting to move into similar direction.