Scaling Data Center Networks Ignas Bagdonas ignas.bagdonas@cisco.com

Scaling Data Center Networks DC players: the Application and the Network. Different interdependent entities and layers Scalability is a broad topic. Different scaling problems for 10 hosts and 10M hosts The topic is rapidly evolving and may be controversial There are multiple concepts in different stages of development in various SDOs. Assuming some foundation knowledge about data center technologies This is not a technology tutorial and does not cover deep technology details It does not cover every possible scalability aspect related to DC networking

DC Network Architecture Context A very abstracted view from high above End System Hypervisor Top of Rack End of Row Edge Gateway Have we been here before? H H VM VS VM ToR ToR VM VS VM VM EoR H VM VM VS VM VM GW

Scalability Requirements It has to work. L2 and L3 identifier scope and rate of growth L3 to L2 mapping Identifiers required for data plane isolation Amount of logical infrastructure entities OAM scalability External connectivity separation and isolation

Address Resolution and Mapping Guest VMs cannot be easily modified Edge gateway needs to have all L3/L2 bindings EoR and ToR nodes build dataplane driven forwarding topology Host addressing is stable Possible solution - L2/L3 mapping Possible solution - shim interceptor layer and lookup directory Both options lead to definition of Network Controller entity Both options interwork with orchestration mechanisms closely Security aspect – any to any reachability is not required

Mobility L2/L3 mapping works for static topology. VMs can move and the pattern is hardly predictable from network point of view. Routing system scale limitations VM to hypervisor to network signalling ToR node state estimation This leads to definition of Application Controller entity Application and Network Controllers must interwork

Overlays and Underlays Partition the network by building another layer and isolate as close to VM as possible, yet not requiring changes to VM. VN-ID: another layer requires another identifier namespace, abstracted away from data plane Overlay Gateway function (External, Intra-DC, Inter-DC) Capsulation points: Hypervisor or ToR? Host signalling Edge signalling and external connectivity interworking

Overlay Architectures VM VS OV ToR Overlay capsulation points can be on hypervisor or ToR or a standalone function H VM VS OV ToR EoR H VM VS ToR OGW GW

Topology Control Overlay tends to hide underlay capabilities Multidestination forwarding constructs do not help scaling Distributed control plane is required but not enough Feedback and interworking with application controller allows for more effective network topology control

Overlay Technology Options Multiple options having different data plane realization, addressing namespace, and control plane options PBB, SPB(M), TRILL VXLAN, NVGRE, OTV EVPN LISP STT MPLS family Pick any type of data plane representation, L2/L3 address namespace, control plane complexity, ease of OAM integration, end system interworking, and architectural brokenness.

Operational Requirements Interwork with orchestration Identifier manageability and scope Service and network discovery Mapping/directory system OAM Security

Scalability Revisited Overlay function in hypervisor First hop data plane ToR state Gateway state Edge state

Summary Adding another layer always solves the problem? Controllers and dynamic control plane need to interwork. Many aspects of scalability have been studied and implemented in SP networks – DC network architecture is starting to move into similar direction.