1. VXLAN Nexus 9000 Module 5 – MP-BGP EVPN
@onecloudinc.com

2. Agenda MP-BGP EVPN Overview VXLAN with EVPN Control Plane
Route Optimization using EVPN
VXLAN EVPN Features
Add Control Plane EVPN For Ingress Replication
VXLAN EVPN Configuration
VXLAN EVPN Design Options
Scalability Limits
Agenda
Control EVPN is a key differentiator for cisco

3. MP-BGP EVPN Overview

4. MP-BGP and E-VPN Terminology
Multiprotocol-BGP (MP-BGP): Carries multiple address-family reachability info (IPv4, IPv6, VPN, EVPN)
MP-iBGP: Internal BGP session between internal peers
MP-eBGP: External BGP session between external peers (in different BGP autonomous systems)
Route-reflector (RR): Propagate BGP prefixes to iBGP peers
Route Distinguisher (RD): 8-byte field, VRF parameters; unique value to make VPN IP routes unique
VPNv4 address: RD + VPN prefix
Route Target (RT): 8-byte field, unique value to define the import/export rules for VPNv4 routes
VRF Table: Hold customer routes at PE/VTEP
VNI: Virtual Network Interface
PE: Provider Edge Router
CE: Customer Router
RR, IBGP and EBGP

5. Multi-Protocol BGP (MP-BGP)
Extension to Border Gateway Protocol (BGP)
Multi-Protocol BGP (MP-BGP) carries multiple address family information
IPv4 (Unicast and Multicast)
IPv6 (Unicast and Multicast)
VPNv4, MVPN, EVPN (Ethernet VPN)
Address families controls type of routed protocol information exchanged
Used to distribute VPN routing information between PE’s
Allows to distinguish overlapping customer routes using Route Distinguisher (RD)
Allows reachability among different virtual networks using Route Target (RT)
Customer routes held in different VPN tables

6. MP-BGP with MPLS VPN Route Distribution
Exchange of VPN Policies Among PE Routers
Full mesh of BGP sessions among all PE routers
BGP Route Reflector
Multi-Protocol BGP extensions (MP-iBGP) to carry VPN policies
PE-CE routing options
Static routes
eBGP
OSPF
IS-IS
BGP Route Reflector
PE-CE Link
PE-CE Link PE P P PE CE CE
Label Switched Traffic
Blue VPN
Blue VPN
Red VPN
Red VPN CE P P CE PE PE

7. VXLAN with EVPN Control Plane

8. (on spine or different box)
VXLAN Evolution
Scalable BGP EVPN Control Plane
VTEP
IBGP Route Reflector*
(on spine or different box)
Route
Reflector
Route
Reflector
VXLAN Overlay
BGP Peers on VTEPs
Any control solution will thus need to enable the following
Discovery of the tunnel e/p
Host reachability information
Uses Multi-Protocol BGP (IBGP or EBGP) w EVPN Address Family for:
Dynamic Tunnel Discovery
Dynamic Host reachability information

9. Control Protocol for VXLAN
BGP propagates routes for the host to all other VTEPs 2
Protocol Learning
BGP Route Reflector 1
VTEPs advertise host routes (IP+MAC) to RR
Protocol Learning
VTEP
VTEP
IP A
IP B
West
East
Overlay Forwarding Table
Host1 <MAC,IP> , VTEP IP A 3
VTEPs obtain host routes for remote hosts and install in RIB/FIB
Overlay Forwarding Table
Host1 <MAC,IP> , VTEP IP A
Host2 <MAC,IP> , VTEP IP B 3
VTEP
IP C
South
BGP MPLS Based Ethernet VPN (draft-ietf-l2vpn-evpn-02)
Network Virtualization Overlay Solution using EVPN (draft-sd-l2vpn-evpn-overlay-02)
IETF 9

10. BGP : The Advantages Scale Policy Security Scale think of the internet
Policy : Prefix Filtering , Traffic Engineering, Traffic tagging
Security – Peer Authentication
Scale
Policy
Security

11. Active/Active Multipathing
With EVPN Address Family
Overlay
Optimally
No Tromboning
Bridge
Route
Carries Host L2-address
Carries Host IP/L3 address
Resilient & Efficient
Active/Active Multipathing

12. Why VXLAN EVPN Control Plane?
Cisco Live 2014
4/24/2017
Why VXLAN EVPN Control Plane?
Using MP-BGP to distribute overlay information
Use MP-BGP with EVPN Address Family for Tunnel Endpoint Discovery and Host Reachability
Distributed protocol – no single point of failure
Proven BGP scalability
Physical and virtual endpoints can become BGP peers to participate
Why not use a Central Controller?
Single point of control could become a single point of failure
Scalability
Scope of the controller is limited to devices that understand the communication protocol used by the controller
Central Controller is an NSX thing .. Vmware is using a proprietary protocol

13. VXLAN Underlay - Unicast
Step 1 – OSPF/IGP as Underlay Control Plane
IGP for underlay network reachability
Node reachability for overlay
Quick reaction to fabric link/node failure
Enhanced for mesh topologies
L3 Core
OSPF Control Plane
Underlay Control Protocol doesn’t distribute
Host Routes
Host originated control traffic
Server subnet information
The control protocol segregation offers a clean separation between three control protocols with clearly defined interaction and roles for them  Fabric runs three protocols within: iBGP, LLDP, and ISIS, with their well defined roles.
Fabric control refers to a use of protocol to keep the fabric nodes and their reachability information updated.
ISIS is used to distribute fabric link state within the fabric nodes.
The purpose of fabric protocol, separate from host route distribution, is to offer quick reaction to fabric and/or link/node failure. Such mechanism also helps with having to do a quicker route computation upon events like this and not having to withdraw many routes in the host distribution.
Improvements are done to scale ISIS to work in dense topologies somewhat like Vinci fabric.
It is important to note that fabric control protocol, to help scale, doesn’t distribute host routes, IGMP joins, or subnet reachability from external networks.
Q: server subvnet info: when does that happen?
- The underlay is only for loopback reachbility
OSPF Adjacencies

14. VXLAN Underlay - Multicast
Step 2 – Enable Multicast on Underlay network
L3 Core
Multicast for underlay network
VXLAN VTEP peer discovery
Forward BUM traffic (broadcast, unknown and multicast)
Multicast Enabled Fabric
The control protocol segregation offers a clean separation between three control protocols with clearly defined interaction and roles for them  Fabric runs three protocols within: iBGP, LLDP, and ISIS, with their well defined roles.
Fabric control refers to a use of protocol to keep the fabric nodes and their reachability information updated.
ISIS is used to distribute fabric link state within the fabric nodes.
The purpose of fabric protocol, separate from host route distribution, is to offer quick reaction to fabric and/or link/node failure. Such mechanism also helps with having to do a quicker route computation upon events like this and not having to withdraw many routes in the host distribution.
Improvements are done to scale ISIS to work in dense topologies somewhat like Vinci fabric.
It is important to note that fabric control protocol, to help scale, doesn’t distribute host routes, IGMP joins, or subnet reachability from external networks.
Q: server subvnet info: when does that happen?
PIM Adjacencies

15. VXLAN Overlay MP-BGP EVPN Control Plane
Step 3 – Host and Subnet Route Distribution
Route-Reflectors deployed for scaling purposes RR
L3 Core
MP-BGP Control Plane
iBGP Adjacencies
Host/Subnet
Route Injection
External Subnet Route Injection
iBGP is also used to distribute IGMP group membership to various leaf nodes within the fabric  no need to run a separate protocol for multicast and unicast route distribution.
Improvements are being done to scale this to carry hundreds of thousands of V4 and V6 routes.
The convergence is expected to be similar to that of an IGP protocol, and new features like configuration profiles, with well tested templates, to offer a simple provisioning of iBGP configuration within fabric without having to deal with complexity of BGP deployment.
Host Route Distribution decoupled from the Underlay IGP protocol
Use MP-BGP on the leaf nodes to distribute internal host/subnet routes and external reachability information
MP-BGP enhancements to carry up to 100s of thousands of routes and reduce convergence time

16. Control-Plane EVPN Peer and Host discovery

17. VXLAN Peer and Host Learning Options
Data-Plane
Control-Plane
Core
Multicast
Unicast
Flood and Learn
Peer Learning: DP
EVPN-Multicast
Peer Learning: BGP
Vlan 2
vn-segment 4098
Interface nve 1
member vni 10000
mcast-group
Vlan 2
vn-segment 10000
Interface nve 1
host-reachability protocol bgp
member vni 4098
mcast-group
Static Ingress-Replication
Peer Learning: CLI
EVPN Ingress-Replication
Peer Learning: BGP
Vlan 2
vn-segment 4098
Interface nve 1
member vni 4098
ingress-replication protocol static
Vlan 2
vn-segment 4098
Interface nve 1
host-reachability protocol bgp
member vni 4098
ingress-replication protocol bgp

18. Dynamic VTEP Peer Discovery
BGP Update
BGP Update 4 4
Route Reflector
Dynamic VTEP peer discovery:
VTEP1 peer via EVPN MP-BGP Update 3 3
Dynamic VTEP peer discovery:
VTEP1 peer via EVPN MP-BGP Update 2
VTEP-2
VTEP-3
BGP Update:
H-MAC-1
H-IP-1
VTEP-1
VNI-1
With EVPN Control Plane: Underlay multicast is NOT needed for VTEP peer discovery
VTEP-1 1
H-MAC-1
H-IP-1
VLAN-1 /VNI-1
Underlay multicast is not needed for peer discovery
- BGP has its own damping feature that it could use to manage large number of macs is an unstable environment
Local learning of host info:
H-MAC-1 (MAC table)
H-IP-1 (VRF IP host table )

19. Dynamic Host Reachability Advertisement
Install host info to RIB/FIB:
H-MAC-1  MAC table
H-IP-1  VRF IP host table
Install host info to RIB/FIB:
H-MAC-1  MAC table
H-IP-1  VRF IP host table
BGP Update:
H-MAC-1
H-IP-1
VTEP-1
VNI-1
BGP Update:
H-MAC-1
H-IP-1
VTEP-1
VNI-1 4 4 3
Route Reflector
MAC
Host IP
VNI
VTEP
H-MAC-1
H-IP-1
VNI-1
VTEP-1 3
MAC
Host IP
VNI
VTEP
H-MAC-1
H-IP-1
VNI-1
VTEP-1 2
VTEP-2
VTEP-3
BGP Update:
H-MAC-1
H-IP-1
VTEP-1
VNI-1
VTEP-1
Local learning of host info:
H-MAC-1 (MAC table)
H-IP-1 (VRF IP host table ) 1
Detection of local hosts using ARP/ND/DHCP (New mac learn)
H-MAC-1
H-IP-1
VLAN-1 /VNI-1
Leaf must discover first locally connected devices
In order to advertise host reachability information, a leaf must discover first locally connected devices
Detection of local hosts
Based on VDP (802.1Qbg) or ARP/ND/DHCP (New mac learn)
Detection of remote hosts
Received MP-BGP notifications
MAC
Host IP
VNI
VTEP
H-MAC-1
H-IP-1
VNI-1
VTEP-1
Detection of remote hosts via Received MP-BGP updates

20. EVPN Control Plane – Reachability Distribution
MP-BGP for VXLAN EVPN Control Plane
EVPN Control Plane – Reachability Distribution
EVPN Control Plane -- Host and Subnet Route Distribution L2 L3 L4 S2 S3 S4
Leaf
VTEP
Spine
BGP Update
Host-MAC
Host-IP
Internal IP Subnet
External Prefixes
Use MP-BGP with EVPN Address Family on VTEPs to distribute:
Internal host MAC/IP addresses
Subnet routes
External reachability information
MP-BGP enhancements to carry up to 100s of thousands of routes with reduced convergence time
VTEP Functions are on leaf layer
Spine nodes don’t need to be VTEP
Move some points form here

21. Anycast Gateway at the Leaf
Optimized Network
Anycast Gateway at the Leaf
Anycast Gateway L3 L2
VTEP-1
VTEP-2
VTEP-3
VTEP-4
VTEP-5
VTEP-6
GW IP:
GW MAC: 0011:2222:3333
GW IP:
GW MAC: 0011:2222:3333
Any subnet anywhere => Any leaf can instantiate any subnet
All leafs share gateway IP and MAC for a subnet (No HSRP)
ARPs are terminated on leafs, no flooding beyond leaf
Facilitates VM Mobility, workload distribution, arbitrary clustering
Seamless L2 or L3 communication between physical hosts and virtual machines
The gateway of any subnet can be instantiated on any or all leafs concurrently. Essentially if there are active workloads in a given vlan/subnet below a leaf, the corresponding gateway/SVI is instantiated on that leaf.

22. EVPN Control Plane - Host Movement
VXLAN BGP Control Plane
EVPN Control Plane - Host Movement
NLRI:
Host MAC1, IP1
NVE IP 1
VNI 5000
Next-Hop: VTEP-3
Ext. Community:
Encapsulation: VXLAN
Cost/Sequence: 1
NLRI:
Host MAC1, IP1
NVE IP 1
VNI 5000
Next-Hop: VTEP-1
Ext. Community:
Encapsulation: VXLAN
Cost/Sequence: 0
VTEP-1
VTEP-2
VTEP-3
VTEP-4
Host 1
MAC1
IP 1
VNI 5000
MAC IP
VNI
Next-Hop
Encap
Seq
MAC-1
IP-1
5000
VTEP-1
VXLAN
MAC IP
VNI
Next-Hop
Encap
Seq
MAC-1
IP-1
5000
VTEP-3
VXLAN 1
VTEP-1 detects Host1 and advertise an EVPN route for Host1 with seq# 0
Host1 Moves behind VTEP-3
VTEP-3 detects Host1 and advertises an EVPN route for Host1 with seq #1
VTEP-1 sees more recent route and withdraws its advertisement

23. EVPN Control Plane - Host Movement
VXLAN BGP Control Plane
EVPN Control Plane - Host Movement
NLRI:
Host MAC1, IP1
NVE IP 1
VNI 5000
Next-Hop: VTEP-3
Ext. Community:
Encapsulation: VXLAN
Cost/Sequence: 1
NLRI:
Host MAC1, IP1
NVE IP 1
VNI 5000
Next-Hop: VTEP-1
Ext. Community:
Encapsulation: VXLAN
Cost/Sequence: 0
VTEP-1
VTEP-2
VTEP-3
VTEP-4
Host 1
MAC1
IP 1
VNI 5000
MAC IP
VNI
Next-Hop
Encap
Seq
MAC-1
IP-1
5000
VTEP-1
VXLAN
MAC IP
VNI
Next-Hop
Encap
Seq
MAC-1
IP-1
5000
VTEP-3
VXLAN 1
VTEP-1 detects Host1 and advertise an EVPN route for Host1 with seq# 0
Host1 Moves behind VTEP-3
VTEP-3 detects Host1 and advertises an EVPN route for Host1 with seq #1
VTEP-1 sees more recent route and withdraws its advertisement

24. Summary Classic Vs EVPN Control Plane
Classic VXLAN
EVPN Control Plane
Peer Discovery
Data-driven flood-&-learn
MP-BGP
Host Route Learning
Local hosts: Data-driven flood-&-learn
Remote hosts: Data-driven flood-&-learn
Local Host: Data-driven
Remote host: MP-BGP
BUM Traffic forwarding
Multicast replication
Unicast/Ingress replication
Multicast Dependency Reduced
Route Optimization
Hair-pinning, Need extra VTEPs and Routers
Routing on source VTEP, Anycast Gateway
Select 3-4 main points and move in challenges/overview
Hair-pinning > we needed an external router for routin
With EVPN all routing is done at the leaf switch

25. Control Plane EVPN Summary
Head-end replication to allow unicast-mode only operation
Introduce a control plane to allow for dynamic VTEP discovery
Multicast Dependency
Workload MAC addresses are known once they are connected to the VXLAN capable devices
Leverage the control plane also to exchange L2/L3 address-to-VTEP association information
Flood and Learn based Learning
Introduce VXLAN Gateways
External Connectivity

26. Route Optimization using EVPN

27. Different integrated Route/Bridge (IRB) Modes
VXLAN Routing
Overlay Networks do follow two slightly different integrated Route/Bridge (IRB) semantics
Asymmetric
Ingress VTEP performs both Layer-2 bridging and Layer-3 routing lookup, whereas the egress VTEP performs only Layer-2 bridging lookup
Symmetric
Both the ingress and egress VTEPs perform Layer-2 and Layer-3 lookups
Cisco follows Symmetric IRB
IP Transport Network
Host 1
H-MAC-1
H-IP-1
VNI-A
VTEP-4
VTEP-3
VTEP-2
VTEP-1
Host 2
H-MAC-2
H-IP-2
VNI-B
SVI A
SVI B
Routing ?
Hide some of the slides related to IRB/SRB
Understand the port mapping

28. Asymmetric IRB (Cont’d)
Routing and Bridging on the ingress VTEP
Bridging on the egress VTEP
Both source and destination VNIs need to reside on the ingress VTEP
VTEP-4
VTEP-3
VTEP-2
VTEP-1
Ingress VTEP routes packets from source VNI to destination VNI. D-MAC in the inner header is the destination host MAC
S-MAC: H-MAC-1
D-MAC: H-MAC-2
S-IP: H-IP-1
D-IP: H-IP-2
S-IP: VTEP-1
D-IP: VTEP-4
VNI: VNI-B 1
S-MAC: H-MAC-1
D-MAC: H-MAC-2
S-IP: H-IP-1
D-IP: H-IP-2
VNI A
VNI B
VNI A
VNI B
Add IP addresses? 2
Egress VTEP bridges packets in the destination VNI
S-MAC: H-MAC-1
D-MAC: H-MAC-2
S-IP: H-IP-1
D-IP: H-IP-2
Host 1
H-MAC-1
H-IP-1
VNI-A
Host 2
H-MAC-2
H-IP-2
VNI-B

29. Symmetric IRB Routing on both ingress and egress VTEPs Layer-3 VNI
Tenant VPN indicator
One per tenant VRF
VTEP Router MAC
Ingress VTEP routes packets onto the Layer-3 VNI
Egress VTEP routes packets to the destination Layer-2 VNI

30. Symmetric IRB (Cont’d)
Egress VTEP routes packets from L3 VNI to the destination VNI/VLAN
VTEP-4
Router MAC-4
VTEP-3
VTEP-2
VTEP-1
Router MAC-1
Ingress VTEP routes packets from source VNI to L3 VNI. D-MAC in the inner header is the egress VTEP router MAC
S-MAC: Router-MAC-1
D-MAC: Router-MAC-4
S-IP: H-IP-1
D-IP: H-IP-2
S-IP: VTEP-1
D-IP: VTEP-4
VNI: L3 VNI 1 2
S-MAC: H-MAC-1
D-MAC: H-MAC-2
S-IP: H-IP-1
D-IP: H-IP-2
VNI A
L3 VNI
L3 VNI
VNI B
S-MAC: H-MAC-1
D-MAC: H-MAC-2
S-IP: H-IP-1
D-IP: H-IP-2
Host 1
H-MAC-1
H-IP-1
VNI-A
Host 2
H-MAC-2
H-IP-2
VNI-B
Layer 3 VNI is unique per VRF (tenant) – dedicated for layer 3
- Any routing in symmetric has to be sent to L3 VNI even if you have VNI A & B configured on the same VTEP

31. Symmetric IRB - Benefits
VTEPs don’t need to learn and maintain MAC address information for the remote hosts attached to egress VNIs for which it doesn’t have local hosts
Better utilization of the MAC address table and ARP adjacencies on a VTEP.
Routing and bridging is more scalable than with asymmetric IRB.
Cisco NX-OS implements symmetric IRB to achieve optimal learning and scaling.

32. EVPN Multi-Tenancy and VNI Types (Cont’d)
vlan 200
vn-segment 20000
vlan 201
vn-segment 20100
vlan 3900
name l3-vni-vlan-for-tenant-1
vn-segment 39000
interface Vlan3900
description l3-vni-for-tenant-1-routing
no shutdown
vrf member evpn-tenant-1
ip address /16
fabric forwarding mode anycast-gateway
vrf context evpn-tenant-1
vni 39000
rd auto
address-family ipv4 unicast
route-target import 39000:39000
route-target export 39000:39000
route-target both auto evpn
interface Vlan200
ip address /24
interface Vlan201
ip address /24 L3
Layer-3 VNI
(VRF VNI)
Layer-2 VNI
(Network VNI)
VTEP

33. EVPN Multi-Tenancy and VNI Types
Tenant A (VRF A)
SVIX
SVIA
SVI B
VLAN A
Layer-3 VNI A’
VLAN B
Layer-2 VNI B’
VLAN C
Layer-2 VNI C’
1 Layer-3 VNI per Tenant (VRF) for routing
VNI X’ is used for routed packets
1 Layer-2 VNI per Layer-2 segment
Multiple Layer-2 VNIs per tenant
VNI A’ and B’ are used for bridged packets

34. Multi-tenant Packet Forwarding in Symmetric IRB
IP Transport Network
Use VTEP addresses in the outer header to route encapsulated packets to the egress VTEP
S-MAC: Router-MAC-1
D-MAC: Router-MAC-2
S-IP: H-IP-1
D-IP: H-IP-2
S-IP: VTEP-1
D-IP: VTEP-2
VNI: L3-VNI-A
S-MAC: Router-MAC-1
D-MAC: Router-MAC-4
S-IP: H-IP-1
D-IP: H-IP-2
S-IP: VTEP-1
D-IP: VTEP-2
VNI: L3 –VNI-A
Use L3-VNI to identify the tenant VRF
VTEP-1
VTEP
VTEP
VTEP-2
Host 1
H-MAC-1
H-IP-1
VNI-A
L3-VNI-A
VRF-A
S-MAC: H-MAC-1
D-MAC: H-MAC-2
S-IP: H-IP-1
D-IP: H-IP-2
S-MAC: H-MAC-1
D-MAC: H-MAC-2
S-IP: H-IP-1
D-IP: H-IP-2 L3
Host 2
H-MAC-2
H-IP-2
VNI-B
L3-VNI-A
VRF-A
Tenant A
VRF-A
L3-VNI-A
H-IP-2
Tenant A
VRF-A
L3-VNI-A
H-IP-2
Tenant B
VRF-B
L3-VNI-B
Tenant C
VRF-C
L3-VNI-C

35. Symmetric IRB vs Asymmetric IRB
Symmetric IRB has optimal utilization of ARP and MAC tables on a VTEP.
Symmetric IRB scales better for end hosts.
Symmetric IRB scales better in terms of the total number of VNIs a VXLAN overlay network can support.
Multi-vendor interoperability:
Some vendors implemented Asymmetric IRB.
It’s been agreed upon among multiple vendors that Symmetric IRB is the ultimate solution.
Cisco implemented Symmetric IRB.
Cisco will introduce backward compatibility with asymmetric IRB by adding the support for it.

36. Optimal VXLAN Routing with Symmetric IRB and Anycast Gateway
Host-based fabric routing and bridging with optimal and flexible VXLAN VNI placement
Every VTEP is an anycast gateway for its VXLAN subnets.
Anycast gateway VTEPs share:
The same virtual Gateway IP
The same virtual MAC address
Spine
SVI 100
SVI 100
SVI 100
SVI 200
SVI
300
SVI 200
SVI 200
Leaf
VTEP-1
VTEP-2
VTEP-3
VTEP-4
VTEP-5
Same as before but worth covering here – more animation?
Distributed inter-vxlan host-based routing on local VTEP
Remote host route learning through MP-BGP
Host IP
Port
IP-A
Eth1/1
IP-B
VTEP-4
Host IP
VTEP
IP-A
VTEP-2
IP-B
Eth1/1
VLAN 100
SVI VLAN 100
VNI 5100
Host IP-A
VLAN 200
SVI VLAN 200
VNI 5200
Host IP-B

37. EVPN VXLAN Fabric External Routing
Host-1 (VLAN 100) External Network or Internet RR RR
VXLAN Overlay
EVPN MP-BGP 2
Border Leaf
VTEP-1
VTEP-2
VTEP-3
VTEP-4
VTEP-5
VTEP-6 1
Host 1
MAC_1/ IP_1 3
VXLAN Overlay
EVPN VRF/VRFs Space
IP source and destination
Type 5 are external routes
Type 2 are host-routes in the fabric
Ingress-replicated routes are Type 3
Routing Protocol of Choice
Global Default VRF
Or User Space VRFs
IP Routing 27

38. EVPN VXLAN Fabric External Routing (Cont’d)
Border Leaf
Tenant VRF or Default VRF
VRF OSPF Process
Overlay EVPN VRF A
Overlay EVPN VRF B
Overlay EVPN VRF C
VRFA
For Layer 3 interfaces, use one per overlay VRF instance. The routing protocol neighbor is in the EVPN VRF address family.
Layer 3 interfaces on the external devices can be in either tenant VRF instances or the global default VRF instance.
External Router
VRFB
VRFC
Interface-Type Options:
Physical Routed Ports
Subinterfaces
VLAN SVIs over Trunk Ports

39. VXLAN EVPN Features

40. ARP Suppression in MP-BGP EVPN
ARP suppression reduces network flooding due to host learning
IP Address
MAC Address
VLAN
Physical Interface Index (ifindex)
Flags
IP-1
MAC-1 10
E1/1
Local
IP-2
MAC-2
Null
Remote
IP-3
MAC-3
Host 1
MAC1
IP 1
VLAN 10
VXLAN 5000
Host 2
MAC2
IP 2 S2 S3 S4
VTEP 1 2 3 4
VTEP-1 intercepts the ARP request and checks in its ARP suppression cache. It finds a match for IP-2 in VLAN 10 in its ARP suppression cache.* 2
VTEP-1 sends an ARP response back to Host-1 with MAC-2.* 3
Host-1 in VLAN 10 sends an ARP request for Host-2’s IP-2 address. 1
Host-1 learns the IP-2 and MAC-2 mapping. 4
* If VTEP-1 doesn’t have a match for IP-2 in its ARP suppression cache table, it will flood the ARP request to all other VTEPs in this VNI

41. ARP Suppression in MP-BGP EVPN (Cont’d)
ARP Suppression can be enabled on a per-VNI basis under the interface nve1 configuration.
interface nve1
no shutdown
source-interface loopback0
host-reachability protocol bgp
member vni 20000
suppress-arp
mcast-group
member vni 21000
mcast-group
member vni associate-vrf
member vni associate-vrf S2 S3 S4
VTEP 1
VTEP 2
VTEP 3
VTEP 4
n9396-vtep-1.sakommu-lab.com# sh ip arp suppression topo-info
ARP L2RIB Topology information
Topo-id ARP-suppression mode
L2 ARP Suppression
L2/L3 ARP Suppression
L2/L3 ARP Suppression

42. Head-end Replication Head-end Replication (aka. Ingress replication):
Eliminate the need for underlay multicast to transport overlay BUM traffic
Spine
Multicast-Free
Underlay
VTEP 1
VTEP 2
VTEP 3
VTEP 4
VTEP-1 receives the overlay BUM traffic, encapsulates the packets into unicast VXLAN packets, sends one copy to each remote VTEP peer in the same VXLAN VNI 2
Leaf
Host-1 sends BUM traffic into the VXLAN VNI 1

43. Local Scoping of VLANs –ToR Local
Host-1 (VLAN 10) Host-2 (VLAN 60)
---- N-Way ----
5000
16 million possible VNIs global scope
VTEP-1
VTEP-2
VTEP-3
VTEP-n
Host -1
MAC_1 /
VLANS are Locally Scoped at Top of Rack/ Gateway
VNI 5000 maps to VLAN 10
Possible VLAN IDs 1-4K
Host -2
MAC_2/
VLANS are Locally Scoped at Top of Rack/ Gateway
VNI 5000 maps to VLAN 60
Possible VLAN IDs 1-4K

44. Local Scoping of VLANs – Port Local
Host-1 (VLAN 10) Host-2 (VLAN 60)
---- N-Way ----
16 million possible VNIs global scope
Host -1
MAC_1 /
VNI maps to (E1/1, VLAN 10)
VLANS are Locally Scoped VLAN to VNI mapping is per-port significant
Possible VLAN IDs 1-4K
(Eth1/1, Vlan10) => VNI 10000
(Eth1/2, Vlan10) => VNI 10001
(Eth1/2, Vlan11) => VNI 10000
VTEP-1
VTEP-2
VTEP-3
VTEP-n
Host -2
MAC_2/
VNI maps to (E1/2, VLAN 10)
VLANS are Locally Scoped VLAN to VNI mapping is per-port significant
Possible VLAN IDs 1-4K
???

45. EVPN Control Plane Advantages
A multi-tenant fabric solution with host-based forwarding
Industry standard protocol for multi-vendor interoperability
Build-in multi-tenancy support
Leverage MP-BGP to deliver VXLAN with L3VPN characteristics
Truly scalable with protocol-driven learning
Host MAC/IP address advertisement through EVPN MP-BGP
Fast convergence upon host movements or network failures
MP-BGP protocol driven re-learning and convergence
Upon host movement, the new VTEP will send out a BGP update to advertise the new location of the host

46. EVPN Control Plane Advantages (Cont’d)
A multi-tenant fabric solution with host-based forwarding
Optimal traffic forwarding supporting host mobility
Anycast IP gateway for optimal forwarding for host generated traffic
No need for hair-pinning to to reach the IP gateway
ARP suppression
Minimize ARP flooding in overlay
Head-end Replication with dynamically learned remote-VTEP list
Head-end replication enables multicast-free underlay network
Dynamically learned remote-VTEP list minimizes the operational overhead of head-end replication
VTEP peer authentication via MP-BGP authentication
Added security to prevent rogue VTEPs or VTEP spoofing

47. Ingress Replication for Control Plane EVPN

48. VxLAN Overview
VxLAN provides a way to extend Layer2 extension across Layer 3 infrastructure using MAC-in-UDP encapsulation and tunneling.
VxLAN uses a 24-bit Virtual Network ID (VNID), allowing a maximum of 16 million VxLAN segments to coexist in the same administrative domain.
VxLAN packets are IP routed through the underlying network based on its its Layer 3 header.

49. VXLAN Peer and Host Learning Options
Data-Plane
Control-Plane
Core
Multicast
Unicast
Flood and Learn
Peer Learning: DP
EVPN-Multicast
Peer Learning: BGP-RnH
Vlan 2
vn-segment 4098
Interface nve 1
member vni 10000
mcast-group
Vlan 2
vn-segment 10000
Interface nve 1
host-reachability protocol bgp
member vni 4098
mcast-group
Static Ingress-Replication
Peer Learning: CLI
EVPN Ingress-Replication
Peer Learning: BGP-IMET
128 peers per fabric with ingress-replication protocol bgp
Vlan 2
vn-segment 4098
Interface nve 1
member vni 4098
ingress-replication protocol static
Vlan 2
vn-segment 4098
Interface nve 1
host-reachability protocol bgp
member vni 4098
ingress-replication protocol bgp

50. VXLAN Flooding with BGP-EVPN
Flooding of Broadcast/Unknown-unicast/Multicast (BUM) packets across underlying core network:
Option 1: Use multicast IP core
Each VNI is mapped to a multicast group. For packets coming on access side interfaces miss MAC lookup, they are encapsulated with VNI group address as DIP and sent out along the multicast tree to core
Option 2: Use Ingress Replication (IR)
Some customers would want to avoid using multicast in their core
Remote VTEPs are automatically learnt through Inclusive Multicast Ethernet Tag (IMET) routes.
When missing MAC lookup, a packet is replicated to all remote VTEPs in the VNI, each is encapsulated with one VTEP IP as unicast Destination IP
Support multiple VTEPs per VNI and a VTEP in multiple VNIs

51. Topology and configuration
Hoist-4
VLAN 2
VTEP-3
Ethernet 1/1:
Loopback0 :
VxLAN L2 Gateway
feature nv overlay
feature vn-segment-vlan-based
vlan 2
vn-segment 4098
vlan 3
vn-segment 4099
interface nve1
no shutdown
source-interface loopback0
member vni 4098
ingress-replication protocol bgp
member vni 4099
Hoist-5
VLAN 3
E1/1
feature nv overlay
feature vn-segment-vlan-based
vlan 2
vn-segment 4098
interface nve1
no shutdown
source-interface loopback0
member vni 4098
ingress-replication protocol bgp
Router-3
IP Network
Router-1
Router-2
E1/1
VTEP-1
Ethernet 1/1:
Loopback0 :
VxLAN L2 Gateway
Ethernet 1/1:
Loopback0 :
VxLAN L2 Gateway
VTEP-2
E1/1
Host-1
Host-2
Host-3
VLAN 2
VLAN 2
VLAN 3

52. VXLAN EVPN Configuration

53. VXLAN with MP-iBGP EVPN Configuration
Spine nodes are iBGP route reflector RR RR
Spine
VTEP
VTEP
VTEP
VTEP
VTEP
VTEP
Leaf

54. Initial configuration – Per Switch
Enable VXLAN and MP-BGP EVPN Control Plane
Enable VXLAN
feature nv overlay
feature vn-segment-vlan-based
feature bgp
nv overlay evpn
Enable VLAN-based VXLAN (the currently only mode)
Enable BGP
Enable EVPN control plane for VXLAN
Other features may need to be enabled
Enable OSPF if it’s chosen to be the underlay IGP routing protocol
feature ospf
feature pim
feature interface-vlan
Enable IP PIM multicast routing in the underlay network
Enable VLAN SVI interfaces if the VTEP needs to be IP gateway and route for the VXLAN VLAN IP subnet.

55. EVPN Tenant VRF Create VXLAN tenant VRF vrf context evpn-tenant-1
Create a VXLAN Tenant VRF
vrf context evpn-tenant-1
vni 39000
rd auto
address-family ipv4 unicast
route-target import 39000:39000
route-target export 39000:39000
route-target both auto evpn
Specify the Layer-3 VNI for VXLAN routing within the tenant VRF
Define VRF RD (route distinguisher)
Define VRF Route Target and import/export policies in address-family ipv4 unicast
vrf context evpn-tenant-2
vni 39010
rd auto
address-family ipv4 unicast
route-target import 39010:39010
route-target export 39010:39010
route-target both auto evpn
vlan 3901
name l3-vni-vlan-for-tenant-2
vn-segment 39010
interface Vlan3901
description l3-vni-for-tenant-2-routing
no shutdown
vrf member evpn-tenant-2
ip address /16
fabric forwarding mode anycast-gateway
vrf context evpn-tenant-2
vni 39010
rd auto
address-family ipv4 unicast
route-target import 39010:39010
route-target export 39010:39010
route-target both auto evpn
Example to create a 2nd tenant VRF following the above steps

56. Layer-3 VNI Per Tenant for EVPN Routing
Configure Layer-3 VNI per EVPN Tenant VRF Routing Instant
vlan 3900
name l3-vni-vlan-for-tenant-1
vn-segment 39000
interface Vlan3900
description l3-vni-for-tenant-1-routing
no shutdown
vrf member evpn-tenant-1
vrf context evpn-tenant-1
vni 39000
rd auto
address-family ipv4 unicast
route-target import 39000:39000
route-target export 39000:39000
route-target both auto evpn
Create the VLAN for the Layer-3 VNI.
One Layer-3 VNI per tenant VRF routing instance
Create the SVI interface for the Layer-3 VNI
Put this SVI interface into the tenant VRF context
Associate the Layer-3 VNI with the tenant VRF routing instance.

57. EVPN Layer-3 VNI Per Tenant for Routing Instance
Configure Layer-3 VNI per EVPN Tenant VRF Routing Instant
vlan 3901
name l3-vni-vlan-for-tenant-2
vn-segment 39010
interface Vlan3901
description l3-vni-for-tenant-2-routing
no shutdown
vrf member evpn-tenant-2
vrf context evpn-tenant-2
vni 39010
rd auto
address-family ipv4 unicast
route-target import 39010:39010
route-target export 39010:39010
route-target both auto evpn
Define Layer-3 VNI for a 2nd tenant following the same steps in the previous slide

58. EVPN Layer-2 Network VXLAN VNI
Map VLANs to VXLAN VNIs and Configure their MP-BGP EVPN Parameters
vlan 200
vn-segment 20000
vlan 210
vn-segment 21000
Map VLAN to VXLAN VNI
evpn
vni l2
rd auto
route-target import auto
route-target export auto
vni l2
Under EVPN configuration, define RD and RT import/export policies for each Layer-2 VNIs

59. EVPN Layer-2 Network VXLAN VLAN SVI Interface
Create SVI interface for Layer-2 VNIs for VXLAN routing
Create SVI interface for a Layer-2 VNI.
Associate it with the tenant VRF.
interface Vlan200
no shutdown
vrf member evpn-tenant-1
ip address /8
fabric forwarding mode anycast-gateway
interface Vlan210
ip address /8
All VTEPs for this VLAN/VNI should have the same SVI interface IP address as the distributed IP gateway.
Enable distributed anycast gateway for this VLAN/VNI

60. EVPN Distributed Gateway
Configure distributed gateway virtual MAC address
One virtual MAC per VTEP
All VTEPs should have the same virtual MAC address
fabric forwarding anycast-gateway-mac
interface Vlan210
no shutdown
vrf member evpn-tenant-2
ip address /8
fabric forwarding mode anycast-gateway
Configure virtual IP address
All VTEPs for this VLAN should have the same virtual IP address
Enable distributed gateway for this VLAN

61. VXLAN Tunnel Interface Configuration
Configure VXLAN tunnel interface nve1
interface nve1
no shutdown
source-interface loopback0
host-reachability protocol bgp
member vni 20000
suppress-arp
mcast-group
member vni 21000
mcast-group
member vni associate-vrf
member vni associate-vrf
interface loopback 0
ip address /32
ip ospf network point-to-point
ip router ospf 1 area
ip pim sparse-mode
Specify loopback0 as the source interface
Define BGP as the mechanism for host reachability advertisement
Associate tenant VNIs to the tunnel interface nve1
Define the mcast group on a per-VNI basis
Enable arp suppression on a per-VNI basis
Add Layer-3 VNIs, one per tenant VRF
The loopback interface to source VXLAN tunnels

62. MP-BGP Configuration on VTEP
router bgp 100
router-id
log-neighbor-changes
address-family ipv4 unicast
address-family l2vpn evpn
neighbor remote-as 100
update-source loopback0
send-community extended
neighbor remote-as 100
vrf evpn-tenant-1
advertise l2vpn evpn
vrf evpn-tenant-2
Address-family ipv4 unicast for prefix-based routing
Address-family l2vpn evpn for evpn host routes
Define MP-BGP neighbors.
Under each neighbor define address-family ipv4 unicast and l2vpn evpn
Send extended community in l2vpn evpn address-family to distribute EVPN route attributes
Under address-family ipv4 unicast of each tenant VRF instance, enable advertising EVPN routes

63. MP-BGP Configuration on iBGP Route Reflector
router bgp 100
router-id
log-neighbor-changes
address-family ipv4 unicast
address-family l2vpn evpn
retain route-target all
template peer vtep-peer
remote-as 100
update-source loopback0
send-community both
route-reflector-client
neighbor
inherit peer vtep-peer
neighbor
neighbor
neighbor
Address-family ipv4 unicast for prefix-based routing
Address-family l2vpn evpn for EVPN vxlan host routes
Retain route-targets attributes
iBGP RR client peer template
Send both standard and extended community in address-family ipv4 unicast
Send both standard and extended community in address-family l2vpn evpn

64. VXLAN Fabric with MP-eBGP EVPN
[BGP configuration on a spine switch as in Figure 16 design]
route-map permit-all permit 10
  set ip next-hop unchanged
router bgp 65000
router-id
address-family ipv4 unicast
redistribute direct route-map permitall
address-family l2vpn evpn
nexthop route-map permit-all
retain route-target all
neighbor remote-as 65001
send-community extended
route-map permit-all out
neighbor remote-as 65002
Set next-hop policy to not change the next-hop attributes.
VTEP
Spine
AS 65001
AS 65002
AS 65003
AS 65004
AS 65005
AS 65006
AS 65000
Retain routes with all route targets when advertising the EVPN BGP routes to eBGP peers.
Set outbound policy to advertise all routes to this eBGP neighbor.

65. VXLAN Fabric MP-eBGP EVPN (Cont’d)
[BGP configuration on a spine switch as in Figure 16 design]
route-map permit-all permit 10
  set ip next-hop unchanged
router bgp 65000
router-id
address-family ipv4 unicast
redistribute direct route-map permitall
address-family l2vpn evpn
nexthop route-map permit-all
retain route-target all
neighbor remote-as 65001
send-community extended
route-map permit-all out
neighbor remote-as 65002
Set next-hop policy to not change the next-hop attributes.
Retain routes with all route targets when advertising the EVPN BGP routes to eBGP peers.
Set outbound policy to advertise all routes to this eBGP neighbour.

66. EVPN VXLAN Fabric External Routing
VXLAN Overlay
EVPN VRF/VRFs Space RR RR
Spine
VXLAN Overlay
EVPN MP-BGP
Border Leaf
Leaf
VTEP
VTEP
VTEP
VTEP
VTEP
VTEP
Routing Protocol of Choice
Global Default VRF
Or User Space VRFs
IP Routing

67. EVPN VXLAN External Routing with BGP
Sample Configuration
Router bgp 100
vrf evpn-tenant-1
address-family ipv4 unicast
network /24
neighbor remote-as 200
prefix-list outbound-no-hosts out RR RR
Spine
VXLAN Overlay
EVPN VRF Instance Space
Border Leaf
interface Ethernet2/9.10
mtu 9216
encapsulation dot1q 10
vrf member evpn-tenant-1
ip address /30
VTEP
VTEP
VTEP
VTEP
VTEP
interface Ethernet1/50.10
mtu 9216
encapsulation dot1q 10
ip address /30
router bgp 200
address-family ipv4 unicast
network /24
network /24
neighbor remote-as 100
IP Routing in the Default VRF Instance

68. EVPN VXLAN External Routing with BGP Sample Configuration – On the Border Leaf
On the VXLAN Border Leaf
router bgp 100
router-id
log-neighbor-changes
address-family ipv4 unicast
address-family l2vpn evpn
neighbor remote-as 100
update-source loopback0
send-community extended
neighbor remote-as 100
vrf evpn-tenant-1
network /24
neighbor remote-as 200
prefix-list outbound-no-hosts out
ip prefix-list outbound-no-hosts seq 5 deny /0 eq 32
ip prefix-list outbound-no-hosts seq 10 permit /0 le 32
The eBGP neighbor is on the outside.
It’s in address-family ipv4 unicast of the tenant VRF routing instance.
For better scalability, apply prefix-list to filter out /32 IP host routes. Advertise prefix routes only to the external eBGP neighbor.

69. EVPN VXLAN External Routing with BGP
This is the external route.
n9396-vtep-1# sh ip bgp vrf evpn-tenant
BGP routing table information for VRF evpn-tenant-1, address family IPv4 Unicast
BGP routing table entry for /24, version 70
Paths: (1 available, best #1)
Flags: (0x08041a) on xmit-list, is in urib, is best urib route
vpn: version 75, (0x100002) on xmit-list
Advertised path-id 1, VPN AF advertised path-id 1
Path type: internal, path is valid, is best path, no labeled nexthop
Imported from unknown dest
AS-Path: NONE, path sourced internal to AS
(metric 3) from ( )
Origin IGP, MED not set, localpref 100, weight 0
Received label 39000
Extcommunity: RT:100:39000 ENCAP:8 Router MAC: ae7
Originator: Cluster list:
VRF advertise information:
Path-id 1 not advertised to any peer
VPN AF advertise information:
n9396-vtep-1#
n9396-vtep-1# sh ip route vrf evpn-tenant /24
IP Route Table for VRF "evpn-tenant-1"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
/24, ubest/mbest: 1/0
*via %default, [200/0], 01:01:14, bgp-100, internal, tag 100 (evpn)segid: 0x9858 tunnelid: 0xa encap: 1
The next hop is the VTEP address of the border leaf.
The tenant is VRF L3 VNI.
is the BGP router ID of the border leaf is the spine route reflector.
This is the iBGP route. The next hop is the VTEP address of the border leaf.

70. EVPN VXLAN External Routing with OSPF Sample Configuration
interface Ethernet2/9.10
mtu 9216
encapsulation dot1q 10
vrf member evpn-tenant-1
ip address /30
ip router ospf 1 area RR RR
Spine
VXLAN Overlay
EVPN VRF and VRF Instance Space
Border Leaf
VTEP
VTEP
VTEP
VTEP
VTEP
route-map permit-bgp-ospf permit 10
route-map permit-ospf-bgp permit 10
router ospf 1
router-id
vrf evpn-tenant-1
redistribute bgp 100 route-map permit-bgp-ospf
router bgp 100
log-neighbor-changes
address-family ipv4 unicast
address-family l2vpn evpn
retain route-target all
advertise l2vpn evpn
redistribute ospf 1 route-map permit-ospf-bgp
interface Ethernet1/50.10
mtu 9216
encapsulation dot1q 10
ip address /30
ip router ospf 1 area
IP Routing in the Default VRF Instance

71. EVPN VXLAN External Routing with OSPF
Sample Configuration - The relevant configuration on the border leaf
ip prefix-list bgp-ospf-no-hosts seq 5 permit /0 eq 32
route-map permit-bgp-ospf deny 5
match ip address prefix-list bgp-ospf-no-hosts
route-map permit-bgp-ospf permit 10
route-map permit-ospf-bgp permit 10
router ospf 1
router-id
vrf evpn-tenant-1
redistribute bgp 100 route-map permit-bgp-ospf
router bgp 100
log-neighbor-changes
address-family ipv4 unicast
address-family l2vpn evpn
retain route-target all
neighbor remote-as 100
update-source loopback0
send-community extended
neighbor remote-as 100
advertise l2vpn evpn
redistribute ospf 1 route-map permit-ospf-bgp
Redistribute BGP routes to OSPF. Filter out /32 IP host routes.
A BGP router will modify route targets in l2vpn evpn routes when it is an autonomous system boundary router. The original route target must be retained.
Redistribute OSPF to BGP. Advertise the redistributed routes to L2VPN EVPN.

72. EVPN VXLAN External Routing with OSPF
The internal VTEPs learn the external routes through MP-BGP EVPN
n9396-vtep-1# sh vrf evpn-tenant-1 detail
VRF-Name: evpn-tenant-1, VRF-ID: 3, State: Up
VPNID: unknown
RD: :3
VNI: 39000
Max Routes: 0 Mid-Threshold: 0
Table-ID: 0x , AF: IPv6, Fwd-ID: 0x , State: Up
Table-ID: 0x , AF: IPv4, Fwd-ID: 0x , State: Up
n9396-vtep-1# sh bgp l2vpn evpn rd :
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: :3 (L3VNI 39000)
BGP routing table entry for [5]:[0]:[0]:[24]:[ ]:[ ]/224, version 396
Paths: (1 available, best #1)
Flags: (0x00001a) on xmit-list, is in l2rib/evpn
Advertised path-id 1
Path type: internal, path is valid, is best path, no labeled nexthop
Imported from :3:[5]:[0]:[0]:[24]:[ ]:[ ]/120
AS-Path: NONE, path sourced internal to AS
(metric 3) from ( )
Origin IGP, MED not set, localpref 100, weight 0
Received label 39000
Extcommunity: RT:100:39000 ENCAP:8 Router MAC: ae7
Originator: Cluster list:
Path-id 1 not advertised to any peer
n9396-vtep-1#
The external route learned through MP-BGP EVPN is imported into the tenant VRF.
The next hop is the VTEP address of the border leaf.
This is the Layer 3 VNI of the tenant VRF routing instance.

73. Scalability Limits

74. Unicast Routing Verified Scalability Limits
Feature
9500 Series Verified Limit
9300 Series Verified Limit
BFD sessions (echo mode)
512 (IPv4 only)
512 (IPv6 only)
256 (IPv4) (IPv6)
256 (IPv4 only)
256 (IPv6 only)
128 (IPv4) (IPv6)
BGP neighbors
2000 (IPv4 only)
2000 (IPv6 only)
1000 (IPv4) (IPv6)
EIGRP routes
20,000
EIGRP neighbors
360 (IPv4 only)
360 (IPv6 only)
180 (IPv4) (IPv6)
128 (IPv4 only)
128 (IPv6 only)
64 (IPv4) + 64 (IPv6)
HSRP groups per interface or I/O module
490
250
IPv4 ARP
45,000 (default system routing mode)
60,000 (max-host routing mode)
45,000

75. Unicast Routing Verified Scalability Limits (Cont’d)
Feature
9500 Series Verified Limit
9300 Series Verified Limit
IPv4 host routes
208,000 (default system routing mode)
60,000 (max-host routing mode)
16,000 (ALPM routing mode)
IPv6 host routes
40,000 (default system routing mode)
30,000 (max-host routing mode)
8,000 (ALPM routing mode)
IPv6 ND
40,000
IPv4 unicast routes (LPM)
128,000 (default system routing mode)
16,000 (max-host routing mode)
128,000 with no IPv6 routes (64-bit ALPM routing mode)
12,000 (default system routing mode)
128,000 (ALPM routing mode)
IPv6 unicast routes (LPM)
20,000 (default system routing mode)
4000 (max-host routing mode)
80,000 with no IPv4 routes (64-bit ALPM routing mode)
7000 (6000 routes < /64, 1000 routes > /64) (default system routing mode)
20,000 (ALPM routing mode)
IPv4 and IPv6 unicast routes (LPM) in 64-bit ALPM routing mode
x IPv6 routes and y IPv4 routes, where 2x +y <= 128,000
Not applicable

76. Unicast Routing Verified Scalability Limits (Cont’d)
Feature
9500 Series Verified Limit
9300 Series Verified Limit
MAC addresses
90,000
OSPFv2 neighbors
1000
256
OSPFv3 neighbors
300
VRFs
VRRP groups per interface or I/O module
250

77. Nexus 2000 Series Fabric Extenders (FEX) Verified Scalability Limits
Feature
9500 Series Verified Limit
9300 Series Verified Limit
Fabric Extenders and Fabric Extender server interfaces
Not applicable
16 and 768
VLANs per Fabric Extender
2000 (across all Fabric Extenders)
VLANs per Fabric Extender server interface 75
Port channels
500

78. Interfaces Verified Scalability Limits
Feature
9500 Series Verified Limit
9300 Series Verified Limit
Generic routing encapsulation (GRE) tunnels 8
Port channel links 32
SVIs
490
250
vPCs
275
100 (280 with Fabric Extenders)

79. Layer 2 Switching Verified Scalability Limits
Feature
9500 Series Verified Limit
9300 Series Verified Limit
MST instances 64
MST virtual ports
85,000
48,000
RPVST virtual ports
22,000
12,000
VLANs
4000
3900
VLANs in RPVST mode
500

80. Multicast Routing Verified Scalability Limits
Feature
9500 Series Verified Limit
9300 Series Verified Limit
IPv4 multicast routes
32,000
8000
Outgoing interfaces (OIFs)
40 (see CSCum58876)

81. Layer 2 Switching Verified Scalability Limits
Feature
9500 Series Verified Limit
9300 Series Verified Limit
MST instances 64
MST virtual ports
85,000
48,000
RPVST virtual ports
22,000
12,000
VLANs
4000
3900
VLANs in RPVST mode
500

82. Thank You