1. The good, the bad and the ugly…
NSX-V
The good, the bad and the ugly…

2. Quick poll… Who has heard of NSX?
Who has played with NSX in a Lab environment?
Who has done training/certifications on NSX?
Who is using it in production?
James Cruickshank

3. Who am I?
James Cruickshank Virtualisation Engineer at Sky VCIX-NV and VCAP-DCA vExpert vSAN vExpert NSX vExpert
James Cruickshank

4. Management Plane
Control Plane
Data Plane

5. Management Plane
vSphere API
REST API
vCenter Server
NSX Manager
API

6. NSX Controller Cluster
Management Plane
vSphere API
REST API
vCenter Server
NSX Manager
API
Control Plane
NSX Controller Cluster

9. NSX Controller Cluster
Management Plane
vSphere API
REST API
vCenter Server
NSX Manager
API
Control Plane
NSX Controller Cluster
Data Plane
esx-vxlan
esx-vsip

12. NSX releases prior to 6.2.x have the esx-dvfilter-switch-security vib
esx-vxlan – This package loads module and configures firewall for vxlan networking
esx-vsip – This package contains DFW and NetX data and control plane components

14. NSX Controller Cluster
Management Plane
vSphere API
REST API
vCenter Server
NSX Manager
API
Control Plane
NSX Controller Cluster
Data Plane
Distributed Routing
VXLAN
Distributed Firewall
esx-vxlan
esx-vsip

15. NSX Controller Cluster
Management Plane
vSphere API
REST API
vCenter Server
NSX Manager
API
Control Plane
NSX Controller Cluster
Data Plane
vSphere Stateful Firewall Daemon
Network Control Plane Agent
VSFWD
NETCPA
Distributed Routing
VXLAN
Distributed Firewall
esx-vxlan
esx-vsip

16. VTEP VXLAN Tunnel End Point
Virtual Extensible LAN (VXLAN) is a encapsulation protocol for running an overlay network on existing Layer 3 infrastructure. An overlay network is a virtual network that is built on top of existing network Layer 2 and Layer 3 technologies to support elastic compute architectures.
MTU – The Maximum Transmission Unit, or the size of the payload (in bytes) that will be used within the frame. The recommended value is 1600, which allows for the overhead incurred by VXLAN encapsulation.
James Cruickshank

18. ESXi001
ESXi002 IP IP IP IP
VLAN 400

19. Segment ID
Ok… It’s now time to create Segment IDs and this is one of the many huge advantages of SDN! In a way, you can think of these like VLANs for VXLAN … except you can have 16,777,216 of them.
Although it is technically possible to use values between 1 and 16 million, VMware has decided to start the count at This was done to avoid any confusion between a VLAN ID, which range from 1 to 4094 and a VXLAN ID (or VNI – VXLAN Network Identifier)

20. Transport Zone
James Cruickshank

21. Non-Production Cluster
Edge Cluster
Global Transport Zone
Production WAPP Logical Switch
Non-Production WAPP Logical Switch

22. Edge Clusters (P and NP) Non-Production Cluster
Production Transport Zone
Non-Production Transport Zone
Production WAPP Logical Switch
Non-Production WAPP Logical Switch

23. Production WAPP Logical Switch (Universal)
DataCenter 1
DataCenter 2
Production Cluster
Edge Cluster
Non-Production Cluster
Production Cluster
Edge Cluster
Non-Production Cluster
Global Transport Zone
Global Transport Zone
Universal Transport Zone
Production WAPP Logical Switch (Universal)
Non-Production WAPP Logical Switch (Global)
Non-Production WAPP Logical Switch (Global)

25. Logical Switch and Distributed Logical Routing
James Cruickshank

26. ESXi001
ESXi002 DB DB
WAPP
WAPP IP IP IP IP
VLAN 400

27. ESXi001
ESXi002 DB DB
WAPP
WAPP IP IP IP IP
VLAN 400

28. ESXi001
ESXi002 DB DB
WAPP
WAPP IP IP IP IP
VLAN 400

29. ESXi001
ESXi002 DB DB
WAPP
WAPP IP IP IP IP
VLAN 400

30. ESXi001
ESXi002 DB DB
WAPP
WAPP IP IP IP IP
VLAN 400

31. ESXi001
ESXi002 DB DB
WAPP
WAPP
/24
/24
/24
/24 IP IP IP IP
VLAN 400

32. ESXi001
ESXi002 DB DB
WAPP
WAPP
/24
/24
/24
/24 IP IP IP IP
VLAN 400

33. ESXi001
ESXi002 DB DB
WAPP
WAPP
/24
/24
/24
/24 IP IP IP IP
VLAN 400

34. Edge Service Gateway
James Cruickshank

35. NSX Controller Cluster
Management Plane
vSphere API
REST API
vCenter Server
NSX Manager
API
Control Plane
NSX Controller Cluster
Data Plane
VSFWD
NETCPA
Distributed Routing
VXLAN
Distributed Firewall
NSX Edge Service Gateway

36. NAT
James Cruickshank

37. DNAT

38. SNAT

39. Load Balancing
James Cruickshank

40. In Line

41. One Armed

42. North/South vs East/West Routing

43. ESXi001 ESXi002 ESXi003 ESXi004 Edge Cluster 10.10.10.254/24
/24
/24
/24
/24
/24
/24
/24
/24
/24
/24
/24
/24
Edge Cluster

44. ESXi001 ESXi002 ESXi003 ESXi004 Edge Cluster 10.10.10.254/24
/24
/24
/24
/24
/24
/24
/24
/24
/24
/24
/24
/24
Edge Cluster

45. Control Plane Protocol
Management Plane
vSphere API
REST API
vCenter Server
NSX Manager
API
Control Plane
Control Plane Protocol
DLR Control VM
NSX Controller Cluster
Data Plane
VSFWD
NETCPA
Distributed Routing
VXLAN
Distributed Firewall
NSX Edge Service Gateway

46. ESXi001 ESXi002 ESXi003 ESXi004 Edge Cluster 10.10.10.254/24
/24
/24
/24
/24
/24
/24
/24
/24
/24
/24
/24
/24
Edge Cluster

47. Forwarding IP Address Protocol IP Address
ESXi001
ESXi002
ESXi003
ESXi004
/24
/24
/24
/24
/24
/24
/24
/24
/24
/24
/24
/24
Forwarding IP Address
/24
Protocol IP Address
Edge Cluster

48. Control Plane Protocol
Management Plane
vSphere API
REST API
vCenter Server
NSX Manager
API
Control Plane
Control Plane Protocol
DLR Control VM
NSX Controller Cluster
Data Plane
VSFWD
NETCPA
Distributed Routing
VXLAN
Distributed Firewall
NSX Edge Service Gateway

49. Central CLI VTEP/MAC/ARP tables from the NSX Controllers
Dynamic routing peer status
Routing tables
Distributed firewall vNIC rules and stats
Edge status
James Cruickshank

50. Populating NSX Controller Tables
James Cruickshank

51. VXLAN 5002 VXLAN 5001 MAC-VT-1 10.11 MAC-VT-2 10.12 MAC-VT-3 20.11
VM-A VM-A-IP
MAC: AA:AA
VM-B VM-B-IP
MAC: BB:BB
VM-C VM-C-IP
MAC: CC:CC
VM-D VM-D-IP
MAC: DD:DD
VM-E VM-E-IP
MAC: EE:EE
VM-F VM-F-IP
MAC: FF:FF
VXLAN 5002
VXLAN 5001
MAC-VT-1
10.11
MAC-VT-2
10.12
MAC-VT-3
20.11
MAC-VT-4
20.12 x x
>show logical-switch controller master vni 5001 vtep
>show logical-switch controller master vni 5001 mac
>show logical-switch controller master vni 5001 arp
VNI – VTEP_IP – Network – VTEP-MAC – Conn_ID
VNI – VM_MAC – VTEP-IP – Conn_ID
VNI – VM-IP – VM-MAC – Conn_ID
>show logical-switch controller master vni 5002 vtep
>show logical-switch controller master vni 5002 mac
>show logical-switch controller master vni 5002 arp

52. VXLAN 5002 VXLAN 5001 MAC-VT-1 10.11 MAC-VT-2 10.12 MAC-VT-3 20.11
VM-A VM-A-IP
MAC: AA:AA
VM-B VM-B-IP
MAC: BB:BB
VM-C VM-C-IP
MAC: CC:CC
VM-D VM-D-IP
MAC: DD:DD
VM-E VM-E-IP
MAC: EE:EE
VM-F VM-F-IP
MAC: FF:FF
VXLAN 5002
VXLAN 5001
MAC-VT-1
10.11
MAC-VT-2
10.12
MAC-VT-3
20.11
MAC-VT-4
20.12 x x
>show logical-switch controller master vni 5001 vtep
>show logical-switch controller master vni 5001 mac
>show logical-switch controller master vni 5001 arp
VNI – VTEP_IP – Network – VTEP-MAC – Conn_ID
VNI – VM_MAC – VTEP-IP – Conn_ID
VNI – VM-IP – VM-MAC – Conn_ID
MAC-VT
AA:AA
VM-A-IP AA:AA 7
>show logical-switch controller master vni 5002 vtep
>show logical-switch controller master vni 5002 mac
>show logical-switch controller master vni 5002 arp

53. VXLAN 5002 VXLAN 5001 MAC-VT-1 10.11 MAC-VT-2 10.12 MAC-VT-3 20.11
VM-A VM-A-IP
MAC: AA:AA
VM-B VM-B-IP
MAC: BB:BB
VM-C VM-C-IP
MAC: CC:CC
VM-D VM-D-IP
MAC: DD:DD
VM-E VM-E-IP
MAC: EE:EE
VM-F VM-F-IP
MAC: FF:FF
VXLAN 5002
VXLAN 5001
MAC-VT-1
10.11
MAC-VT-2
10.12
MAC-VT-3
20.11
MAC-VT-4
20.12 x x
>show logical-switch controller master vni 5001 vtep
>show logical-switch controller master vni 5001 mac
>show logical-switch controller master vni 5001 arp
VNI – VTEP_IP – Network – VTEP-MAC – Conn_ID
VNI – VM_MAC – VTEP-IP – Conn_ID
VNI – VM-IP – VM-MAC – Conn_ID
MAC-VT
AA:AA
VM-A-IP AA:AA 7
BB:BB
VM-B-IP BB:BB 7
>show logical-switch controller master vni 5002 vtep
>show logical-switch controller master vni 5002 mac
>show logical-switch controller master vni 5002 arp

54. VXLAN 5002 VXLAN 5001 MAC-VT-1 10.11 MAC-VT-2 10.12 MAC-VT-3 20.11
VM-A VM-A-IP
MAC: AA:AA
VM-B VM-B-IP
MAC: BB:BB
VM-C VM-C-IP
MAC: CC:CC
VM-D VM-D-IP
MAC: DD:DD
VM-E VM-E-IP
MAC: EE:EE
VM-F VM-F-IP
MAC: FF:FF
VXLAN 5002
VXLAN 5001
MAC-VT-1
10.11
MAC-VT-2
10.12
MAC-VT-3
20.11
MAC-VT-4
20.12 x x
>show logical-switch controller master vni 5001 vtep
>show logical-switch controller master vni 5001 mac
>show logical-switch controller master vni 5001 arp
VNI – VTEP_IP – Network – VTEP-MAC – Conn_ID
VNI – VM_MAC – VTEP-IP – Conn_ID
VNI – VM-IP – VM-MAC – Conn_ID
MAC-VT
AA:AA
VM-A-IP AA:AA 7
MAC-VT
BB:BB
VM-B-IP BB:BB 7
EE:EE
VM-E-IP EE:EE 9
>show logical-switch controller master vni 5002 vtep
>show logical-switch controller master vni 5002 mac
>show logical-switch controller master vni 5002 arp

55. VXLAN 5002 VXLAN 5001 MAC-VT-1 10.11 MAC-VT-2 10.12 MAC-VT-3 20.11
VM-A VM-A-IP
MAC: AA:AA
VM-B VM-B-IP
MAC: BB:BB
VM-C VM-C-IP
MAC: CC:CC
VM-D VM-D-IP
MAC: DD:DD
VM-E VM-E-IP
MAC: EE:EE
VM-F VM-F-IP
MAC: FF:FF
VXLAN 5002
VXLAN 5001
MAC-VT-1
10.11
MAC-VT-2
10.12
MAC-VT-3
20.11
MAC-VT-4
20.12 x x
>show logical-switch controller master vni 5001 vtep
>show logical-switch controller master vni 5001 mac
>show logical-switch controller master vni 5001 arp
VNI – VTEP_IP – Network – VTEP-MAC – Conn_ID
VNI – VM_MAC – VTEP-IP – Conn_ID
VNI – VM-IP – VM-MAC – Conn_ID
MAC-VT
AA:AA
VM-A-IP AA:AA 7
MAC-VT
BB:BB
VM-B-IP BB:BB 7
EE:EE
VM-E-IP EE:EE 9
>show logical-switch controller master vni 5002 vtep
>show logical-switch controller master vni 5002 mac
>show logical-switch controller master vni 5002 arp
MAC-VT
FF:FF
VM-F-IP FF:FF

56. Controller Disconnected Operation (CDO) Mode
Provides additional resiliency for the NSX Control Plane
Targets specific scenario where control plane connectivity is lost to a host
Possibility of host losing connectivity to controllers
Possibility of the controller cluster being down
CDO creates and maintains a “global” VTEP list of all hosts attached to a CDO enabled Transport Zone. This is used for BUM traffic when control plane connectivity is lost.
James Cruickshank

57. The multitenancy problem…
James Cruickshank

58. Tenant 1 - Production Cluster Tenant 1 - Non-Production Cluster
Edge Cluster
Tenant 1 - Production Cluster
Tenant 1 - Non-Production Cluster
Tenant 2 - Production Cluster
Tenant 2 - Non-Production Cluster

60. Tenant 1 Tenant 2 Edge Cluster Tenant 1 - Production Cluster
Tenant 1 - Non -Production Cluster
Edge Cluster
Tenant 2 - Production Cluster
Tenant 2 - Non -Production Cluster

61. Edge Cluster

62. Thanks…
James Cruickshank