1. Windows Server 2012 NIC Teaming and Multichannel Solutions Rick Claus Sr. Technical Evangelist @RicksterCDN http://RegularITGuy.com WSV321

4. NIC Teaming

8. Team members --or-- Network Adapters Team Team Interfaces, Team NICs, or tNICs

9. Switch dependent team Switch independent team

11. Address HashHyper-V port Switch Independent Sends on all active members, receives on one member (primary member) Sends on all active members, receives on all active members, traffic from same port always on same NIC Switch DependentSends on all active members, receives on all active members, inbound traffic may use different NIC than outbound traffic for a given stream (inbound traffic is distributed by the switch) All outbound traffic from a port will go on a single NIC. Inbound traffic may be distributed differently depending on what the switch does to distribute traffic

13. Address HashHyper-V port Switch Independent Sends on all active members, receives on one member (primary member) Sends on all active members, receives on all active members, traffic from same port always on same NIC Switch DependentSends on all active members, receives on all active members, inbound traffic may use different NIC than outbound traffic for a given stream (inbound traffic is distributed by the switch) All outbound traffic from a port will go on a single NIC. Inbound traffic may be distributed differently depending on what the switch does to distribute traffic Sends on all active members using the selected level of address hashing (defaults to 4-tuple hash). Because each IP address can only be associated with a single MAC address for routing purposes, this mode receives inbound traffic on only one member (the primary member). Best used when: a) Native mode teaming where switch diversity is a concern; b) Active/Standby mode c) Servers running workloads that are heavy outbound, light inbound workloads (e.g., IIS)

14. Address HashHyper-V port Switch Independent Sends on all active members, receives on one member (primary member) Sends on all active members, receives on all active members, traffic from same port always on same NIC Switch DependentSends on all active members, receives on all active members, inbound traffic may use different NIC than outbound traffic for a given stream (inbound traffic is distributed by the switch) All outbound traffic from a port will go on a single NIC. Inbound traffic may be distributed differently depending on what the switch does to distribute traffic Sends on all active the hashed Hyper-V switch port. Each Hyper- V pormembers using t will be bandwidth limited to not more than one team member’s bandwidth. Because each VM (Hyper-V port) is associated with a single NIC, this mode receives inbound traffic for the VM on the same NIC it sends on so all NICs receive inbound traffic. This also allows maximum use of VMQs for better performance over all. Best used for teaming under the Hyper-V switch when - number of VMs well-exceeds number of team members - restriction of a VM to one NIC’s bandwidth is acceptable

15. Address HashHyper-V port Switch Independent Sends on all active members, receives on one member (primary member) Sends on all active members, receives on all active members, traffic from same port always on same NIC Switch DependentSends on all active members, receives on all active members, inbound traffic may use different NIC than outbound traffic for a given stream (inbound traffic is distributed by the switch) All outbound traffic from a port will go on a single NIC. Inbound traffic may be distributed differently depending on what the switch does to distribute traffic Sends on all active members using the selected level of address hashing (defaults to 4-tuple hash). Receives on all ports. Inbound traffic is distributed by the switch. There is no association between inbound and outbound traffic. Best used for: - Native teaming for maximum performance and switch diversity is not required; or - teaming under the Hyper-V switch when an individual VM needs to be able to transmit at rates in excess of what one team member can deliver

16. Address HashHyper-V port Switch Independent Sends on all active members, receives on one member (primary member) Sends on all active members, receives on all active members, traffic from same port always on same NIC Switch DependentSends on all active members, receives on all active members, inbound traffic may use different NIC than outbound traffic for a given stream (inbound traffic is distributed by the switch) All outbound traffic from a port will go on a single NIC. Inbound traffic may be distributed differently depending on what the switch does to distribute traffic Sends on all active members using the hashed Hyper-V switch port. Each Hyper-V port will be bandwidth limited to not more than one team member’s bandwidth. Receives on all ports. Inbound traffic is distributed by the switch. There is no association between inbound and outbound traffic. Best used when: - Hyper-V teaming when VMs on the switch well-exceed the number of team members and - when policy calls for e.g., LACP teams and when an individual VM does not need to transmit faster than one team member’s bandwidth

17. TEAM VLAN =42 Default (all but 42) TEAM VLAN =42 VLAN =99 Black hole TEAM Default Hyper-V switch

19. TEAM Default Hyper-V switch

20. TEAM VLAN =42 VLAN =99 VLAN =13 VLAN =3995

25. FeatureComments RSSProgrammed directly by TCP/UDP when bound to TCP/UDP. VMQProgrammed directly by the Hyper-V switch when bound to Hyper-V switch IPsecTO, LSO, Jumbo frames, all checksum offloads (transmit) Yes – advertised if all NICs in the team support it RSC, all checksum offloads (receive) Yes – advertised if any NICs in the team support it DCBYes – works independently of NIC Teaming RDMA, TCP Chimney offload No support through teaming SR-IOVTeaming in the guest allows teaming of VFs Network virtualizationYes

27. demo NIC Teaming

28. SMB Multichannel

29. Multiple RDMA NICs Multiple 1GbE NICs Single 10GbE RSS-capable NIC SMB Server SMB Client SMB Server SMB Client SMB Server SMB Client Sample Configurations Multiple 10GbE in a NIC team SMB Server SMB Client NIC Teaming Switch 10GbE Switch 10GbE NIC 10GbE NIC 10GbE NIC 10GbE NIC 10GbE Switch 10GbE Switch 10GbE NIC 10GbE NIC 10GbE NIC 10GbE NIC 10GbE NIC 10GbE NIC 10GbE NIC 10GbE NIC 10GbE Switch 1GbE Switch 1GbE NIC 1GbE NIC 1GbE NIC 1GbE NIC 1GbE Switch 1GbE Switch 1GbE NIC 1GbE NIC 1GbE NIC 1GbE NIC 1GbE Switch 10GbE/IB Switch 10GbE/IB NIC 10GbE/IB NIC 10GbE/IB NIC 10GbE/IB NIC 10GbE/IB Switch 10GbE/IB Switch 10GbE/IB NIC 10GbE/IB NIC 10GbE/IB NIC 10GbE/IB NIC 10GbE/IB Switch 10GbE Switch 10GbE RSS

30. SMB Server SMB Client Switch 10GbE Switch 10GbE NIC 10GbE NIC 10GbE NIC 10GbE NIC 10GbE SMB Server SMB Client Switch 10GbE Switch 10GbE NIC 10GbE NIC 10GbE NIC 10GbE NIC 10GbE CPU utilization per core Core 1 Core 2 Core 3 Core 4 CPU utilization per core Core 1 Core 2 Core 3 Core 4 RSS

31. 1 session, with Multichannel 1 session, without Multichannel SMB Server 1 SMB Client 1 Switch 10GbE Switch 10GbE SMB Server 2 SMB Client 2 NIC 10GbE NIC 10GbE NIC 10GbE NIC 10GbE Switch 10GbE Switch 10GbE NIC 10GbE NIC 10GbE NIC 10GbE NIC 10GbE Switch 10GbE Switch 10GbE Switch 10GbE Switch 10GbE NIC 10GbE NIC 10GbE NIC 10GbE NIC 10GbE NIC 10GbE NIC 10GbE NIC 10GbE NIC 10GbE SMB Server 1 SMB Client 1 Switch 10GbE Switch 10GbE SMB Server 2 SMB Client 2 NIC 10GbE NIC 10GbE NIC 10GbE NIC 10GbE Switch 10GbE Switch 10GbE NIC 10GbE NIC 10GbE NIC 10GbE NIC 10GbE Switch 10GbE Switch 10GbE Switch 10GbE Switch 10GbE NIC 10GbE NIC 10GbE NIC 10GbE NIC 10GbE NIC 10GbE NIC 10GbE NIC 10GbE NIC 10GbE RSS

32. Preliminary results based on Windows Server “8” Developer Preview

33. 1 session, with NIC Teaming and MC 1 session, with NIC Teaming, no MC SMB Server 1 SMB Client 1 SMB Server 2 SMB Client 2 NIC Teaming Switch 10GbE Switch 10GbE Switch 10GbE Switch 10GbE NIC 10GbE NIC 10GbE NIC 10GbE NIC 10GbE NIC 10GbE NIC 10GbE NIC 10GbE NIC 10GbE Switch 1GbE Switch 1GbE NIC 1GbE NIC 1GbE NIC 1GbE NIC 1GbE Switch 1GbE Switch 1GbE NIC 1GbE NIC 1GbE NIC 1GbE NIC 1GbE SMB Server 2 SMB Client 1 Switch 1GbE Switch 1GbE SMB Server 2 SMB Client 2 NIC 1GbE NIC 1GbE NIC 1GbE NIC 1GbE Switch 1GbE Switch 1GbE NIC 1GbE NIC 1GbE NIC 1GbE NIC 1GbE Switch 10GbE Switch 10GbE Switch 10GbE Switch 10GbE NIC 10GbE NIC 10GbE NIC 10GbE NIC 10GbE NIC 10GbE NIC 10GbE NIC 10GbE NIC 10GbE NIC Teaming

34. 1 session, with Multichannel 1 session, without Multichannel SMB Server 2 SMB Client 2 SMB Server 1 SMB Client 1 SMB Server 2 SMB Client 2 SMB Server 1 SMB Client 1 Switch 10GbE Switch 10GbE Switch 10GbE Switch 10GbE R-NIC 10GbE R-NIC 10GbE R-NIC 10GbE R-NIC 10GbE R-NIC 10GbE R-NIC 10GbE R-NIC 10GbE R-NIC 10GbE Switch 54GbIB Switch 54GbIB R-NIC 54GbIB R-NIC 54GbIB R-NIC 54GbIB R-NIC 54GbIB Switch 54GbIB Switch 54GbIB R-NIC 54GbIB R-NIC 54GbIB R-NIC 54GbIB R-NIC 54GbIB Switch 10GbE Switch 10GbE Switch 10GbE Switch 10GbE R-NIC 10GbE R-NIC 10GbE R-NIC 10GbE R-NIC 10GbE R-NIC 10GbE R-NIC 10GbE R-NIC 10GbE R-NIC 10GbE Switch 54GbIB Switch 54GbIB R-NIC 54GbIB R-NIC 54GbIB R-NIC 54GbIB R-NIC 54GbIB Switch 54GbIB Switch 54GbIB R-NIC 54GbIB R-NIC 54GbIB R-NIC 54GbIB R-NIC 54GbIB

35. Switch 1GbE Switch 1GbE SMB Server SMB Client NIC 1GbE NIC 1GbE NIC 1GbE NIC 1GbE Switch 1GbE Switch 1GbE Switch Wireless Switch Wireless SMB Server SMB Client NIC 1GbE NIC 1GbE NIC Wireless NIC Wireless NIC 1GbE NIC 1GbE Switch 1GbE Switch 1GbE SMB Server SMB Client NIC 1GbE NIC 1GbE NIC 1GbE NIC 1GbE Switch 10GbE Switch 10GbE SMB Server SMB Client R-NIC 10GbE R-NIC 10GbE R-NIC 10GbE R-NIC 10GbE Switch 1GbE Switch 1GbE SMB Server SMB Client NIC 1GbE NIC 1GbE NIC 1GbE NIC 1GbE Single NIC configurations where full bandwidth is already available without MC Configurations with different NIC type or speed SMB Server SMB Client Switch Wireless Switch Wireless NIC Wireless NIC Wireless NIC Wireless NIC Wireless Switch 10GbE Switch 10GbE NIC 10GbE NIC 10GbE NIC 10GbE NIC 10GbE Switch IB Switch IB R-NIC 32GbIB R-NIC 32GbIB R-NIC 32GbIB Switch 10GbE Switch 10GbE R-NIC 10GbE R-NIC 10GbE R-NIC 10GbE R-NIC 10GbE RSS

36. Throughput Fault Tolerance for SMB Fault Tolerance for non-SMB Lower CPU utilization Single NIC (no RSS) ▲ Multiple NICs (no RSS) ▲▲ ▲ Multiple NICs (no RSS) + NIC Teaming ▲▲ ▲ Single NIC (with RSS) ▲ Multiple NICs (with RSS) ▲▲ ▲ Multiple NICs (with RSS) + NIC Teaming ▲▲ ▲ Single NIC (with RDMA) ▲ ▲ Multiple NICs (with RDMA) ▲▲ ▲▲

38. demo SMB Multichannel

39. Virtualizing Storage for Scale, Resiliency, and Efficiency http://go.microsoft.com/fwlink/?LinkID=254536 How to Configure Clustered Storage Spaces in Windows Server 2012 http://go.microsoft.com/fwlink/?LinkID=254538 Storage Spaces FAQ http://go.microsoft.com/fwlink/?LinkID=254539