Didier Van Hoye Technical FGIA MVP – Virtual Machine Microsoft Extended Experts Team

In the host networking stack In the NICs In the switches & routers

Receive Side Scaling (RSS) Receive Segment Coalescing (RSC) Dynamic Virtual Machine Queuing (DVMQ) Single Root I/O Virtualization (SR-IOV) NIC TEAMING RDMA/Multichannel support for virtual machines on SMB3.0 DHCP Guard/Router Guard/Port Mirroring

RSS exists for many years. Windows Server 2012 takes RSS to the next generation of servers Spreads interrupts across all available CPUs Even for those very large scale hosts RSS now works across k-groups Even RSS is “Numa Aware” to optimize performance Now load balances UDP traffic across CPUs

Coalesces packets in the NIC so the stack processes fewer headers Multiple packets belonging to connection that arrive within a single interrupt are coalesced to a larger packet (max of 64 K) by the NIC 10 – 30% improvement in I/O overhead

VMQ is to virtualization what RSS is to native workloads Dynamic VMQ reassigns available queues based on changing networking demands of the VMs

No VMQ Adaptive processing = optimal performance across changing workloads Root Partition Physical NIC CPU 0 CPU 0 CPU 1 CPU 1 CPU 2 CPU 2 CPU 3 CPU 3 Static VMQ Root Partition Physical NIC CPU 0 CPU 0 CPU 1 CPU 1 CPU 2 CPU 2 CPU 3 CPU 3 Dynamic VMQ Root Partition Physical NIC CPU 0 CPU 0 CPU 1 CPU 1 CPU 2 CPU 2 CPU 3 CPU 3 Root Partition Physical NIC CPU 0 CPU 0 CPU 1 CPU 1 CPU 2 CPU 2 CPU 3 CPU 3

Windows Server 2012 supports direct device assignment to virtual machines without compromising flexibility Network I/O path without SRIOV Network I/O path with SRIOV Host Root Partition Hyper-V Switch Physical NIC Virtual Machine Virtual NIC Routing VLAN Filtering Data Copy Routing VLAN Filtering Data Copy Host Root Partition Hyper-V Switch SR-IOV Physical NIC Virtual Machine Virtual Function Routing VLAN Filtering Data Copy Routing VLAN Filtering Data Copy

Reduces CPU utilization for processing network traffic Reduces latency of network path Increases throughput Supports Live Migration Requires: – Chipset: Interrupt and DMA remapping – BIOS Support – CPU: Hardware virtualization, EPT or NPT DIRECT DEVICE ASSIGNMENT TO VIRTUAL MACHINES WITHOUT COMPROMISING FLEXIBILITY Network I/O path with SR-IOV Network I/O path without SR-IOV Physical NIC Root Partition Hyper-V Switch Routing VLAN Filtering Data Copy Routing VLAN Filtering Data Copy Virtual Machine Virtual NIC SR-IOV Physical NIC Virtual Function VMBUS

Windows Server 8 – Developer Preview Virtual Machine Network Stack Software NIC Enable IOV (VM NIC Property) Virtual Function is “Assigned” “NIC” automatically created Traffic flows through VF Turn On IOV Switch back to Software path Reassign Virtual Function Assuming resources are available Migrate as normal Live MigrationPost Migration Remove VF from VM VM has connectivity even if Switch not in IOV mode IOV physical NIC not present Different NIC vendor Different NIC firmware SR-IOV Physical NIC Physical NIC Software Switch (IOV Mode) SR-IOV Physical NIC Software path is not used Virtual Function “NIC”“NIC” Software NIC Virtual Function Software Switch (IOV Mode) “NIC”“NIC”

Even when hardware fails … … our customers want continuous availability Windows Server 8 – Developer Preview Tenant 2: Multiple VM Workloads Data Center Tenant 1: Multiple VM Workloads TEAMING

Customers are dealing with way to many issues. NIC vendors would like to get rid of supporting this. Microsoft needs this to be competitive & complete the solution stack. No more 3 rd party drivers & utilities

Hyper-V Extensible Switch Network switch IM MUX Protocol edge Virtual miniport 1 Port 1 Port 2 Port 3 LBFO Configuration DLL LBFO Admin GUI Kernel mode User mode WMI IOCTL NIC 1 NIC 2 NIC 3 Multiple modes: switch dependent and switch independent Hashing modes: port and 4-tuple Active -Active and Active - Standby LBFO Provider Frame distribution/aggregation Failure detection Control protocol implementation Frame distribution/aggregation Failure detection Control protocol implementation

Parent NIC TeamingGuest NIC Teaming Hyper-V virtual switch VM (Guest Running Any OS) SR-IOV NIC LBFO Teamed NIC SR-IOV Not exposed Hyper-V virtual switch VM (Guest Running Windows Server 2012) LBFO Teamed NIC Hyper-V virtual switch SR-IOV NIC

Addresses congestion in network stack by offloading the stack to the network adapter Great for storage traffic: high throughput with low CPU utilization SMB-Direct uses new RDMA capability if the NICs support this Windows Server 2012 now supports RDMA low latency, high speed application-to-application data transfer

DCTCP/DCB Consistent Device Naming Network virtualization Generic Routing Encapsulation (GRE) IPSEC Task Offload for Virtual Machines (IPsecTOv2) Wireless Network Support

1Gbps flow controlled by TCP Requires 400 to 600KB of memory TCP saw tooth visible 1 Gbps flow controlled by DCTCP Requires 30KB of memory Smooth

W2K12 deals with network congestion by reacting to the degree & not merely the presence of congestion. DCTCP aims to achieve low latency, high burst tolerance, and high throughput, with small buffer switches. Requires Explicit Congestion Notification (ECN, RFC 3168) capable switches Algorithm enabled when it makes sense (low round trip times, i.e. in the data center)

Running out of buffer in a switch gets you in to stop/go hell by getting a boatload of green, orange & red lights along your way Big buffers mitigate this but are very expensive

You want to be in a green wave Windows Server 2012 & ECN provides network traffic control

Prevents congestion in NIC & network by reserving bandwidth for particular traffic types Windows 2012 provides support & control for DCB, tags packets by traffic type Provides lossless transport for mission critical workloads

1.Enhanced Transmission Selection (IEEE 802.1Qaz) 2.Priority Flow Control (IEEE 802.1Qbb) 3.(Optional) Datacenter Bridging Exchange protocol 4.(Not required) Congestion Notification (IEEE 802.1Qau)

Multi-tenant scenarios: hide the tenant’s multi- premise networking from the datacenter’s networking. GRE (RFCs 2784 & 2890) provides the mechanism to tunnel tenant networks over the datacenter network GRE breaks today’s task offloads if the NIC vendors don’t support GRE offload



IPsec is a CPU intensive workload => Offload to NIC In demand due to compliance (SOX, HIPPA, etc.) IPsec is required & needed for secured operations Only available to host/parent workloads in W2K8R2 – Now extended to VMs – Managed by the Hyper-V switch

Manage the Network Bandwidth with a Maximum and a Minimum value SLAs for hosted Virtual Machines Control per VMs and not per HOST

Physical NIC Root Partition Extensible Switch Extension Protocol Extension Miniport Capture Extensions WFP Extensions Filtering Extensions Forwarding Extensions Host NIC VM NIC VM1 VM NIC VM2 Capture extensions can inspect traffic and generate new traffic for report purposes Capture extensions do not modify existing Extensible Switch traffic Example: sflow by inMon Windows Filter Platform (WFP) Extensions can inspect, drop, modify, and insert packets using WFP APIs Windows Antivirus and Firewall software uses WFP for traffic filtering Example: Virtual Firewall by 5NINE Software Filtering extensions can also be implemented using NDIS filtering APIs Example: VM DoS Prevention by Broadcom Forwarding extensions direct traffic, defining the destination(s) of each packet Forwarding extensions can capture and filter traffic Examples: Cisco Nexus 1000V and UCS NEC OpenFlow Capture Extensions WFP Extensions Filtering Extensions Forwarding Extensions Filtering Engine BFE Service Firewall Callout