Virtual Machine Queue Architecture Review Ali Dabagh Architect Windows Core Networking Don Stanwyck Sr. Program Manager NDIS Virtualization

Agenda Introduction Network device sharing in virtualization environment Performance bottlenecks Virtual Machine Queue (VMQ) Architecture NIC embedded switch Coexistence with offload technologies Call to action Q&A

Why VMQ? Virtualization is now Networking optimization supports virtualization Virtualization impacts performance – VMQ helps overcome those effects VMQ supports Live Migration VMQ coexists with other optimizations Broad industry support

Terminology Virtual Service Provider (VSP) A driver running in the parent partition Provides a specific functionality to Virtual Machines Network VSP (NVSP) implemented as an NDIS intermediate driver Virtual Service Client (VSC) A driver running in the child partition Usually provides a device class functionality in child partitions Communicates with its VSP counterpart in the parent partition Network VSC (NVSC) implemented as an NDIS virtual miniport driver Virtual Machine Bus (VMBus) A communication bus for passing control and data messages between child and parent partition VMBus Channel A communication channel created between a VSC in child partition and VSP in parent partition

Synthetic Device Data Path Parent PartitionVM1VM2 Ethernet VM BUS TCP/IP VM NIC 1VM NIC 2 Network Virtual Service Provider NIC Miniport Driver Routing VLAN Filtering Data Copy Port 1Port 2

VMQ Concepts: Receive Queues Receive Queues Are allocated by Network Virtual Service Provider Collect network packets that pass filter tests Properties of a receive queue Unique queue ID per miniport adapter Interrupt processor affinity Filters Receive buffers Default Queue Always exists Collects packets that are not collected by any other queue

VMQ Concepts: Receive Filters Receive filters are set on receive queues One or more “network header field tests” Unique filter ID per miniport adapter Specifying network headers in a filter VMQ uses MAC header Specifying fields within the headers VMQ uses destination MAC address and VLAN tag Setting multiple tests in one receive filter A packet must pass all the network header field tests Setting multiple filters on the same queue Packet is accepted if it passes any of those filters Specifying test operation and test value VMQ uses “Equal” test operation

Receive Filters and Receive Queues

Packet Filtering Data Flow

Virtual Machine Queue Overview Classifying received packets Parsing the packets in hardware Setting received packet queue ID Grouping received packets All packets in a receive indication call are for the same VM NIC Applying VLAN filtering in hardware Packets with invalid VLAN IDs are dropped by the NIC Concurrent receive processing for multiple VMs Every queue is serviced by a different processor Avoiding copy from NIC receive buffers to VM address space Using switch on the NIC to do route lookup on transmit

Synthetic Device Data Paths With VMQ Parent PartitionVM1VM2 Ethernet VM BUS TCP/IP VM NIC 1VM NIC 2 Network Virtual Service Provider Miniport Driver Switch/Routing unit Q1 Default Queue Q2 Routing VLAN filtering Data Copy Port 1Port 2 NIC

Queue 1 Using VM Address Space for Receive Buffers VM1VM2 D D D D D Parent Partition H H H H H Queue 2 D D D D D H H H H H Address Space

NIC Embedded Switch Receive queues are paired with transmit queues Each queue pair is a switch port No MAC address learning is required The MAC address of each port is the one set on the receive queue Switch inspects destination MAC address +VLAN ID of transmit packets If they pass the filter set on a receive queue, DMA to that queue Otherwise send the packet on the wire Advantages in VM to VM communication Avoids route lookup in software Avoids a packet copy Offload support in hardware Advantages in VM to wire communication Avoids route lookup in software

Coexistence with Other Offload Technologies Transmit and receive checksum Still needs to be calculated and verified Should work for both VM-to-VM and VM-to-wire data flow Large send offload Should work for both VM-to-VM and VM-to-wire data flow Jumbo frame support Maximum packet size Scatter/gather support for received packets Chimney offload If NIC supports both VMQ and Chimney, VMQ is used

Summary VMQ improves network throughput Distributes processing of network traffic for multiple VMs among multiple processors VMQ reduces CPU utilization by: Offloading packet classification to hardware Avoiding network data copy Avoiding route lookup on transmit path VMQ is compatible with most task offloads VMQ is secure VMQ supports live migration

Call to Action Virtualization is NOW Network optimizations are crucial for virtualization support These vendors are among those committed to supporting VMQ in their NICs: Alacritech Broadcom Intel Neterion ServerEngines Solarflare Build/update all your drivers to be NDIS 6.20 compliant

Resources Windows 7 WDK will be available as of WinHEC Windows Logo Program Web site: NDIS 6 Feedback alias:

Related Sessions SessionDay / Time Virtual Machine Queue Driver DevelopmentTues. 4-5 Developing TCP Chimney Drivers for Windows 7Tues Driver ScalabilityMon and Tues NDIS 6.20 OverviewMon. 9:45-10:45 and Tues. 9:45-10:45 Windows Logo Program Tests for NDISMon and Wed. 9:45-10:45 Discussion: Porting Drivers to NDIS 6.20Wed. 8:30-9:30 Lab: How to Debug an NDIS 6.20 DriverMon. 2:45-3:45 and Wed. 9:45-10:45