Virtual Machine Queue Driver Development Sambhrama Mundkur Sr. Software Design Engineer Core Networking

Agenda Overview of Virtual Machine Queue (VMQ) VMQ capability advertisement VM Queue allocation and deletion Setting and clearing VMQ filters Shared memory resources allocation VMQ receive path Embedded NIC switch Diagnostics and Tools Questions

VMQ Architecture Parent Partition VM1VM2 Ethernet VM BUS TCP/IP VM NIC 1VM NIC 2 Network Virtual Service Provider Miniport Driver Switch/Routing unit Q1 Default Queue Q2 Routing VLAN filtering Data Copy Port 1Port 2 NIC

Overview of Virtual Machine Queue Performance bottlenecks in receive processing in virtualized environment Each incoming packet has to be examined to identify the target virtual machine (VM) Received data has to be copied from the parent partition’s memory to the child partition’s memory address space Lack of concurrency of interrupts and DPCs

Overview of Virtual Machine Queue (contd.) How does VMQ help? Determine the target VM by MAC filtering in hardware DMA received packets directly to the child partition’s memory address space Provide interrupt and DPC concurrency by indicating received packets based on the target VM Use switch capabilities of the NIC to provide VM to VM loopback functionality

VMQ Capability Advertisement Receive filter capabilities advertised during miniport adapter initialization Use standardized keywords to enable or disable VMQ functionality Capabilities advertised Enabled filters and queue types Number of queues and filters Supported queue properties Lookahead-split support NIC switch capabilities

Queue State Transitions

Queue Allocation Protocol driver issues an OID to allocate a queue and sets the parameters for the queue Per-queue receive indication flag Processor affinity Queue name Lookahead-split parameters NDIS generates a unique Queue ID and sets it before issuing the OID to the miniport driver Protocol driver can set filters after allocating the queue Protocol driver issues an OID to complete the queue allocation Miniport driver can allocate shared memory and other resources Default queue (Queue ID 0) is always allocated

Queue Deletion Protocol driver clears all filters before freeing the queue Protocol driver issues an OID to free the queue Miniport driver Must immediately stop DMA to shared memory resources allocated for this queue Must generate status indication to indicate that the DMA has been stopped Waits for all outstanding NET_BUFFER_LIST (NBLs) for this queue to be returned Frees the allocated shared memory and hardware resources Default queue can never be freed

Updating Queue Parameters Protocol driver can issue an OID to update the queue parameters Queue parameters that can be updated are Processor affinity Queue name

VMQ Filter Operations

Setting and Clearing Filters Filters can be set and cleared any time after the queue has been allocated Protocol issues an OID to set the filter and the filter’s parameters Queue ID Array of header field test parameters. For VMQ two header field tests can be set: Destination MAC address in the packet == specified MAC address VLAN ID in the packet == specified VLAN ID NDIS generates a unique filter ID and issues the OID to the miniport driver

Setting and Clearing Filters Miniport driver programs the hardware based on the filters All header field tests of a particular filter must match to indicate the packet to that queue Multiple filters can be set on a queue Packets must be indicated from the queue if any of the filters pass Protocol driver can set filters on the default queue, too Protocol driver issues an OID to clear a filter Provides the Filter ID of the filter

Shared Memory Resource Allocation Miniport driver calls the new DDI, NdisAllocateSharedMemory, to allocate shared memory resources for a queue Initial allocation occurs when processing the queue allocation complete OID Miniport driver can allocate shared memory as long as the queue has not been freed Miniport driver specifies: Queue ID Length Usage – Lookahead, post-Lookahead etc. NdisAllocateSharedMemory returns: Virtual address of the allocated memory Scatter/gather List Shared memory handle (used in receive indications) Allocation handle (used to free the memory)

Shared Memory Resources If lookahead-split is not enabled, specify that the memory will be used for receive buffer If lookahead-split has been enabled for the queue, allocate two shared memory buffers Specify that usage is for the lookahead buffer Specify that usage is for the post-lookahead (data) buffer Call NdisFreeSharedMemory to free the memory Driver frees the resources when the queue is freed Driver must wait for all NBLs using the shared memory to be returned before calling this DDI

VMQ Packet Structures

VMQ Receive Path Indicate packets on a queue only if it passes all the filter field tests for a particular filter If per-queue receive indication flag is set: NBLs from this queue must not be mixed with others Must set ‘single queue indication’ flag in the Receive Flags Must set flag to indicate that the shared memory information is valid Out-of-band (OOB) information Specify the Queue ID in packet OOB information For VMQ, set the Filter ID in the OOB to zero

VMQ Receive Path Specify the SharedMemoryInfo in the NET_BUFFER Set SharedMemoryHandle returned by NdisAllocateSharedMemory Offset is the offset from the beginning of the shared memory buffer If lookahead split is enabled for the queue, each NET_BUFFER: Must have two MDLs and corresponding SharedMemoryInfo structures The post-lookahead buffer must have a backfill The protocol driver copies the contents of the lookahead buffer to the backfill Backfill must exist even if the packet is entirely in the lookahead buffer MDL Length and DataLength in the packet must be set as before

Queue Statistics NIC must maintain following per-queue statistics: Number of transmitted and received packets Number of transmitted and received bytes NIC can maintain per-packet type statistics The statistics can be queried using the new statistics OID This is a mandatory Direct OID request VMQ miniport driver must provide a Direct OID request handler Return statistics for a particular queue based on Queue ID Default queue can be queried, too

Embedded NIC Switch Embedded switch on the NIC can assist in advanced features Advertise support during miniport adapter initialization Loopback support Protocol driver issues an OID to enable these features Loopback On transmit, Queue ID of the outgoing queue is set in packet OOB data NIC indicates the packet if it matches filters set on any other queue If no match is found, NIC transmits the packet on the wire

Diagnostics OIDs and WMI GUIDs Enumerate the queues allocated on a miniport adapter Query the parameters of a particular queue Enumerate the filters allocated on a particular queue Query the parameters of a particular filter Handled by NDIS NDIS kernel debugger extensions Performance counters exposed by NDIS

Call to Action Update drivers to NDIS 6.20 to take advantage of Windows 7 functionality Implement VMQ support in the miniport driver Attend other related sessions in the DDC

Resources Windows 7 WDK will be available as of WinHEC Windows Logo Program Web site: NDIS 6 feedback alias:

Related Sessions SessionDay / Time Network Power Management for Windows 7Tues. 5:15-6:15 Windows Logo Program Tests for NDISMon and Wed. 9:45-10:45 Discussion: Porting Drivers to NDIS 6.20Wed. 8:30-9:30 Lab: How to Debug an NDIS 6.20 DriverMon. 2:45-3:45 and Wed. 9:45-10:45 Mobile Broadband Driver Development for Windows 7Mon. 1:30-2:30 Virtual Machine Queue Architecture ReviewTues. 2:45-3:45 Virtual Machine Queue Driver DevelopmentTues. 4-5 Implementing Efficient RSS Capable Hardware and Drivers for Windows 7Tues. 1:30-2:30 Developing TCP Chimney Drivers for Windows 7Tues Driver ScalabilityMon and Tues

Questions?