2. Implementing Efficient RSS Capable Hardware and Drivers for Windows 7
Rade Trimceski
Program Manager
Windows Core Networking

3. Agenda Receive-Side Scaling (RSS) and system scalability
RSS enhancements in Windows Server 2008
Dynamic MSI-X mapping
Processor selection algorithm
RSS enhancements in Windows 7
Support for more than 64 processors
Per interface indirection table
Support for 2-tuple hash
NUMA support
Load balancing algorithm
New DDIs for processor topology and RSS information
Summary
Q&A

4. Scalability on Highly Parallel Systems
Systems are getting larger and more parallel
Devices that handle lots of data should use processors, memory, and I/O links efficiently for optimal system scalability
Real, measurable performance differences are available to drivers that implement scalability techniques
RSS remains the best solution for addressing the scalability challenges of networking subsystems

5. RSS Enhancements in Windows Server 2008
MSI-X support
Not present in Windows Server 2003 RSS
Dynamic MSI-X resource configuration
Changing the processor affinity of an RSS queue at runtime
More robust load balancing algorithm
The activity level of a connection is taken into account
Manageability
Using netsh interface to turn RSS on/off

6. Configuring MSI-X Resources
Available starting with Windows Server 2008
MSI-X messages vs. MSI-X table entries
Miniport driver can allocate an MSI-X message for each processor
Initial mapping of MSI-X table entries to MSI-X messages
Remapping MSI-X table entries at runtime
Setting any arbitrary affinity at run-time
Following RSS indirection table
Also available: interfaces to mask/unmask an MSI-X table entry

7. RSS Enhancements in Windows 7
Per-interface indirection table
Removing limitations on using a global table
Driver workarounds can be removed
Number of RSS processors is always less than or equal to the number of RSS queues
No need to choose a subset of processors in the indirection table
Support for more than 64 processors
In all RSS and processor-related DDIs
NUMA support
Fallback to 2-tuple hash when necessary
More robust load balancing algorithm used by stack
Processor topology and RSS information made available to drivers

8. System Processor Topology
New DDI returns the processor topology
NdisGetProcessorInformationEx
Can be called by NDIS 6.20 miniport and protocol drivers
NdisGetProcessorInformation is deprecated for NDIS 6.20 drivers
Group and processor number
Supporting more than 64 processors
Socket, core, and hyper-thread information
NUMA Node ID for each processor
NUMA “distance” of the NIC to each processor
RSS processor information is returned by another DDI - NdisGetRssProcessorInformation
Formerly returned by NdisGetProcessorInformation

9. Per-Interface RSS Information
New DDI to return the RSS processor information
NdisGetRssProcessorInformation
Can be used by NDIS 6.20 miniport and protocol drivers
Returns per interface RSS information
RSS base processor
Number of RSS processors
NUMA node of the NIC if known
RSS processor set for the interface is sorted based on preference

10. Initial Per-Interface RSS Processor Set
RSS base processor
Default is zero
Can be set by a standard registry key
Maximum number of RSS processors
Limited by the number of the RSS queues
Can further be controlled by a standard registry key
Default is based on MSI-X capabilities and speed
NDIS using only one processor from each core
Not using hyper-threaded processors on the same core
NUMA systems
NDIS sets the “preference index”

11. More than 64 Processor Support
Changes to RSS parameters
Include both processor group and processor numbers
RSS keywords
Support both group and processor number
Processor-related DDIs
Queue DPC /current processor DDIs
RSS information DDIs
MSI-X resource allocation and configuration
RSS on processors in different groups

12. Using Two-Tuple Hash Scenario: IPsec packets
Ideal solution is to support IPsec task offload
Ideally do IPsec decryption before RSS hash calculation
Scenario: Unknown options and extension headers
NIC can fall back to calculating hash on source and destination IP only
TCP will revert to using 2-tuple hash also
Using 4-tuples is still preferred
Using 2-tuples does not always provide the desired granularity in hash value

13. RSS Configuration Global keywords affect all interfaces
Standard keywords per interface
Set the RSS base processor group and number
Set the number of RSS Processors
Default RSS parameters based on:
Speed of the NIC
Number of NIC queues
MSI-X support
Enable/disable RSS through netsh
netsh int tcp set global rss=enabled/disabled

14. Diagnostics and Tools Per-processor performance counters NICSTATS tool
Interrupts
DPCs
CPU cycles spent in miniport driver, stack, or NDIS
Packets received/transmitted
NICSTATS tool
Display RSS capabilities
Display current RSS parameters
Display MSI-X capabilities
Enable/disable RSS through netsh
netsh int tcp set global rss=enabled/disabled

15. Summary
As systems become larger and more parallel, efficient RSS implementation becomes even more important
Windows 7 provides a more robust RSS support
Improved load-balancing algorithm
> 64 processor support
Using two-tuple hash when necessary
Windows 7 makes it easier to develop RSS-capable drivers
Per-interface indirection table
Provides information on system processor topology and RSS processor set
Windows 7 makes it easier to configure and manage RSS
Per-interface reserved registry keywords, diagnostic tools, perfmon counters

16. Resources Windows 7 WDK will be available as of WinHEC
Windows Logo Program Web site:
NDIS 6 Feedback alias:

17. Related Sessions Session Day / Time
Virtual Machine Queue Architecture Review
Tues. 2:45-3:45
Virtual Machine Queue Driver Development
Tues. 4-5
Driver Scalability
Mon and
Tues
Developing TCP Chimney Drivers for Windows 7
NDIS 6.20 Overview
Mon. 9:45-10:45 and
Tues. 9:45-10:45
Windows Logo Program Tests for NDIS
Wed. 9:45-10:45
Discussion: Porting Drivers to NDIS 6.20
Wed. 8:30-9:30
Lab: How to Debug an NDIS 6.20 Driver
Mon. 2:45-3:45 and