WHDC PowerPoint Template Notes & Handouts Monday, April 24, 2017 Advanced Storport Development Matthew D Hendel Software Engineer BaseOS/Drivers

WHDC PowerPoint Template Notes & Handouts Contents WHDC PowerPoint Template Notes & Handouts Monday, April 24, 2017 This presentation is for driver writers already familiar with Storport. The material covered will be on advanced Storport topics such as the best usage of Build I/O and Start I/O routines, error handling, queue implementation, as well as dispatch level processing in Storport and implementation of MSI interrupts in Storport.

WHDC PowerPoint Template Notes & Handouts Overview WHDC PowerPoint Template Notes & Handouts Monday, April 24, 2017 Full Duplex Drivers Build I/O routine Example usage of Full Duplex and Build I/O routines Dispatch Level Processing Passive Initialization Routine DPC Routines SpinLocks Message Signaled Interrupts in Storport

WHDC PowerPoint Template Notes & Handouts Full Duplex Drivers WHDC PowerPoint Template Notes & Handouts Monday, April 24, 2017 Storport adds the notion of “Full Duplex” I/O to the miniport model In Full Duplex mode interrupts may occur while in StartIo routine Simultaneous access to I/O submission and I/O completion Miniport authors are responsible for synchronizing access to data structures that are shared between StartIo and Interrupt routines Storport will not raise IRQL during I/O submission

Full Duplex Example: I2O Example Miniport WHDC PowerPoint Template Notes & Handouts Monday, April 24, 2017 I2O has separate submission and completion ports Inbound Post_List is used for command submission Outbound Post_List is used for completed commands Because the command submission and completion use separate lists the miniport’s Start I/O and Interrupt routines do not require any synchronization Inbound (to IOP) Free_List Post_List Outbound (from IOP) Free_List Post_List

Using a Build I/O Routine WHDC PowerPoint Template Notes & Handouts Monday, April 24, 2017 Miniports spend a significant amount of time in StartIo routines performing work that does not need to be serialized such as building scatter-gather lists and translating commands Since StartIo routines are serialized, this represents wasted time BuildIo routine is called before StartIo allowing miniport to perform per-I/O initialization and translation No locks are held during BuildIo routine, if a miniport needs to access external data structures during BuildIo routine it must manually synchronize using StorPortSynchronizeAccess or StorPortAcquireSpinLock

Build I/O Example: I2O Example Miniport WHDC PowerPoint Template Notes & Handouts Monday, April 24, 2017 I2O uses non-SCSI command set, and different scatter-gather list format For read and write requests, I2O miniport performs translation between SCSI and I2O commands in Build I/O routine I2O Build I/O routine performs the following: If SCSI Read command, translate SCSI Read to I2O read command If SCSI Write command, translate SCSI Write to I2O write command, also translate SCSI Force Unit Access flag to I2O write-thru flag Set appropriate I2O callback routine Translate Storport scatter-gather list to I2O scatter-gather list I2O command is built in SRB Extension

Example Of Build I/O Routine And Full Duplex WHDC PowerPoint Template Notes & Handouts Monday, April 24, 2017 Example illustrates how the use of a Build I/O routine and Full Duplex mode can improve performance Example parameters Start I/O routine takes 100 time units Interrupt routine takes 30 time units Thirty percent of Start I/O can be deserialized form other Start I/O work Start I/O is completely independent of interrupt Example shows two I/O commands issued and two interrupts handled

SCSIport Synchronization WHDC PowerPoint Template Notes & Handouts Monday, April 24, 2017 Access to miniport synchronized with interrupt spinlock Simple model is good for many adapters I/O Request 4 at time = 20 Complete at time = 260 Interrupt 1 at time = 50 Interrupt 2 at time = 120 I/O Request 3 at time = 0 Processor 1 I/O Request 3 ISR 1 Processor 2 I/O Request 4 ISR 2 Time 100 200 300 Total time is 260 units

Storport Synchronization Using Buildio Routine WHDC PowerPoint Template Notes & Handouts Monday, April 24, 2017 Breaking the I/O Request into Start and Build portions reduces time Build routine creates the lower level I/O packet Start issues the I/O request to the hardware I/O Request 4 at time = 20 Complete at time = 230 Interrupt 1 at time = 50 Interrupt 2 at time = 120 I/O Request 3 at time = 0 Processor 1 Build 3 StartIo 3 ISR 1 Processor 2 Build 4 StartIo 4 ISR 2 Time 100 200 300 Total time is 230 units

Storport Synchronization Using Buildio And Full Duplex WHDC PowerPoint Template Notes & Handouts Monday, April 24, 2017 Adding Full Duplex allows StartIo and interrupt routines to be deserialized I/O Request 4 at time = 20 Interrupt 1 at time = 50 Interrupt 2 at time = 120 Complete at time = 170 I/O Request 3 at time = 0 Processor 1 Build 3 StartIo 3 StartIo 4 Processor 2 Build 4 ISR 1 ISR 2 Time 100 200 300 Total time is 170 units

WHDC PowerPoint Template Notes & Handouts Error Handling WHDC PowerPoint Template Notes & Handouts Monday, April 24, 2017 Hierarchical reset instead of bus reset Perform reset of logical unit, failing that, reset of target, failing that reset of bus Much more efficient on non-SCSI buses Used by clusters to prevent resetting all targets on a bus

WHDC PowerPoint Template Notes & Handouts Reset Requests WHDC PowerPoint Template Notes & Handouts Monday, April 24, 2017 Logical unit reset Send asynchronously via SCSI REQUEST BLOCK All I/O to logical unit is blocked while handling reset request Default timeout of 30 seconds Bus reset routine invoked if not handled within timeout Target reset All I/O to adapter is blocked while handling the reset request Bus reset Sent synchronously via HwStorReset routine Must be handled

WHDC PowerPoint Template Notes & Handouts Storport Queues WHDC PowerPoint Template Notes & Handouts Monday, April 24, 2017 No per-adapter queue limitations Logical unit may have up to 255 requests outstanding Logical unit Pause and Resume routines Logical unit Busy and Ready routines Adapter Pause and Resume routines Adapter Busy and Ready routines

Direct Access To Scatter-gather List WHDC PowerPoint Template Notes & Handouts Monday, April 24, 2017 Direct access to Storport Scatter-Gather lists improves code and prevents traditional get-physical-address loop in port driver while (BytesLeft) { PhysicalAddress = ScsiPortGetPhysicalAddress ( Extension, Srb, DataBuffer, &Length ); BytesLeft -= Length; DataBuffer += Length; } SCSIport Scatter-Gather Loop ScatterGatherList = StorPortGetScatterGatherList ( Extension, Srb ); Storport Scatter-Gather Call

Dispatch Level Processing WHDC PowerPoint Template Notes & Handouts Monday, April 24, 2017 In some early Storport drivers, performance analysis showed large amounts of time spent in ISR. Further analysis showed that some drivers need to perform significant post-processing of requests. It is very expensive to perform this processing at ISR time. Instead we added a new Storport mechanism to allow miniports to create and issue deferred procedure calls (DPCs).

Windows Driver Model IRQLs WHDC PowerPoint Template Notes & Handouts Monday, April 24, 2017 Interrupt Request Levels (IRQLs) represent different levels at which code runs in kernel-mode Interrupts for lower IRQLs are masked when running at a given IRQL Following IRQLs are important to device driver writers: PASSIVE_LEVEL – May be interrupted by any interrupt, may synchronously wait for completion APC_LEVEL – May be interrupted by DISPATCH_LEVEL or higher level routines DISPATCH_LEVEL – May be interrupted by any interrupts at device interrupt level (DIRQL); in full duplex mode, the following are all called at DISPATCH level: Start I/O routine, Build I/O routine, DPC routine, timer routine and reset bus routine DIRQL – Level at which a device interrupts, may be interrupted by a higher level DIRQL

Summary Of Storport Dispatch Level Processing WHDC PowerPoint Template Notes & Handouts Monday, April 24, 2017 To support Dispatch Level Processing, Storport adds the following five new routines: StorPortEnablePassiveInitialization – Invoke a callback routine at PASSIVE level where StorPortInitializeDpc may be called StorPortInitializeDpc – Initialize a miniport DPC routine StorPortIssueDpc – Issue a DPC StorPortAcquireSpinLock – Acquire a Storport spinlock StorPortReleaseSpinLock – Release a spinlock previously acquired via StorPortAcquireSpinLock Additional miniport callbacks HwStorPassiveInitialization HwStorDpcRoutine

Passive Initialization WHDC PowerPoint Template Notes & Handouts Monday, April 24, 2017 StorPortEnablePassiveInitialization routine Requests that the port driver call back the miniport at PASSIVE_LEVEL to initialize DPCs Passive initialization may be used for other things besides initializing DPCs (e.g., passive level enumeration) May only be called from within Miniport’s HwInitialize routine Routine will fail under some circumstances Crashdump and hibernation Down-level operating systems (prior to SP1) Driver will still load on older operating systems

Initialization Of DPCs WHDC PowerPoint Template Notes & Handouts Monday, April 24, 2017 StorPortInitializeDpc initialized a DPC routine Must be called at PASSIVE_LEVEL, i.e., from HwPassiveInitialize routine May initialized any number of DPCs necessary Simple wrapper over KeInitializeDpc

WHDC PowerPoint Template Notes & Handouts Issuing DPCs WHDC PowerPoint Template Notes & Handouts Monday, April 24, 2017 StorPortIssueDpc routine Invoke the DPC routine DPCs are not queued. This is very important – if you call StorPortIssueDpc multiple times before the DPC executes, the DPC will only be invoked a single time. Miniports who use DPCs must handle this scenario, for example, by queuing pending requests. Wrapper around KeInsertQueueDpc

WHDC PowerPoint Template Notes & Handouts Miniport DPC Routine WHDC PowerPoint Template Notes & Handouts Monday, April 24, 2017 Miniport HwDpcRoutine Invoked as a result of calling StorPortIssueDpc No synchronization of DPCs with other portions of the driver is done by Storport, including synchronization between different instances of this DPC routine Miniports must use StorPortAcquireSpinLock and StorPortReleaseSpinLock to synchronize between DPC routine and other portions of miniport Use DPC routine to perform operations that would take too long to perform at interrupt level

WHDC PowerPoint Template Notes & Handouts StorPort Locks WHDC PowerPoint Template Notes & Handouts Monday, April 24, 2017 No locks are held when executing the following miniport callbacks: Find Adapter Routine Build I/O Routine Adapter Control Routine Passive Initialization Routine DPC Routine Interrupt spinlock is held during the execution of the miniport callbacks: Hardware Initialize Routine Hardware Interrupt Routine Start I/O lock is held during the execution of the following miniport callbacks: Hardware Start I/O routine Hardware Timer routine Hardware Reset Bus routine

WHDC PowerPoint Template Notes & Handouts DPC Routine Lock WHDC PowerPoint Template Notes & Handouts Monday, April 24, 2017 No locking is automatically done when accessing the DPC routine, including locking the DPC against simultaneously running on multiple processors of the machine at the same time Miniport must protect itself using StorPortAcquire/ReleaseSpinLock routines Protecting Lock Same DPC routine on a separate processor DpcLock StartIo routine StartIoLock Interrupt routine InterruptLock Multiple different DPCs StartIo lock

Message Signaled Interrupts In Storport WHDC PowerPoint Template Notes & Handouts Monday, April 24, 2017 Message Signaled Interrupts provide an advanced interrupting mechanism that solves several problems with line based interrupts No need for interrupt sharing Multiple interrupt messages for same device Increased bus throughput by eliminating some reads from devices

Storport InterruptSynchronizationMode WHDC PowerPoint Template Notes & Handouts Monday, April 24, 2017 InterruptSynchronizeAll synchronizes all MSI interrupts using the same spinlock In this mode no two MSI interrupts will occur at the same time Much simpler for driver writers InterruptSynzhronizePerMessage only acquires the per-message lock when calling an MSI interrupt Multiple MSI interrupts for different messages may occur simultaneously More complicated for miniport Cannot synchronize with a single interrupt, so StorPortSynchronizeAccess and StorPortAcquireSpinLock with a parameter of InterruptLock are not allowed A common MSI case will be with a single MSI interrupt

Message Signaled Interrupt Routine WHDC PowerPoint Template Notes & Handouts Monday, April 24, 2017 Miniports MessageSignaledInterrupt routine is called back with device extension and MessageId parameters MessageId may be used in call to StorPortGetMessageInterruptInformation to obtain the data structure describing the MSI interrupt MessageId MessageData MessageAddress InterruptVector InterruptLevel InterruptMode

WHDC PowerPoint Template Notes & Handouts References WHDC PowerPoint Template Notes & Handouts Monday, April 24, 2017 Further information on Storport is available in the Microsoft Windows Server 2003 DDK. Links http://www.microsoft.com/windowsserversystem/storage/storport.mspx http://www.microsoft.com/whdc/hwdev/tech/storage/entstor/default.mspx

WHDC PowerPoint Template Notes & Handouts Monday, April 24, 2017 © 2003 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.