2. Improving Driver Concurrency Testing James Moe and Dieter Achtelstetter Windows Devices & Storage Technologies drvtest@microsoft.com

3. Agenda Concurrency and driver bugs Observations about our existing tests A tool for looking at IRP concurrency How we used that data to improve our tests The new Windows 7 tests in depth How you can use these tools to improve your in-house tests in respect to concurrency

4. Concurrency and Drivers Driver are highly concurrent Much of a driver’s complexity derives from concurrency Many late or post-ship bugs are related to concurrency

5. Observations about Our Existing Tests Tests tend to be very specific PnP tests do PnP I/O tests do I/O Existing tests tend not to work well when run side by side Unclear what concurrency scenarios they cover

6. Introducing IRP Concurrency Coverage Provides an additional coverage dimension to code coverage Can identify test coverage holes Complements other tools like code coverage, static analysis, etc.

7. IRP Pair Coverage A form of concurrency coverage A pair is covered if: The 2 IRPs making up the pair are active within a single driver stack 1326 pairs From 52 IRPS Kernel rules removed 227 pairs; for example: Only one state-changing PnP IRP active within a stack Leaves 1099 pairs

8. What Is an active IRP?

9. Why Pairs? Too many combinations of all possible IRPs Equal distribution through the problem space Improvements visible as percentage of coverage IRPSPossible combinations 12 MJ IRPS4096 27 MN & MH IRPS134,217,728

10. Data Perspectives 2 identical pair hits are not identical A pair hit doesn’t necessarily indicate a valuable test case Repetitive pair hits increase the odds of valuable test cases Low number of hits per pair indicates missing test coverage

11. Example: Full Test Pass Metrics MetricCoverage percentage MJ-only IRP coverage 84.62 IRP coverage 55.77 IRP pair coverage 2.91

12. Example of Pair Coverage

13. Tests Comprising the Example Test Pass Tests Create all different variations of volumes/partition on both GPT and MBR disks and run I/O to each drive created. Bigfile (I/O test) DC2 against the miniport Disable/enable test Automated test to open and populate the “Volumes” tab in the properties dialog for the disk in devman Disable_Enable_With_IO Sleep_Stress_With_IO Mapfile (I/O test) PnPDTest Query/Remove SCSI break reservation SCSI compliance test SPTI (SCSI passthrough test) Storioctls

14. Test Improvement Focus Test List Plug and Play Driver Test (PnPDTest) Disable/Enable with I/O (Disable_Enable_With_IO) Sleep stress with I/O (Sleep_Stress_With_IO) DC2 Static tools

15. What’s New in PnPDTest – More Than Just PnP Coupling I/O with PnP IRPs Configurable—old test functionality is still there IOCTLs Read/write Power

16. PnPDTest - I/O and PnP IOCTL Generic buffered IO Planning to integrate more DC2 functionality Read/Write Uses Windows Device Testing Framework (WDTF) SimpleIO Existing WDTF SimpleIO interfaces for net, audio, video, and volume Extensible for other device types

17. PnPDTest – Power + I/O, Power +PnP Uses WDTF interfaces to set system power, set timers for wake Combine S-IRPs and resulting D-IRPs with IOCTL and Read/Write Combine power IRPs with PnP IRPs All three together!

18. Device Path Exerciser (DC2), Static Tools, WLK Scripts Work is underway to modify DC2 to play well with other tools Goal is to get features like I/O fuzzing to run concurrently with tools like PnPDTest, Sleep Stress with I/O, etc. Some functionality is being integrated into other tools Static Tools (Static Analysis, PREfast for Drivers) Look at IRP pair coverage data to find holes Use that data to identify areas where static analysis could find bugs in untested areas Sleep Stress with I/O, Disable_Enable_With_IO Work on improving concurrency in these as well

19. Old PnP Test coverage

20. New PnP Test coverage

21. Concurrency Coverage Tools Enable you to do your own analysis on in-house tests Tools will be available in the Windows 7 Beta

22. How to Improve Concurrency Coverage Today Combine existing tests Simple and low tech Look for crashes and hangs Write your own device-specific tests You know the behavior of your device and driver best

23. Create Your Own Device-specific Test Cases Combine your device-specific operations with: PnP Disable / Enable device Surprise remove device Power Close laptop lid Put system to sleep Examples I/O IRPS Query and change device data WMI and IOCTL IRPs

24. Automate These Test Cases Run our tests and your device-specific tests side-by-side WDTF test scripts we supply in the WLK PNPDTEST DC2 Write a WDTF Simple I/O plug-in for your device class Existing WDTF scripts and PNPDTEST will use this plug-in Enables you to easily write your own combinatorial WDTF test scripts using PnP and power

25. Call to Action Use the new tests in your regular day-to-day testing Use the Concurrency Coverage tools to identify areas for additional concurrency testing Improve your in-house tests to get better concurrency coverage

26. Resources WLK and WDK on the WHDC Web site http://www.microsoft.com/whdc/DevTools/WDK/WDKpkg.mspx http://www.microsoft.com/whdc/DevTools/WDK/WDKpkg.mspx WDTF documentation on MSDN http://msdn.microsoft.com/en-us/library/aa972915.aspx http://msdn.microsoft.com/en-us/library/aa972915.aspx Send feedback to drvtest@microsoft.comdrvtest@microsoft.com