WHQL Code Coverage Prototype Program Andy Wen

2 Agenda What is Code Coverage Prototype Program? What is Code Coverage Prototype Program? A prototype program to address hardware independent driver signing A prototype program to address hardware independent driver signing How does Code Coverage Work? How does Code Coverage Work? Hardware Dependent Drivers versus Hardware Independent Drivers Hardware Dependent Drivers versus Hardware Independent Drivers Code Coverage Notes Code Coverage Notes Code Coverage Testing Code Coverage Testing Submission Process Submission Process Code Coverage Tools Code Coverage Tools Best Practices Best Practices Call To Action Call To Action Questions and Answers Questions and Answers

3 What Is Code Coverage Prototype Program A new innovative solution to test & sign drivers A new innovative solution to test & sign drivers Hardware independent drivers Hardware independent drivers Display Mirror Driver Display Mirror Driver Virtual CD Driver Virtual CD Driver Sound enhancement driver Sound enhancement driver Hardware dependent drivers Hardware dependent drivers Device hardware does not fall under any existing WHQL programs Device hardware does not fall under any existing WHQL programs Standalone Windows 2000 driver for hardware Standalone Windows 2000 driver for hardware Current Status of Code Coverage Prototype Program Current Status of Code Coverage Prototype Program Target goal is 70% of code covered during testing Target goal is 70% of code covered during testing 35 companies participated, 58 drivers (including updates) 35 companies participated, 58 drivers (including updates) Average request for test result parsing per driver submission to achieve 70% is 4 times Average request for test result parsing per driver submission to achieve 70% is 4 times

4 Code Coverage Notes No Source code is exposed to Microsoft No Source code is exposed to Microsoft Because we “stamp” a compiled driver, we do not see source code Because we “stamp” a compiled driver, we do not see source code Legal agreement is required for Code Coverage participation Legal agreement is required for Code Coverage participation Uses technology under development, not ready for wide distribution Uses technology under development, not ready for wide distribution Not covered under existing agreements Not covered under existing agreements Code coverage is not the ultimate solution for ensuring driver quality Code coverage is not the ultimate solution for ensuring driver quality It is a testing metric that can be applied to all drivers It is a testing metric that can be applied to all drivers Not all code is executed Not all code is executed Conditional branch error paths Conditional branch error paths

5 What Is Code Coverage Prototype Program Benefits Benefits Measure code testing effectiveness of existing test tools Measure code testing effectiveness of existing test tools Use your own test tools Use your own test tools You control your own testing schedule You control your own testing schedule Get driver digitally signed by WHQL Get driver digitally signed by WHQL Clear measurement of what is being tested in the driver Clear measurement of what is being tested in the driver Easier to test the appropriate sections of updated driver Easier to test the appropriate sections of updated driver Enable you to easily select test cases for the future submissions Enable you to easily select test cases for the future submissions Help developer to identify redundant/unused functions in the driver Help developer to identify redundant/unused functions in the driver

6 How Does Code Coverage Work? The driver binary must be recompiled The driver binary must be recompiled WHQL provides information (instructions and additional compile and link flags) WHQL provides information (instructions and additional compile and link flags) Kernel-mode driver binary only Kernel-mode driver binary only No Source code is exposed to Microsoft No Source code is exposed to Microsoft Developer follows instructions and builds Developer follows instructions and builds An instrumented kernel-mode driver binary used in testing An instrumented kernel-mode driver binary used in testing A “retail” driver that can be signed and distributed A “retail” driver that can be signed and distributed Developer sends instrumented kernel-mode driver binary to WHQL Developer sends instrumented kernel-mode driver binary to WHQL WHQL “stamps” the instrumented kernel-mode driver binary WHQL “stamps” the instrumented kernel-mode driver binary WHQL returns “stamped” driver binary and provides monitoring tool WHQL returns “stamped” driver binary and provides monitoring tool Developer tests the “stamped” driver binary and sends encrypted log to WHQL Developer tests the “stamped” driver binary and sends encrypted log to WHQL WHQL parses log and sends coverage report to Developer WHQL parses log and sends coverage report to Developer If target coverage 70% is met or exceeded, Developer makes a WHQL submission with “retail” version of driver for signing If target coverage 70% is met or exceeded, Developer makes a WHQL submission with “retail” version of driver for signing

7 Code Coverage Testing Preliminary requirements Preliminary requirements Contact Code Coverage Prototype Program Contact Code Coverage Prototype Program The Code Coverage Prototype Program is offered through both the WHQL U.S.A. and the WHQL Japan labs and is available only if the following criteria are met The Code Coverage Prototype Program is offered through both the WHQL U.S.A. and the WHQL Japan labs and is available only if the following criteria are met The driver is NOT one of the following types The driver is NOT one of the following types Device driver stack (bus driver or bus filter-driver) Device driver stack (bus driver or bus filter-driver) File system filter driver File system filter driver Transport Driver Interface (TDI) driver Transport Driver Interface (TDI) driver 64-bit driver 64-bit driver The driver is installed with a device setup Information file (INF) The driver is installed with a device setup Information file (INF) The driver runs in kernel mode The driver runs in kernel mode Accept Code Coverage Agreement Accept Code Coverage Agreement

8 Code Coverage Testing Submission Process Submission Process 1. Contact WHQL Code Coverage Prototype Program 2. Accept Code Coverage Agreement 3. Establish a Winqual Account on Click "How to Establish an Account" and follow the steps; Your company must obtain a VeriSign Code-3 Signing ID, which can take weeks While waiting, proceed to step Download HCT 11.2 at You must use a logo'd system running Windows XP Professional to test devices; For a USB device, you need a USB 2.0 host controller and an external USB 2.0 self-powered hub between the test system and the test device; HCT documentation describes the test environment 5. Instrument driver 6. Run Tests 7. Return coverage result to WHQL Code Coverage Prototype Program 8. After the test result meets all the requirements, you can make a submission by using the account that you establish from #3

9 Hardware Dependent Drivers Example Example Personal Digital Assistants (PDAs) Personal Digital Assistants (PDAs) Cell-phone connectivity adapters Cell-phone connectivity adapters Biometric security devices Biometric security devices Hardware-based software protection devices Hardware-based software protection devices Device follows the Windows Logo Program requirements Device follows the Windows Logo Program requirements Driver follows the Windows DDK for the target OS Driver follows the Windows DDK for the target OS Pass HCT 11.2 or later version of Unclassified\Universal kit Pass HCT 11.2 or later version of Unclassified\Universal kit Use your own test cases and test tools Use your own test cases and test tools Meets or exceeds 70% of code coverage Meets or exceeds 70% of code coverage 2 hardware samples are required for submission 2 hardware samples are required for submission Up to 30 days of submission turnaround time (Goal of 2 weeks) Up to 30 days of submission turnaround time (Goal of 2 weeks) Driver is digitally signed Driver is digitally signed “Designed for Windows” logo for hardware is issued “Designed for Windows” logo for hardware is issued Windows Catalog listing Windows Catalog listing

10 Hardware Independent Drivers Example Example Network VPN drivers Network VPN drivers Virtual CD drivers Virtual CD drivers Sound enhancement drivers Sound enhancement drivers Driver follows the Windows DDK for the target OS Driver follows the Windows DDK for the target OS Pass all the applicable test tools from HCT 11.2 or later version Pass all the applicable test tools from HCT 11.2 or later version Use your own test cases and test tools Use your own test cases and test tools Meets or exceeds 70% of code coverage Meets or exceeds 70% of code coverage Provide WHQL a copy of the software uses the driver Provide WHQL a copy of the software uses the driver Up to 48 hours of submission turnaround time Up to 48 hours of submission turnaround time Driver is digitally signed Driver is digitally signed No “Designed for Windows” hardware logo will be issued No “Designed for Windows” hardware logo will be issued May be eligible for “Designed for Windows” software logo process May be eligible for “Designed for Windows” software logo process

11 Code Coverage Testing Your own test cases and test tools Your own test cases and test tools Core Driver Quality HCT Tests Core Driver Quality HCT Tests Public Import Public Import Signability Signability ChkINF ChkINF Driver Verifier Driver Verifier Device Path Exerciser Device Path Exerciser Core Functionality HCT Tests Core Functionality HCT Tests ACPI Stress ACPI Stress Enable/Disable Device I/O Enable/Disable Device I/O Core Bus HCT Tests Core Bus HCT Tests PCI Specific test tools PCI Specific test tools USB Specific test tools USB Specific test tools 1394 Specific test tools 1394 Specific test tools

12 Code Coverage Testing Core Device Specific HCT Tests Core Device Specific HCT Tests DVD Annex – Decode DVD Annex – Decode NDIS NDIS DirectMusic DirectMusic Best Practices Best Practices

13 Best Practices Multiple Processor environment makes it easier to find weak code Multiple Processor environment makes it easier to find weak code Use a Multiple Processor (or Hyper-Threading) system for testing Use a Multiple Processor (or Hyper-Threading) system for testing Use two or more test devices simultaneously (in the same system) Use two or more test devices simultaneously (in the same system) Test on fastest processor Test on fastest processor Enable Driver Verifier while running all test tools Enable Driver Verifier while running all test tools Enable Driver Verifier prior to running any HCT Tools and your own test tools Enable Driver Verifier prior to running any HCT Tools and your own test tools Don't leave "KdPrint()" in your retail driver binaries Don't leave "KdPrint()" in your retail driver binaries Remove this, and any other debug code from your retail driver binaries before you begin testing Remove this, and any other debug code from your retail driver binaries before you begin testing Device Specific Applications Device Specific Applications IMPORTANT: If a device requires an application, ensure that the app IMPORTANT: If a device requires an application, ensure that the app Calls RegisterDeviceNotification on the device handle Calls RegisterDeviceNotification on the device handle Listens for WM_DEVICECHANGE in its message loop Listens for WM_DEVICECHANGE in its message loop On receiving this message with type DBT_DEVICEQUERYREMOVE, it closes the device handle so the device can be properly removed On receiving this message with type DBT_DEVICEQUERYREMOVE, it closes the device handle so the device can be properly removed

14 Call To Action Contact to participate the program Contact to participate the Refer to Windows DDK and Windows Logo Program requirements when developing new devices and drivers Refer to Windows DDK and Windows Logo Program requirements when developing new devices and drivers Subscribe WHQL Weekly Newsletter for the program updates Subscribe WHQL Weekly Newsletter for the program updates Ask The Expert – Thursday 05/06/2004 from 12:00 P.M. to 1:30 P.M. Ask The Expert – Thursday 05/06/2004 from 12:00 P.M. to 1:30 P.M.

15 Community Resources WHQL Website WHQL Website Device and System testing Device and System testing Winqual Online Submission Winqual Online Submission Attend a free chat or Webcast Attend a free chat or Webcast Locate a local user group(s) Locate a local user group(s) Non-Microsoft Community Sites Non-Microsoft Community Sites