How to Manage a Quality Device Driver Donald D. Burn Chief Software Architect acm.org Reliable Technologies, Inc

Session Outline Fundamentals Documentation Development Maintenance

Session Goals Attendees should leave this session with the following: A better understanding of the driver development process Approaches to driver development that can Improve the quality of device drivers Speed time to market Reduce long-term costs

Fundamentals Driver development is different than application work Recognizing the differences is the best way to improve driver efforts

Fundamentals – Drivers Are a Specialty Driver writers need training Driver development continually changes Continue to send your developers to the Microsoft Windows Driver Developers Conference Watch the Microsoft Windows Hardware and Driver Central website for new information Experts can be cheaper There are many types of drivers and no developer is a specialist in all of them

Fundamentals – Drivers & Product Creation Engage driver development at the product design phase Involve driver development with hardware design Start driver planning early Schedule lots of time for testing

Fundamentals – No More Win9x You cannot design for just a single processor x86 with < 4GB memory We now have: Multiprocessors everywhere Low-cost memory for more than 4 GB 64-bit processors Headless and/or multi-user systems Hot-plug devices widespread

Fundamentals – Drivers Are Part of Windows Build the driver using the DDK’s tools Install the driver following Windows best practices Related Session – “DMI: Driver Install Frameworks Tools and Strategies” Test the driver to WHQL standards and beyond Validate the driver on a variety of platforms Submit the driver to WHQL

Fundamentals – Windows Drivers Drivers are derived from existing drivers Bugs can be propagated to other drivers Windows Driver Foundation addresses this reuse Drivers typically need small teams Requires good teamwork Adding manpower to a late software project makes it later (The Mythical Man-Month, Fred Brooks)

Fundamentals – Driver Cost Rule of thumb: System software costs twenty times the salaries of the developers for the life of the product Typically two thirds of the cost of software is in maintenance and support Poor quality can push support costs to a much higher percentage Rule of thumb: If you are lucky 10% of your dissatisfied customers will let you know they are unhappy!

Documentation Critical for tomorrow Gives you a baseline Allows for fix priced quotes Can be an incremental effort Can be contracted

Documentation – Hardware Documents Hardware Guidelines Specify common criteria for good hardware Create your guidelines from: Windows Logo Program Requirements Industry standards such as: PCI, USB, etc Your company’s past experience Driver developers need to give input Hardware Specification Document programming the device Joint effort of hardware and software

Documentation – Coding Guidelines Styles for interfaces and security Rules on undocumented and deprecated functions Diagnostic model Standard tool set for building drivers Build options for reliability and testing

Documentation – Driver Specification Define all interfaces and their security Note the Installation Model Specify projected performance goals Document deviations from the guidelines and standards Identify the ancestry of the driver source code Explain the design for unique features May be part of a system architecture / larger product

Documentation – Test Documents Test Guidelines Engineering Verification Test Plan QA and Certification Plan Beta Test Plan

Development Rule of thumb: A defect that costs $1 to fix on the programmer's desktop costs $100 to fix once it is incorporated into a complete program and many thousands of dollars if it is identified only after the software has been deployed in the field. (Building a Better Bug Trap - The Economist June 2003) Microsoft provides tools for finding many errors when the cost is low! Perform code reviews to improve quality

Development – Driver Development Kits Use the latest Device Driver Kit (DDK) Works for all systems back to Windows 2000 Most up to date APIs, tools, samples and documentation It is FREE!!! Use the compiler and tools from the DDK Windows itself is built with these tools

Development – Source Control Source control is a debugging tool Encourage checking in source often – as much as multiple times a day Track individual changes to isolate errors Provides an undo capability for development Use the source control data to feed the code review Nothing is done until it is under source control Set your milestones based on when a feature is tested and checked in Use the above as the basis for determining if the project is on schedule

Development – Debugging Environment A test machine separate from the development system The test machine should be multi-processor or at least hyper-threaded Higher end systems find more bugs than lower end systems Consider hot-plug drives for easy OS changes Additional test machines: Other CPU architectures and other HAL’s Large multiprocessor (8 or more if possible) Large memory > 4GB Hot-plug support

Development – Compile for Reliability Compile with /WALL /WX Build with PREfast Enable Deprecated Function checking Use C_ASSERT – compile time assertions Consider PC-Lint and custom checking Validate INF files with ChkINF More tools are coming Stay up to date with the latest tools Provide time for your developers to learn the new tools

Development – Execute for Quality Debug drivers under the Checked Build of Windows Run drivers under Driver Verifier Enable the Call Usage Verifier Consider compiler runtime checks Use ASSERT – runtime assertions Conditionally instrument the driver Allocate and free memory with tags

Development – Test for Validity Engineering Verification Test Basic tests for most functionality Quality Assurance and Certification Tests Test drivers with HCT Custom tests Integration tests Stress tests Regression tests

Development – Profile Code coverage Identifies code with little or no utilization Shows where tests can use improvements Performance Profiling Identify performance hotspots to improve Can indicate errors in a driver Use the above data to perform a code review of the driver

Development – Code Review Use walkthroughs during development Have a formal inspection before the driver leaves development Consider a review by an outside firm Code reviews find twice the number of bugs as testing!

Development – Threat Analysis Perform a threat analysis on your drivers Identify all possible avenues of a attack on your driver Writing Secure Code by Howard and LeBlanc is an excellent reference Drivers need secure interfaces Secure device objects Check all data accessible from user space Be careful with IOCTL’s Add additional security checks for dangerous operations Do not develop “all-in-one” interfaces that can be abused

Maintenance Provide reporting and diagnostics in your driver Take advantage of Online Crash Analysis Create a feedback model to stop bugs from recurring

Maintenance – Customer Reports Event Logging Don’t log the ordinary! Use unique error log entries WMI performance data

Maintenance – Diagnostics WPP tracing Extremely low performance impact Not decodable by users WinDBG extensions Custom dump data

Maintenance – OCA Data Windows Error Reporting – provides direct customer data about Windows crashes Microsoft’s Online Crash Analysis database – prioritizes problems with your driver

Maintenance – Defect Feedback Let the developer know Fix all the drivers that use the code Create a regression test so the bug does not reappear

Call To Action Update your practices to improve driver quality Involve your driver developers early in the product cycle Create a culture of quality for your drivers

Community Resources Windows Hardware & Driver Central (WHDC) Technical Communities Non-Microsoft Community Sites Microsoft Public Newsgroups Technical Chats and Webcasts Microsoft Blogs

Additional Resources Web Resources: Windows Hardware and Driver Central Related Sessions The WDK for Engineering Managers and Product Planners: An Introduction Windows Driver Foundation: An Introduction DMI: Driver Install Frameworks Tools and Strategies