Using Model-Checking to Debug Device Firmware Sanjeev Kumar Microprocessor Research Labs, Intel Kai Li Princeton University

Using Model Checking to Debug Device Firmware2 Programmable Devices Network Card Bus CPU D Mem Disk Network CPU D D D Mem Main CPU Main Memory Move functionality from main CPUs to devices Device firmware is more complex

Using Model Checking to Debug Device Firmware3 Firmware for Programmable Devices Difficult to write and debug –Use concurrency Inherently difficult to program correctly –Optimized for high performance Tradeoff program simplicity for performance –Limited debugging support available Firmware reliability is important –Firmware is trusted by the OS –Bugs in firmware can crash the entire machine Model checking is a promising approach

Using Model Checking to Debug Device Firmware4 Model Checking Systematically verify properties of concurrent systems Using State-space exploration –Try all possible scheduling options Advantages –Automatic –Produces counter example Disadvantages –Computationally expensive Exponential search Models cannot be too big ModelTest CodeProgram

Using Model Checking to Debug Device Firmware5 Using Model Checking to Debug Firmware Extracting models from programs –Manually (by the programmer) –Automatic (using the compiler) Reduces programmer effort Reduces mismatch between program and model Extracting smaller models models –Support for abstraction Discard irrelevant details in the program

Using Model Checking to Debug Device Firmware6 Our Work Extract abstract models using a compiler –General compiler techniques Programmer controls the abstraction process –Specifies what needs to be abstracted Compiler performs the abstractions conservatively –Extract models for Spin model checker from programs written in the ESP language –Make practical choices Goal: Debugging and not Verification Used to debug VMMC firmware for a network card –Found 7 bugs that can cause the firmware to deadlock Could not find these bugs without support for abstraction

Using Model Checking to Debug Device Firmware7 Related Work Manual Model Extraction –Harmony, RUBIS, Plan 9, Fluke OS Used model checking to debug a subsystem Automatic Model Extraction –Teapot, Promela++, Esterel, Java Pathfinder Domain-specific and general-purpose languages Automatic Extraction + Support for abstraction –Feaver, Lie et. al., Bandera

Using Model Checking to Debug Device Firmware8 Outline Background Extracting Abstract Models using a Compiler –From ESP Language –For Spin Model Checker Evaluation: Debugging VMMC Firmware Conclusion and Future Work

Using Model Checking to Debug Device Firmware9 ESP: A Language for Programmable Devices pgm1.spin pgmN.spin pgm.C ESP Compiler pgm.ESP help.C Generate Firmware Develop and Test using Model Checker test1.spin testN.spin Goals 1)Easy to program 2)Allow extensive testing 3)Performance

Using Model Checking to Debug Device Firmware10 The ESP Language Concurrent language: Processes & Channels –Pure message-passing communication –in, out, alt operations on channels –Channels are synchronous or unbuffered –Processes and channels are static A number of interesting features –Explicit memory management scheme that uses model-checking to ensure safety –Supports dispatch on channels –Efficient and powerful interface to C

Using Model Checking to Debug Device Firmware11 Extracting Models for Spin Step 1 : Detailed models [ PLDI’01 ] –Translate each language construct into Spin –Simple translation int, bool, records, arrays, unions If-then-else, while-loops process, channel –Spin does not support Dynamic memory allocation & Pointers Additional bookkeeping necessary to support these Can be used to check local properties –Debug subsystems (1-2 processes) separately –Too big to debug the entire system

Using Model Checking to Debug Device Firmware12 Extracting Abstract Models Conservatively Step 2: Abstract models –Necessary to check global properties (like deadlocks) –Drop unnecessary details Depending on the property being verified –Programmer controls the abstraction Abstraction specified by the programmer –Drop variables –Drop fields from records and unions Compiler used the abstraction specified conservatively –Could introduce fast-positive bugs –All bugs in the programs will be present in the extracted model –Involves dealing with a number of tricky cases

Using Model Checking to Debug Device Firmware13 Examples $b2: boolean = true;... $b1: boolean = b2; $b2: boolean = true;... $b1: boolean = b2; type recT = #record of { int count; } $r1: recT = {0}; if (b) { r2 = r1; }... r1.count = 5; $r1: recT = {0}; if (b) { r2 = r1; }... r1.count = 5; if :: b1 = true :: b1 = false fi if :: b1 = true :: b1 = false fi if :: r2.count = 5 :: skip fi if :: r2.count = 5 :: skip fi X X X X X Conservative: Use nondeterminism to broaden the state-space searched

Using Model Checking to Debug Device Firmware14 Outline Background Extracting Abstract Models using a Compiler Evaluation: Debugging VMMC Firmware Conclusion and Future Work

Using Model Checking to Debug Device Firmware15 VMMC High-performance communication –Bypass OS for data transfers Used Myrinet network cards –Gigabit network –33 MHz CPU, 1 MB memory Original VMMC firmware –Implemented in C Several man-years of debugging Still encounter bugs Some involve complex race conditions that are triggered only occasionally Data OS Network Card Application Network

Using Model Checking to Debug Device Firmware16 Debugging VMMC Firmware Reimplemented VMMC firmware using ESP Used model checking to debug –Global property of program (deadlocks) –Hard-to-find bugs –Found 7 bugs using abstract models 4 Bugs would cause deadlock during normal operations 3 Bugs would be triggered only by a malicious machine –Could not find these bugs without abstractions No firmware bugs encountered on device –Microbenchmarks –SPLASH2 parallel application suite On a 16-processor SMP cluster

Using Model Checking to Debug Device Firmware17 Resource used for Model Checking Spin Mode States Searched (in Millions) CPU Time (in seconds) Memory (in Mbytes) Exhaustive * Partial mode * Only partial search was possible Even partial searches were effective * Limiting Resource VMMC Firmware

Using Model Checking to Debug Device Firmware18 Model extracted from VMMC Firmware FileLines of Code ESP Program453 Abstraction Specification108 Abstract Model Extracted2202 Test Code128 Programmer only write a small amount of Spin Code Program can be rechecked with little effort

Using Model Checking to Debug Device Firmware19 Outline Background Extracting Abstract Models using a Compiler Evaluation: Debugging VMMC Firmware Conclusion and Future Work

Using Model Checking to Debug Device Firmware20 Conclusions Use compiler to extract abstract models –Evaluation: Debugged VMMC firmware Using compiler to extract models is good –Significantly reduces effort required to model check Abstraction is required –To check global properties like deadlocks Programmer can control the abstraction –Compiler is conservative –Does not require the programmer to be correct Only partial search was possible –Still effective in finding bugs

Using Model Checking to Debug Device Firmware21 Future Work Optimizations to reduce size of state space –Eliminating more redundancies Quantify the effectiveness of a partial seach –Estimate the fraction of state-space searched Use type systems to reduce the size of state space that has to be searched

Using Model Checking to Debug Device Firmware22 To find out more, Visit Questions?

Using Model Checking to Debug Device Firmware23 Debug not Verify Several sources of incompleteness and unsoundness remain –Programmer supplied Spin code –Partial model checking The goal is to isolate/reduce the unsound portions of the code

Using Model Checking to Debug Device Firmware24 ABCDEF Abcdef Ghijk