Aditya V. Nori, Sriram K. Rajamani Microsoft Research India

 An industrial strength program verifier  Idea: Synergize verification and testing  Synergy [ FSE ’06 ], Dash [ ISSTA ‘08 ], SMASH [ POPL ‘10 ] algorithms to perform scalable analysis  Engineered a number of optimizations for scalability  Integrated with Microsoft’s Static Driver Verifier ( SDV ) toolkit and used internally

 Overview of Yogi  Overview of optimizations  Evaluation setup  Empirical Results  Summary

void foo() { *p = 4; *q = 5; if (condition) error(); } Question Is error() unreachable for all possible inputs? Verification: can prove the absence of bugs, but can result in false errors Testing: finds bugs, but can’t prove their absence

no Can extend test beyond frontier? Refine abstraction Construct initial abstraction Construct random tests Test succeeded? Bug! Abstraction succeeded? τ = error path in abstraction f = frontier of error path yes no yes Proof! yes Input: Program P Property ψ

no Can extend test beyond frontier? Refine abstraction Construct initial abstraction Construct random tests Test succeeded? Bug! Abstraction succeeded? τ = error path in abstraction f = frontier of error path yes no yes Proof! yes Input: Program P Property ψ void foo(int y) { 0: int x, lock = 0; 1: do { 2: lock = 1; 3: x = y; 4: if (*) { 5: lock = 0; 6: y = y+1; } 7: } while (x != y); 8: if (lock != 1) 9: error(); 10: } y = × × × × × × × × × × × × × × 10 × Symbolic execution + Theorem proving

no Can extend test beyond frontier? Refine abstraction Construct initial abstraction Construct random tests Test succeeded? Bug! Abstraction succeeded? τ = error path in abstraction f = frontier of error path yes no yes Proof! yes Input: Program P Property ψ void foo(int y) { 0: int x, lock = 0; 1: do { 2: lock = 1; 3: x = y; 4: if (*) { 5: lock = 0; 6: y = y+1; } 7: } while (x != y); 8: if (lock != 1) 9: error(); 10: } :ρ 9 × × × × × × × × × × × × × × 10 × 8:¬ρ ×

no Can extend test beyond frontier? Refine abstraction Construct initial abstraction Construct random tests Test succeeded? Bug! Abstraction succeeded? τ = error path in abstraction f = frontier of error path yes no yes Proof! yes Input: Program P Property ψ void foo(int y) { 0: int x, lock = 0; 1: do { 2: lock = 1; 3: x = y; 4: if (*) { 5: lock = 0; 6: y = y+1; } 7: } while (x != y); 8: if (lock != 1) 9: error(); 10: } :¬s 5:¬s 6:¬r 9 × × × × × × × ×× × × 7:¬q × 8:¬p × 4:s 5:s 6:r 7:q 8:p × 10

 Initial abstraction from property predicates  Relevance heuristics for predicate abstraction  Suitable predicates ( SP )  Control dependence predicates ( CD )  Interprocedural analysis  Global modification analysis  Summaries for procedures  Thresholds for tests  Fine tuning environment models

 Benchmarks:  30 WDM drivers and 83 properties ( 2490 runs)  Anecdotal belief: most bugs in the tools are usually caught with this test suite  Presentation methodology:  Group optimizations logically such that related optimizations are in the same group  Total time taken, total number of defects found for every possible choice of enabling/disabling each optimization in the group

state { enum {Locked = 0, Unlocked = 1} state = Unlocked; } KeAcquireCancelSpinlock.Entry { if (state != Locked) { state = Locked; } else abort; } KeReleaseCancelSpinlock.Entry { if (state == Locked) { state = Unlocked; } else abort; }

Abstraction using SLIC predicates Total time (minutes) #defects#timeouts yes no %

A C B D A C B D C Irrelevant?

 Abstract assume statements that are not potentially relevant by skip statements  If Yogi proves that the program satisfies property, we are done.  Otherwise, validate the error trace and refine the abstraction by putting back assume statements, if the error trace is spurious

int x; void foo() { bool protect = true; … if (x > 0) protect = false; … if (protect) KeAcquireCancelSpinLock(); for (i = 0; i < 1000; i++) { a[i] = readByte(i); } if (protect) KeReleaseCancelSpinLock(); } A C B D A C B D C

int x; void foo() { bool protect = true; … if (x > 0) protect = false; … if (protect) KeAcquireCancelSpinLock(); for (i = 0; i < 1000; i++) { a[i] = readByte(i); } if (protect) KeReleaseCancelSpinLock(); } A C B D A C B D C

int x; void foo() { bool protect = true; … if (x > 0) protect = false; … if (protect) KeAcquireCancelSpinLock(); for (i = 0; i < 1000; i++) { a[i] = readByte(i); } if (protect) KeReleaseCancelSpinLock(); }

SP heuristic CD heuristic Total time (minutes) #defects#timeouts yes yes no no yes no %

SP heuristic CD heuristic Total time (minutes) #defects#timeouts yes yes no no yes no %

SP heuristic CD heuristic Total time (minutes) #defects#timeouts yes yes no no yes no %

A C B D A C B D C foo(…)

Modification analysis SummariesTotal time (minutes) #defects#timeouts yes yes no no yes no %

Modification analysis SummariesTotal time (minutes) #defects#timeouts yes yes no no yes no %

Modification analysis SummariesTotal time (minutes) #defects#timeouts yes yes no no yes no %

Test threshold Total time (minutes) #defects#timeouts

if (DestinationString) { DestinationString->Buffer = SourceString; // DestinationString->Length should be set to the // length of SourceString. The line below is missing // from the original stub SDV function DestinationString->Length = strlen(SourceString); } if (SourceString == NULL) { DestinationString->Length = 0; DestinationString->MaximumLength = 0; } Issue type#issues Integers used as pointers 8 Uninitialized variables 15 Type inconsistencies 9

 Described optimizations implemented in Yogi  Evaluated optimizations on the WDM test suite  Empirical data used to decide which optimizations to include in Yogi  We believe that this detailed empirical study of optimizations will enable tool builders to decide which optimizations to include and how to engineer their tools 