1. The Yogi Project Software property checking via static analysis and testing Aditya V. Nori, Sriram K. Rajamani, Sai Deep Tetali, Aditya V. Thakur Microsoft Research India

2. Synergizing Testing and Static Analysis Idea: Combine testing and static analysis through a “synergy” of both New algorithms ( Synergy, Dash, SMASH ) to do scalable proofs using testing Industrial strength tool ( 3 person years) Synergy : Gulavani, Henzinger, Kannan, N., Rajamani, FSE 2006 Dash : Beckman, N., Rajamani, Simmons, ISSTA 2008 SMASH : Godefroid, N., Rajamani, Tetali, MSR-TR-2009

3. The problem Question Is error() unreachable for all possible inputs? Input Testing: finds bugs, but can’t prove their absence Static analysis: can prove the absence of bugs, but can result in false errors

4. a↦true b↦false limit↦2 × Testing × a↦true b↦false limit↦2 i↦0 lock↦1 x=0 y=0

5. a↦true b↦false limit↦2 × Testing × × × × × × × × × × × × ×

6. a↦true b↦false limit↦2 × Testing × × × × × × × × × × × × × test

7. Abstraction 1 2:ρ × × s1s1 s2s2

8. Abstraction 0 2 34 5 1 9 67 8 11 10

9. Proof 2:ρ 0 2: ¬ρ 3:ρ4: ¬ρ 5: ¬ρ 1:ρ 9:ρ 6:ρ7: ¬ρ 8: ¬ρ 11 10 9: ¬ρ 1: ¬ρ 4:ρ 3: ¬ρ 5:ρ 7:ρ 6::¬ρ 8:ρ ρ = (lock != 1)

10. Key Ideas  Frontier: Boundary between tested and untested regions  WP α : New refinement operation that does not depend on whole program alias information 0 1 2 3 4 7 8 9 × × × × × × × × × 5 6 ×× × × 10 frontier

11. Key Ideas Step 1 : Try to generate a test that crosses the frontier – Perform symbolic simulation on the path until the frontier and generate a constraint  1 – Conjoin with the condition  2 needed to cross frontier – Is  1  2 satisfiable? 0 1 2 3 4 7 8 9 × × × × × × × × × 5 6 ×× × × 10 11 22 frontier

12. Key Ideas 0 1 2 3 4 7 8 9 × × × × × × × × × 5 6 ×× × × 10 frontier Step 1 : Try to generate a test that crosses the frontier – Perform symbolic simulation on the path until the frontier and generate a constraint  1 – Conjoin with the condition needed to cross frontier  2 – Is  1  2 satisfiable? [YES] Step 2 : run the test and extend the frontier × × ×

13. Key Ideas Step 1 : Try to generate a test that crosses the frontier – Perform symbolic simulation on the path until the frontier and generate a constraint  1 – Conjoin with the condition needed to cross frontier  2 – Is  1  2 satisfiable? [NO] 0 1 2 3 4 7 8 9 × × × × × × × × × 5 6 ×× × × 10 11 22 frontier

14. Key Ideas Step 1 : Try to generate a test that crosses the frontier – Perform symbolic simulation on the path until the frontier and generate a constraint  1 – Conjoin with the condition needed to cross frontier  2 – Is  1  2 satisfiable? [NO] Step 2 : use WP α to refine so that the frontier moves back! 0 1 2 3 4 :¬ρ 7 8 9 × × × × × × × × × 5 6 ×× × × 10 frontier 4:ρ4:ρ

15. DASH : Proofs from Tests – Algorithm uses only test case generation operations – Maintains two data structures: A forest of reachable concrete states (tests) – Under-approximates executions of the program A region graph (an abstraction) – Over-approximates all executions of the program – Our goal: bug finding and proving If a test reaches an error, we have found bug If we refine the abstraction so that there is *no* path from the initial region to error region, we have a proof – Handles the richness of C New operator WP α uses only aliases α that are present along concrete tests that are executed Algorithm uses recursive invocations to handle inter-procedural analysis

16. Can extend test beyond frontier? Refine abstraction Construct initial abstraction Construct random tests Test succeeded? Bug! Abstraction succeeded? τ = error path in abstraction f = frontier of error path yes no yes no Proof! yes no Input: Program P Property ψ The DASH Algorithm

17. Example Can extend test beyond frontier? Refine abstraction Construct initial abstraction Construct random tests Test succeeded? Bug! Abstraction succeeded? τ = error path in abstraction f = frontier of error path yes no yes no Proof! yes no Input: Program P Property ψ

18. τ=(0,1,2,3,4,7,8,9) Example Symbolic execution + Theorem proving frontier Can extend test beyond frontier? Refine abstraction Construct initial abstraction Construct random tests Test succeeded? Bug! Abstraction succeeded? τ = error path in abstraction f = frontier of error path yes no yes no Proof! yes no Input: Program P Property ψ 0 1 2 3 4 5 6 7 8 9 × × × × × × × × × × × × × × 10 × y = 1

19. Refinement 0 1 2 3 4 5 6 7 8 9 × × × × × × × × × × × × × × 10 × 8:¬ ρ 8:ρ 9 8 9 ρ= (lock.state != L) ×× refine

20. Refinement 8:¬ ρ 8:ρ 9 8 9 ρ= (lock.state != L) ×× 0 1 2 3 4 5 6 7 8 :¬ρ 9 × × × × × × × × × × × × × × 10 × 8:ρ refine

21. Example τ=(0,1,2,3,4,7,,9) 0 1 2 3 4 5 6 7 8 :¬ρ 9 × × × × × × × × × × × × × × 10 × 8:ρ Can extend test beyond frontier? Refine abstraction Construct initial abstraction Construct random tests Test succeeded? Bug! Abstraction succeeded? τ = error path in abstraction f = frontier of error path yes no yes no Proof! yes no Input: Program P Property ψ frontier

22. Correct, the program is 0 1 2 3 4⋀¬s 5⋀¬s 6⋀¬r 9 × × × × × × × × × × × 7⋀¬q × 8⋀¬p × 4⋀s 5⋀s 6⋀r 7⋀q 8⋀p × 10

23. Handling procedure calls Key idea Perform a recursive Dash query on the called procedure and use the result to either generate a test or compute WP α S k-1 SkSk × S k-2 T × CALL(foo(i, j)) frontier

24. Interprocedural analysis S k-1 SkSk × S k-2 T × CALL(foo(i, j)) Dash[assume(φ 1 ), foo(i, j), assert(¬φ 2 )] - pass: perform refinement - fail: generate test 11 22

25. Correctness Theorem. If Dash terminates on (P, φ), then either of the following is true: – If Dash returns (“pass”, Σ ≃ ), then Σ ≃ is a proof that P cannot reach ¬φ – If Dash returns (“fail”, t), then t certifies that P reaches ¬φ Theorem. Every Dash iteration entails exactly one theorem prover call However … Dash need not terminate! Thus, usefulness is based on empirical evidence

26. Locked do { //get the write lock KeAcquireSpinLock(&devExt->writeListLock); nPacketsOld = nPackets; request = devExt->WriteListHeadVa; if(request && request->status){ devExt->WriteListHeadVa = request->Next; KeReleaseSpinLock(&devExt->writeListLock); irp = request->irp; if(request->status > 0){ irp->IoStatus.Status = STATUS_SUCCESS; irp->IoStatus.Information = request->Status; } else{ irp->IoStatus.Status = STATUS_UNSUCCESSFUL; irp->IoStatus.Information = request->Status; } SmartDevFreeBlock(request); IoCompleteRequest(irp, IO_NO_INCREMENT); nPackets++; } } while (nPackets != nPacketsOld); KeReleaseSpinLock(&devExt->writeListLock); L L Windows Device Drivers Unlocked Error U U

28. Source Code Testing Development API Usage Rules (SLIC) Software Model Checking Read for understanding New API rules Drive testing tools Defects 100% path coverage Rules Static Driver Verifier

29. Source Code Testing Development API Usage Rules (SLIC) Software Model Checking Read for understanding New API rules Drive testing tools Defects 100% path coverage Rules Static Driver Verifier

30. Architecture of Yogi Yogi IR (yir) C Program slamcl Slam.li database slicc SLIC property Instrumented Slam.li database li2yir Test case that exposes a bug Proof that program satisfies property Alias and mod-ref information Z3 theorem prover YAbsManYSim polymorphicregion graphs initial functions summaries

31. Implementation ~6K lines of F# Z3 theorem prover Integrated with SDV for Windows Engineering effort: 3 person years

32. Static Driver Verifier Runs #drivers = 69, #properties = 85 largest driver = ~30 KLOCs #KLOCs = ~350 KLOCS Yogi works on 129 runs where SLAM “gives up” Yogi takes ~ 12 hours whereas SLAM takes ~42 hours

33. Thanks to … Nels Beckman (CMU) Bhargav Gulavani (IIT Bombay) Thomas Henzinger (EPFL) Yamini Kannan (Berkeley) Robert Simmons (CMU) Leonardo de Moura (MSR Redmond) Nikolaj Bjorner (MSR Redmond)

34. http://research.microsoft.com/yogi