1. Poirot – A Concurrency Sleuth Shaz Qadeer Research in Software Engineering Microsoft Research

2. Concurrent programming is difficult

3. … if (irp->Cancel) { IoCompleteIrp(irp); } else { IoSetCancelRoutine(irp, CancelRoutine); IoMarkIrpPending(irp); } … irp->Cancel = TRUE; fn = IoSetCancelRoutine(Irp, NULL); if (fn) { fn(irp); } … void CancelRoutine(IRP *irp) { IoCompleteIrp(irp); } NormalCancellation IO_REQUEST_PACKET *irp; irp->Cancel = FALSE; irp->CancelRoutine = NULL;

4. … if (irp->Cancel) { IoCompleteIrp(irp); } else { IoSetCancelRoutine(irp, CancelRoutine); IoMarkIrpPending(irp); } … irp->Cancel = TRUE; fn = IoSetCancelRoutine(Irp, NULL); if (fn) { fn(irp); } … void CancelRoutine(IRP *irp) { IoCompleteIrp(irp); } NormalCancellation IO_REQUEST_PACKET *irp; irp->Cancel = FALSE; irp->CancelRoutine = NULL;

5. … if (irp->Cancel) { IoCompleteIrp(irp); } else { IoSetCancelRoutine(irp, CancelRoutine); IoMarkIrpPending(irp); } … irp->Cancel = TRUE; fn = IoSetCancelRoutine(Irp, NULL); if (fn) { fn(irp); } … void CancelRoutine(IRP *irp) { IoCompleteIrp(irp); } NormalCancellation IO_REQUEST_PACKET *irp; irp->Cancel = FALSE; irp->CancelRoutine = NULL;

6. … if (irp->Cancel) { IoCompleteIrp(irp); } else { IoSetCancelRoutine(irp, CancelRoutine); IoMarkIrpPending(irp); } … irp->Cancel = TRUE; fn = IoSetCancelRoutine(Irp, NULL); if (fn) { fn(irp); } … void CancelRoutine(IRP *irp) { IoCompleteIrp(irp); } NormalCancellation IO_REQUEST_PACKET *irp; irp->Cancel = FALSE; irp->CancelRoutine = NULL;

7. … if (irp->Cancel) { IoCompleteIrp(irp); } else { IoSetCancelRoutine(irp, CancelRoutine); IoMarkIrpPending(irp); } … irp->Cancel = TRUE; fn = IoSetCancelRoutine(Irp, NULL); if (fn) { fn(irp); } … void CancelRoutine(IRP *irp) { IoCompleteIrp(irp); } NormalCancellation IO_REQUEST_PACKET *irp; irp->Cancel = FALSE; irp->CancelRoutine = NULL;

8. … if (irp->Cancel) { IoCompleteIrp(irp); } else { IoSetCancelRoutine(irp, CancelRoutine); IoMarkIrpPending(irp); } … irp->Cancel = TRUE; fn = IoSetCancelRoutine(Irp, NULL); if (fn) { fn(irp); } … void CancelRoutine(IRP *irp) { IoCompleteIrp(irp); } NormalCancellation IO_REQUEST_PACKET *irp; irp->Cancel = FALSE; irp->CancelRoutine = NULL;

9. … if (irp->Cancel) { IoCompleteIrp(irp); } else { IoSetCancelRoutine(irp, CancelRoutine); IoMarkIrpPending(irp); } … irp->Cancel = TRUE; fn = IoSetCancelRoutine(Irp, NULL); if (fn) { fn(irp); } … void CancelRoutine(IRP *irp) { IoCompleteIrp(irp); } NormalCancellation Fatal error! IO_REQUEST_PACKET *irp; irp->Cancel = FALSE; irp->CancelRoutine = NULL;

10. Concurrent programming is difficult Multiple loci of control resulting in non-local control flow Code difficult to understand and review

11. What about verification? Assertion-based modular reasoning becomes complicated due to non-local interactions – Floyd-Hoare morphs into Owicki-Gries Even with simple (finite) abstractions, the presence of concurrency makes the analysis computationally very expensive SequentialConcurrent Single Procedure P-timePSPACE-complete Multi Procedure P-timeUndecidable

12. What about testing? Number of executions = O( n nk ) Exponential in both n and k x = 1; … x = k; x = 1; … x = k; … Thread 1Thread n Scheduling nondeterminism Uncontrollable Unobservable Exponential

13. Concurrency is important More than ever before Increasing importance of communicating systems – networked devices – cyber-physical systems Distributed programs running on the cloud – EC2, Azure, AppEngine, … Parallel programs running on multicores and GPUs – TBB, TPL, CUDA, AMP, …

14. Concurrency testing with CHESS Deterministic scheduling – make scheduling choices observable and controllable Search prioritization – combating the combinatorial explosion of possible schedules

15. Deterministic scheduling Kernel: Threads, Scheduler, Synchronization Objects While(not done){ TestScenario() } TestScenario(){ … } Program CHESS Win32 API Tester Provides a Test Scenario CHESS runs the scenario in a loop Each run is a different interleaving Each run is repeatable

16. Search prioritization (I) Given p ≥ 0, generate all schedules with up to p preemptions Pseudo-polynomial number of schedules – polynomial in preemption bound and schedule points – exponential in number of threads Many bugs with fewer than 2 preemptions Simple error traces for easier debugging

17. Search prioritization (II) Given p ≥ 0 and deterministic schedulers S 0, …, S p-1, schedule according to S 0, …, S p-1 in sequence moving from one to next nondeterministically – e.g., round-robin non-preemptive scheduling with p different round-robin orders Polynomial number of schedules Testers can innovate by designing domain- specific deterministic schedulers

18. CHESS is available Used internally by Microsoft product groups and externally by Microsoft customers Binary and source code available at: –http://chesstool.codeplex.comhttp://chesstool.codeplex.com

19. Limitations of CHESS Exposing and gaining control of scheduling choices is difficult – most implementation effort and user frustration due to this problem Testing components that interact extensively with the environment is difficult Input coverage is not addressed

20. Static program exploration with Poirot Symbolic instead of concrete execution C: Source code for software component E: Model for environment and scheduler Explore behaviors of C+E – for all symbolic inputs – for all scheduling choices

21. Disk AsyncRead(…) { } DiskReader(…) { } DiskReader(…) { } headtail Request queueIn-memory cache cache cacheSize Demo: Asynchronous File I/O

22. Poirot architecture Trace Viewer Concurrent.NET Program Concurrent Boogie Program Coverage Report.NET  Boogie Concurrent C Program C  Boogie Corral

23. Sequentialization Stratified Search Error Trace Concurrent Boogie Program Sequential Boogie Program Coverage Report Searching with Corral Refinement Abstraction Concurrent Boogie Program

24. Abstraction Set of global variables G Set of tracked variables T Drop writes to variables in G-T Replace reads to variables in G-T with nondeterministic values

25. Sequentialization Stratified Search Error Trace Concurrent Boogie Program Sequential Boogie Program Coverage Report Searching with Corral Refinement Abstraction Concurrent Boogie Program

26. Refinement Path p – feasible if only variables in T are tracked – infeasible if all variables in G are tracked Expand tracked set T to U such that p infeasible while tracking only variables in U Naïve algorithm: linear scan of G-T New divide-and-conquer algorithm – best case log(|G-T|) – worst case 2*|G-T|

27. Sequentialization Stratified Search Error Trace Concurrent Boogie Program Sequential Boogie Program Coverage Report Searching with Corral Refinement Abstraction Concurrent Boogie Program

28. Sequentialization (I) Given a concurrent program P, construct a sequential program Q such that Q  P Drop each occurrence of async-call Convert each occurrence of async-call to call

29. Sequentialization (II) Given a concurrent program P, construct a family of programs Q i such that – Q 0  Q 1  Q 2  …  P –  i Q i = P Even better if interesting behaviors of P manifest in Q i for low values of i

30. Context-bounding Under-approximation parameterized by K ≥ 0 – executions in which each thread gets at most K contexts to execute As K  , we get all behaviors Can we create sequentializations for context- bounding?

31. Sequentializing context switches Shared Memory T1T1 T2T2 Local Memory Execution: T1T1 T2T2 T1T1 T2T2 T1T1 T1T1 T2T2 T1T1 (s 1, l 1 )(s 2, l 2 ) s2s2 l2l2

32. Guess and verify T1T1 T2T2 T1T1 (s 1, l 1 ) (s 2, l 2 )(s 3, l 2 ) Guess the effect of T 2 Verify the guess Make copies of global variables Source-to-source translation – linear in program size and K Generalizes to dynamically-created threads

33. Sequentialization Stratified Search Error Trace Concurrent Boogie Program Sequential Boogie Program Coverage Report Searching with Corral Refinement Abstraction Concurrent Boogie Program

34. Stratified search main … Call tree given recursion bound r VC(T) assert no bug Summaries(L) T L assert no bug VC(p) Convert loops to recursive calls

35. Poirot status Medium-sized C programs – up to 20K low-level systems code – reports precise traces at scale Small.NET programs – bytecode to Boogie translator in progress Try: http://www.rise4fun.com/Poirothttp://www.rise4fun.com/Poirot Download available

36. Why bounded search? Data: Boolean, Integers, Arrays Control: Sequencing, Choice, Iteration, Call, Async-Call Sequencing Choice NP-complete Sequencing Choice Iteration Call Async-call Undecidable Sequencing Choice Iteration Call Async-call + bound Decidable PSPACE-hard Advances in SAT/SMT-solvers have made this problem tractable HAVOC verifier deployed for security analysis in Windows/IE Rationale: It is better to fail at the simpler problem!

37. Poirot collaborators Akash Lal, MSR Bangalore Shuvendu Lahiri, MSR Redmond

38. Questions