1. Practical Formal Verification of MPI and Thread Programs Sarvani Vakkalanka Anh Vo* Michael DeLisi Sriram Aananthakrishnan Alan Humphrey Christopher Derrick Yu Yang Ganesh Gopalakrishnan* Robert M. Kirby* * = presenters School of Computing, University of Utah, Salt Lake City, UT 84112, USA http:// www.cs.utah.edu / formal_verification / europvm09-tutorial-mpi-threading-fv Supported by NSF CNS 0509379, CCF 0811429, CCF 0903408, SRC tasks TJ 1847.001 and TJ 1993, and Microsoft 1

2. Other students involved: Salman Pervez, Robert Palmer, Guodong Li, Geof Sawaya, Subodh Sharma, Grzegorz Szubzda, Jason Williams, Simone Atzeni, Wei-Fan Chiang External Collaborators: ANL / UIUC : Rajeev Thakur, Bill Gropp, Rusty Lusk IBM : Beth Tibbits LLNL : Bronis de Supinski, Martin Schulz, Dan Quinlan Microsoft : Robert Palmer, Dennis Crain, Shahrokh Mortazavi Additional Acknowledgements for this tutorial

3. Overview of Formal Verification, especially Dynamic Verification Overview of MPI Demo of our tool ISP Architecture of ISP Presentation of Any_src_can_deadlock (from Umpire test suite) Our algorithm POE (Partial Order avoiding Elusive interleavings) Presentation of POE-Illustration Present details of POE-Illustration: ISP’s Eclipse framework and GUI Boot into LiveDVD and practice on POE-Illustration 9:00 to 10:30

4. Coffee Break IMPORTANT : Please give feedback before it is too late Too fast ? Too slow ? Just right !! ? Assuming a lot ? Other suggestions ? We will TRY to take into account these valuable suggestions! 10:30 to 11:00

5. Illustration of Resource Dependent Deadlocks, and Detection Illustration of Resource Leak, and Detection Iprobe behavior, and illustration using GUI Assertion Violation in Red/Blue Problem Audience Participation in Above Exercises ISP’s Theory : MPI Happens-before Also called “matches before, completes before” in the tool 11:00 to 12:00

6. Example of Matrix Multiplication: Four Variations Analysis of these variations using ISP, with Audience Participation 12:00 to 12:30

7. Assisted Problem Solving by Audience 14:00 to 15:00

8. Overview of Dynamic Verification of Shared Memory Thread Programs 15:00 to 15:30

9. Dynamic Verification of Thread Programs using Inspect Concluding Remarks 16:00 to 17:30

10. Overview of Formal Verification methods for Validating Concurrent Systems About 30 minutes – by Ganesh 10

11. 11 Problem: Engineering Reliable Concurrent Systems

12. Designers require a push-button debugger-like interface – But one that offers coverage guarantees and deeper insights For many important reasons, we advocate Dynamic Formal Verification methods

13. Testing methods suffer from bug omissions 13 X Bug Omissions For many important reasons, we advocate Dynamic Formal Verification methods

14. Testing methods suffer from bug omissions Static analysis methods generate many false alarms 14 XX Bug OmissionsFalse Alarms For many important reasons, we advocate Dynamic Formal Verification methods

15. Testing methods suffer from bug omissions Static analysis methods generate many false alarms Model based verification requires tedious model building 15 XXX Bug OmissionsFalse AlarmsTedious Modeling For many important reasons, we advocate Dynamic Formal Verification methods

16. Testing methods suffer from bug omissions Static analysis methods generate many false alarms Model based verification requires tedious model building Dynamic verification methods are ideal for designers! 16 XXX√ Bug OmissionsFalse AlarmsTedious Modeling No omissions No false alarms No need for modeling For many important reasons, we advocate Dynamic Formal Verification methods

17. Code written using mature libraries (MPI, OpenMP, PThreads, …) Code written using mature libraries (MPI, OpenMP, PThreads, …) API calls made from real programming languages (C, Fortran, C++) API calls made from real programming languages (C, Fortran, C++) Runtime semantics determined by realistic compilers and runtimes Dynamic Verification Methods are going to be very important for real engineers ! (static analysis and model based verification can play important supportive roles) 17 Growing Importance of Dynamic Verification

18. Verisoft Project – Used for telephone switch software verification in Bell Labs – Available A Brief Survey of Dynamic Verification tools

19. Verisoft Project – Used for telephone switch software verification in Bell Labs – Available The Java Pathfinder Project – Developed at NASA for Java Control Software – On SourceForge A Brief Survey of Dynamic Verification tools

20. Verisoft Project – Used for telephone switch software verification in Bell Labs – Available The Java Pathfinder Project – Developed at NASA for Java Control Software – On SourceForge The CHESS Project – Microsoft Research ; available for academic institutions – In use within Microsoft product groups, and used by academics A Brief Survey of Dynamic Verification tools

21. Verisoft Project – Used for telephone switch software verification in Bell Labs – Available The Java Pathfinder Project – Developed at NASA for Java Control Software – On SourceForge The CHESS Project – Microsoft Research ; available for academic institutions – In use within Microsoft product groups, and used by academics Inspect : Our fairly unique Pthread / C verifier – Discussed in this tutorial A Brief Survey of Dynamic Verification tools

22. Verisoft Project – Used for telephone switch software verification in Bell Labs – Available The Java Pathfinder Project – Developed at NASA for Java Control Software – On SourceForge The CHESS Project – Microsoft Research ; available for academic institutions – In use within Microsoft product groups, and used by academics Inspect : Our fairly unique Pthread / C verifier – Discussed in this tutorial ISP : Our very unique MPI / C program verifier – Main focus of THIS TUTORIAL !! A Brief Survey of Dynamic Verification tools

23. – Through PMPI Runs the code under a verification scheduler – ‘Hijacks’ native scheduler By interposing a profiler – Exerts its own Interleaving Generation Control Selective replay, Dynamic Instruction Rewriting – TRIES HARD to generate only RELEVANT interleavings Only replays around “non-determinism” – Does ‘stateless’ (replay) verification Restarts from MPI_Init for each new interleaving Example : How ISP Effects Dynamic Verification

24. Somehow Instruments the Source / Binary – Through PMPI Runs the code under a verification scheduler – ‘Hijacks’ native scheduler By interposing a profiler – Exerts its own Interleaving Generation Control Selective replay, Dynamic Instruction Rewriting – TRIES HARD to generate only RELEVANT interleavings Only replays around “non-determinism” – Does ‘stateless’ (replay) verification Restarts from MPI_Init for each new interleaving Example : How ISP Effects Dynamic Verification

25. Somehow Instruments the Source / Binary – Through PMPI at source level Runs the code under a verification scheduler – ‘Hijacks’ native scheduler By interposing a profiler – Exerts its own Interleaving Generation Control Selective replay, Dynamic Instruction Rewriting – TRIES HARD to generate only RELEVANT interleavings Only replays around “non-determinism” – Does ‘stateless’ (replay) verification Restarts from MPI_Init for each new interleaving Example : How ISP Effects Dynamic Verification

26. Somehow Instruments the Source / Binary – Through PMPI Runs the code under a verification scheduler – ‘Hijacks’ native scheduler By interposing a profiler – Exerts its own Interleaving Generation Control Selective replay, Dynamic Instruction Rewriting – TRIES HARD to generate only RELEVANT interleavings Only replays around “non-determinism” – Does ‘stateless’ (replay) verification Restarts from MPI_Init for each new interleaving Example : How ISP Effects Dynamic Verification

27. Somehow Instruments the Source / Binary – Through PMPI Runs the code under a verification scheduler – ‘Hijacks’ MPI Function Calls By interposing a profiler – Exerts its own Interleaving Generation Control Selective replay, Dynamic Instruction Rewriting – TRIES HARD to generate only RELEVANT interleavings Only replays around “non-determinism” – Does ‘stateless’ (replay) verification Restarts from MPI_Init for each new interleaving Example : How ISP Effects Dynamic Verification

28. Somehow Instruments the Source / Binary – Through PMPI Runs the code under a verification scheduler – ‘Hijacks’ MPI Function Calls By interposing a profiler – Exerts its own Interleaving Generation Control Selective replay, Dynamic Instruction Rewriting – TRIES HARD to generate only RELEVANT interleavings Only replays around “non-determinism” – Does ‘stateless’ (replay) verification Restarts from MPI_Init for each new interleaving Example : How ISP Effects Dynamic Verification

29. Somehow Instruments the Source / Binary – Through PMPI Runs the code under a verification scheduler – ‘Hijacks’ MPI Function Calls By interposing a profiler – Exerts its own Interleaving Generation Control Selective replay, Dynamic Instruction Rewriting – TRIES HARD to generate only RELEVANT interleavings Only replays around “non-determinism” – Does ‘stateless’ (replay) verification Restarts from MPI_Init for each new interleaving Example : How ISP Effects Dynamic Verification

30. Somehow Instruments the Source / Binary – Through PMPI Runs the code under a verification scheduler – ‘Hijacks’ MPI Function Calls By interposing a profiler – Exerts its own Interleaving Generation Control Selective replay, Dynamic Instruction Rewriting – TRIES HARD to generate only RELEVANT interleavings Only replays around “non-determinism” – Does ‘stateless’ (replay) verification Restarts from MPI_Init for each new interleaving Example : How ISP Effects Dynamic Verification

31. Somehow Instruments the Source / Binary – Through PMPI Runs the code under a verification scheduler – ‘Hijacks’ MPI Function Calls By interposing a profiler – Exerts its own Interleaving Generation Control Selective replay, Dynamic Instruction Rewriting – TRIES HARD to generate only RELEVANT interleavings Only replays around “non-determinism” – Does ‘stateless’ (replay) verification Restarts from MPI_Init for each new interleaving Example : How ISP Effects Dynamic Verification

32. Somehow Instruments the Source / Binary – Through PMPI Runs the code under a verification scheduler – ‘Hijacks’ MPI Function Calls By interposing a profiler – Exerts its own Interleaving Generation Control Selective replay, Dynamic Instruction Rewriting – TRIES HARD to generate only RELEVANT interleavings Only replays around “non-determinism” – Does ‘stateless’ (replay) verification Restarts from MPI_Init for each new interleaving Example : How ISP Effects Dynamic Verification

33. Somehow Instruments the Source / Binary – Through PMPI Runs the code under a verification scheduler – ‘Hijacks’ MPI Function Calls By interposing a profiler – Exerts its own Interleaving Generation Control Selective replay, Dynamic Instruction Rewriting – TRIES HARD to generate only RELEVANT interleavings Only replays around “non-determinism” – Does ‘stateless’ (replay) verification Restarts from MPI_Init for each new interleaving Example : How ISP Effects Dynamic Verification

34. Somehow Instruments the Source / Binary – Through PMPI Runs the code under a verification scheduler – ‘Hijacks’ MPI Function Calls By interposing a profiler – Exerts its own Interleaving Generation Control Selective replay, Dynamic Instruction Rewriting – TRIES HARD to generate only RELEVANT interleavings Only replays around “non-determinism” – Does ‘stateless’ (replay) verification Restarts from MPI_Init for each new interleaving Example : How ISP Effects Dynamic Verification

35. L0L0 U0U0 L1L1 L2L2 U1U1 U2U2 L0L0 U0U0 L2L2 U2U2 L1L1 U1U1 35 Sketch of Stateless / Replay Verification Red, Green, and Blue moves Belong to different processes Dotted arrow shows some Dependency (e.g., runtime non-determinism) Start system In Initial State

36. Exponential number of TOTAL Interleavings – most are EQUIVALENT – generate only RELEVANT ones !! 36 P0 P1 P2 P3 P4 TOTAL > 10 Billion Interleavings !!

37. Exponential number of TOTAL Interleavings – most are EQUIVALENT – generate only RELEVANT ones !! 37 P0 P1 P2 P3 P4 A B1 These are the only dependent actions E.g. One ANY-SOURCE (wildcard) receive And two of its MATCHING SENDS Point-to-point actions can be issued in ANY order TOTAL > 10 Billion Interleavings !! B2 A B1

38. Exponential number of TOTAL Interleavings – most are EQUIVALENT – generate only RELEVANT ones !! 38 P0 P1 P2 P3 P4 A B1 These are the only dependent actions E.g. One ANY-SOURCE (wildcard) receive And two of its MATCHING SENDS Point-to-point actions can be issued in ANY order Only TWO RELEVANT Interleavings ! TOTAL > 10 Billion Interleavings !! B2 A B1

39. Executable Proc 1 Proc 2 …… Proc n Scheduler that generates ALL RELEVANT schedules (one per partial order) Run MPI Runtime 39 MPI Program Interposition Layer Workflow of ISP

40. 40 P0 P1 P2 Barrier Isend(1, req) Wait(req) Scheduler Irecv(*, req) Barrier Recv(2) Wait(req) Isend(1, req) Wait(req) Barrier Isend(1) sendNext Barrier MPI Runtime Hijack Calls, Generate Relevant Interleavings

41. P0 P1 P2 Barrier Isend(1, req) Wait(req) Scheduler Irecv(*, req) Barrier Recv(2) Wait(req) Isend(1, req) Wait(req) Barrier Isend(1) sendNext Barrier Irecv(*) Barrier 41 MPI Runtime Hijack Calls, Generate Relevant Interleavings

42. P0 P1 P2 Barrier Isend(1, req) Wait(req) Scheduler Irecv(*, req) Barrier Recv(2) Wait(req) Isend(1, req) Wait(req) Barrier Isend(1) Barrier Irecv(*) Barrier Barrier Barrier 42 MPI Runtime Hijack Calls, Generate Relevant Interleavings

43. P0 P1 P2 Barrier Isend(1, req) Wait(req) MPI Runtime Scheduler Irecv(*, req) Barrier Recv(2) Wait(req) Isend(1, req) Wait(req) Barrier Isend(1) Barrier Irecv(*) Barrier Wait (req) Recv(2) Isend(1) SendNext Wait (req) Irecv(2) Isend Wait No Match-Set No Match-Set 43 Deadlock! Hijack Calls, Generate Relevant Interleavings

44. lucky.c has a deadlock that shows upon testing unlucky.c does not reveal a deadlock upon testing Testing is done using mpicc ; mpirun Verification is done using ispcc ; isp Let us see ISP in action on ‘lucky.c’ and ‘unlucky.c’

45. Process P0 R(from:*, r1) ; R(from:2, r2); S(to:2, r3); R(from:*, r4); All the Ws… Process P1 Sleep(3); S(to:0, r1); All the Ws… Process P2 //Sleep(3); S(to:0, r1); R(from:0, r2); S(to:0, r3); All the Ws… 45 Example MPI program ‘lucky.c’ (lucky for tester)

46. Process P0 R(from:*, r1) ; R(from:2, r2); S(to:2, r3); R(from:*, r4); All the Ws… Process P1 // Sleep(3); S(to:0, r1); All the Ws… Process P2 Sleep(3); S(to:0, r1); R(from:0, r2); S(to:0, r3); All the Ws… 46 MPI program ‘unlucky.c’

47. Runs of lucky.c and unlucky.c on mpich using “standard testing” (“lucky” for tester) 47 mpicc lucky.c -o lucky.out mpirun -np 3./lucky.out (0) is alive on ganesh-desktop (1) is alive on ganesh-desktop (2) is alive on ganesh-desktop Rank 0 did Irecv Rank 2 did Send Sleep over Rank 1 did Send [.. hang..] mpicc unlucky.c -o unlucky.out mpirun -np 3./unlucky.out (0) is alive on ganesh-desktop (2) is alive on ganesh-desktop (1) is alive on ganesh-desktop Rank 0 did Irecv Rank 1 did Send Rank 0 got 11 Sleep over Rank 2 did Send (2) Finished normally (1) Finished normally (0) Finished normally [.. OK..]

48. Runs of lucky.c and unlucky.c on mpich using “standard testing” (“lucky” for tester) 48 mpicc lucky.c -o lucky.out mpirun -np 3./lucky.out (0) is alive on ganesh-desktop (1) is alive on ganesh-desktop (2) is alive on ganesh-desktop Rank 0 did Irecv Rank 2 did Send Sleep over Rank 1 did Send [.. hang..] mpicc unlucky.c -o unlucky.out mpirun -np 3./unlucky.out (0) is alive on ganesh-desktop (2) is alive on ganesh-desktop (1) is alive on ganesh-desktop Rank 0 did Irecv Rank 1 did Send Rank 0 got 11 Sleep over Rank 2 did Send (2) Finished normally (1) Finished normally (0) Finished normally [.. OK..] ispcc ; isp will detect deadlock in both cases !!

49. With ISP at hand, WE ARE LUCKY IN BOTH CASES Not just ‘feeling lucky’ !! COMMANDS RUN : Ispcc lucky.c [ later try unlucky.c ] Isp -n 3 -log /tmp/log1./a.out ispUI /tmp/log1 Commands to verify lucky.c or unlucky.c

50. End of A 50