1. EFFICIENT DYNAMIC VERIFICATION ALGORITHMS FOR MPI APPLICATIONS Dissertation Defense Sarvani Vakkalanka Committee: Prof. Ganesh Gopalakrishnan (advisor), Prof. Mike Kirby (co- advisor), Prof. Suresh Venkatasubramanian, Prof. Matthew Might, Prof. Stephen Siegel (Univ. of Delaware) 1

2. Necessity for Verification Software testing is ad-hoc. Software Errors expensive - $59.5 Billion/yr (2001 NTSI Study). Software written today is complex and uses many existing libraries. Our focus – contribute to – Parallel scientific software written using MPI 2

3. Motivation Concurrent software debugging is hard! Very little formal support for Message Passing concurrency. Active testing (schedule enforcement) is important. Reducing redundant (equivalent) verification runs is crucial. Verification for portability – another important requirement. 3

4. Approaches to Verification Testing methods suffer from bug omissions. Static analysis based methods generate many false alarms. Model based verification is tedious. Dynamic verification – no false alarms 4

5. Contributions New dynamic verification algorithms for MPI. New Happens-Before models for Message Passing concurrency. Verification to handle resource dependency. MPI dynamic verification tool ISP that handles non-trivial codes for safety properties. 5

6. Agenda Intro to Dynamic Verification Intro to MPI – Four MPI Operations (S, R, W, B). – MPI Ordering Guarantees. – Applying DPOR to MPI Dynamic verification algorithms avoiding redundant searches and handling resource dependencies Formal MPI Transition System Experimental Results Conclusions 6

7. EFFICIENT DYNAMIC VERIFICATION 7

8. Code written using mature libraries (MPI, OpenMP, PThreads, …) Code written using mature libraries (MPI, OpenMP, PThreads, …) API calls made from real programming languages (C, Fortran, C++) API calls made from real programming languages (C, Fortran, C++) Runtime semantics determined by realistic compilers and runtimes Dynamic Verification abstracts verification details. (static analysis and model based verification can play important supportive roles) 8 Growing Importance of Dynamic Verification

9. Exponential number of TOTAL Interleavings – most are EQUIVALENT – generate only RELEVANT ones !! 9 P0 P1 P2 P3 P4 TOTAL > 10 Billion Interleavings !! a++ b--g=2g=3

10. Dynamic Partial Order Reduction 10 P0 P1 P2 P3 P4 Dependent actions Only these 2 are RELEVANT!!! TOTAL > 10 Billion Interleavings !! All other actions are pairwise independent g=2g=3

11. DPOR A state σ consists of the following sets: – enabled(σ) – backtrack(σ) : sufficient subset of enabled(σ) – enabled(σ) = backtrack(σ), then the full state space is explored. Co-enabledness of transitions Dependence among transitions 11 σ σ

12. Co-enabledness & Dependence 12 t1t1 t1t1 t2t2 t2t2 {t 2 } {t 1 } {t 1, t 2 }

13. DPOR Concepts DPOR requires the identification of dependence and co-enabledness among transitions Identifying dependence is simple – Two lock accesses on the same mutex. – Two writes to the same global variable. – Similar concepts for MPI. Identifying co-enabledness is difficult (like will happen in parallel). 13

14. P1 P2 lock(l) x = 1 x = 2 unlock(l) lock(l) y = 1 x = 2 unlock(l) 14 Illustration of DPOR Concepts

15. P1 P2 lock(l) x = 1 x = 2 unlock(l) lock(l) y = 1 x = 2 unlock(l) 15 Illustration of DPOR Concepts

16. Thread Verification vs MPI Verification Thread verification – well studied!. – Well known dynamic verification tools on thread verification [CHESS, INSPECT]. – Thread verification follows traditional dynamic partial order reduction. DPOR does not extend directly for MPI MPI Verification – not so! – requires a formal definition. – out-of-order completion semantics. – Must define dependence 16

17. INTRODUCTION TO MPI 17

18. 18 IBM Blue Gene (Picture Courtesy IBM) LANL’s Petascale machine “Roadrunner” (AMD Opteron CPUs and IBM PowerX Cell) The choice for ALL large-scale parallel simulations (earthquake, weather..) Runs “everywhere”. Very mature codes exist in MPI – tens of person years. Performs critical simulations in science and engineering. The Ubiquity of MPI

19. Overview of Message Passing Interface (MPI) API One of the major Standardization Successes. Lingua franca of Parallel Computing Runs on parallel machines of a WIDE range of sizes Standard is published at www.mpi-forum.orgwww.mpi-forum.org MPI 2.0 includes over 300 functions 19

20. MPI Execution Environment MPI execution environment consists of two main components: – MPI processes. – The MPI runtime daemon. All processes statically created. Process rank between 0 and n-1. The MPI processes issue instructions into MPI runtime. The MPI runtime implements and executes the MPI library. 20

21. MPI Execution Contd… Every process starts execution with MPI_Init(int argc, char **argv); MPI_Finalize – at the end 21

22. Abbreviated as S MPI_Isend (void *buff, …, int dest, int tag, MPI_Comm comm, MPI_Request handle); MPI_Isend (void *buff, …, int dest, int tag, MPI_Comm comm, MPI_Request handle); 22

23. Abbreviated as R MPI_Irecv (void *buff, …, int src, int tag, MPI_Comm comm, MPI_Request *handle); MPI_Irecv (void *buff, …, int src, int tag, MPI_Comm comm, MPI_Request *handle); 23

24. Abbreviated as W MPI_Wait (MPI_Request *handle, MPI_Status *status); 24

25. Abbreviated as B. All processes must invoke B before any can get past. MPI_Barrier (MPI_Comm comm); 25

26. MPI Ordering Guarantees 26

27. MPI Ordering Guarantees 27

28. MPI Ordering Guarantee 28

29. Applying DPOR to MPI 29 Programs like this – almost impossible to test on real platforms.

30. Why DPOR does not work! 30

31. Modifying Runtime Doesn’t Help! Assume that the MPI runtime is modified to support verification The sends are matched with receives in the order they are issued to the MPI runtime Is this sufficient? 31

32. Crooked Barrier Example P0 P1 P2 Isend(1, req) Isend(1, req) Barrier Irecv(*, req) Irecv(*, req) Barrier Wait(req) Irecv(2, req1) Wait(req1) Wait(req) Isend(1, req) Wait(req) Verification Support does not work! 32

33. Our Main Algorithms Partial Order avoiding Elusive Interleavings (POE). POE OPT : Reduced interleavings even further. POE MSE : Handle resource dependencies. 33

34. Illustration of POE P0 P1 P2 Barrier Isend(1, req) Wait(req) MPI Runtime Scheduler Irecv(*, req) Barrier Recv(2) Wait(req) Isend(1, req) Wait(req) Barrier Isend(1) sendNext Barrier 34

35. P0 P1 P2 Barrier Isend(1, req) Wait(req) MPI Runtime Scheduler Irecv(*, req) Barrier Recv(2) Wait(req) Isend(1, req) Wait(req) Barrier Isend(1) sendNext Barrier Irecv(*) Barrier 35 Illustration of POE

36. P0 P1 P2 Barrier Isend(1, req) Wait(req) MPI Runtime Scheduler Irecv(*, req) Barrier Recv(2) Wait(req) Isend(1, req) Wait(req) Barrier Isend(1) Barrier Irecv(*) Barrier Barrier Barrier 36 Illustration of ISP’s Verification Algorithm

37. P0 P1 P2 Barrier Isend(1, req) Wait(req) MPI Runtime Scheduler Irecv(*, req) Barrier Recv(2) Wait(req) Isend(1, req) Wait(req) Barrier Isend(1) Barrier Irecv(*) Barrier Wait (req) Recv(2) Isend(1) SendNext Wait (req) Irecv(2) Isend Wait No Matching Send No Matching Send Deadlock! 37 Illustration of POE IntraCB

38. Notations MPI_Isend : S i,j (k), where – i is the process issuing the send, – j is the dynamic execution count of S in process i and – k is the destination process rank where the message is to be sent MPI_Irecv: R i,j (k) – k is the source MPI_Barrier: B i,j MPI_Wait : W i,j’ (h i,j ) – h i,j is the request handle of S i,j (k) or R i,j (k) 38

39. POE Issue: Redundancy 39 POE explores both the match-sets resulting in 2 interlevings while just 1 interleaving is sufficient. SOLUTION : Explore only match-sets for single wildcard receive. DOES NOT WORK! BREAKS PERSISTENCE.

40. POE and Persistent Sets Add only this match-set to bactrack 40 Maintaining Persistent backtrack sets is important. Otherwise, verification algorithm is broken

41. POE Issue: Buffering Deadlocks When no sends are buffered Deadlock! 41

42. POE Issue: Redundancy Simple Optimization: If there is no more sends targeting a wildcard receive, then add only of of the match-sets to backtrack set. 42

43. Redundancy : POE OPT P0 P1 P2 P3 W 0,2 (h 0,1 ) S 0,1 (1) R 1,1 (*) W 1,2 (h 1,1 ) S 1,3 (3) W 0,4 (h 1,3 ) R 1,5 (*) W 1,6 (h 1,5 ) W 2,2 (h 2,1 ) R 2,1 (1) S 3,1 (*) W 3,2 (h 3,1 ) R 3,3 (1) W 3,4 (h 3,3 ) S 3,5 (1) W 3,6 (h 3,5 ) 43

44. Detecting Matching Exploring all non-deterministic matchings in a state is not a solution The IntraHB relation is not sufficient to detect matchings across processes We introduce the notion of Inter-HB 44

45. InterHB Relation 45

46. Redundancy : POE OPT P0 P1 P2 P3 W 0,2 (h 0,1 ) S 0,1 (1) R 1,1 (*) W 1,2 (h 1,1 ) S 1,3 (3) W 0,4 (h 1,3 ) R 1,5 (*) W 1,6 (h 1,5 ) W 2,2 (h 2,1 ) R 2,1 (*) S 3,1 (2) W 3,2 (h 3,1 ) R 3,3 (1) W 3,4 (h 3,3 ) S 3,5 (1) W 3,6 (h 3,5 ) 46

47. Redundancy : POE OPT W 0,2 (h 0,1 ) S 0,1 (1) R 1,1 (*) W 1,2 (h 1,1 ) R 1,3 (3) W 1,4 (h 1,3 ) W 2,2 (h 2,1 ) R 2,1 (*) S 3,1 (2) W 3,2 (h 3,1 ) S 3,3 (1) W 3,4 (h 3,3 ) P0 P1 P2 P3 P4 P5 R 4,1 (*) W 4,2 (h 4,1 ) S 5,1 (1) W 5,2 (h 5,1 ) NO PATH 47

48. Slack/Buffering Deadlocks Deadlocks only when S 0,1 or S 1,1 or both are buffered 48

49. Buffer All Sends ??? ZERO SLACK 49

50. Buffer All Sends ??? ZERO SLACK 50

51. Buffer All Sends ??? INF SLACK 51

52. Buffer All Sends ??? INF SLACK 52

53. Buffer All Sends ??? ONLY S 0,0 Deadlock! 53

54. Slack/Buffering : POE MSE P0 P1 P2 W 0,2 (h 0,1 ) S 0,1 (1) S 1,1 (2) W 1,2 (h 1,1 ) R 1,3 (0) W 1,4 (h 1,3 ) R 2,3 (0) W 2,4 (h 2,3 ) W 2,2 (h 2,1 ) R 2,1 (*) S 0,3 (2) W 0,4 (h 0,3 ) 54

55. Slack/Buffering : POE MSE P0 P1 P2 W 0,2 (h 0,1 ) S 0,1 (1) S 1,1 (2) W 1,2 (h 1,1 ) R 1,3 (0) W 1,4 (h 1,3 ) R 2,3 (0) W 2,4 (h 2,3 ) W 2,2 (h 2,1 ) R 2,1 (*) S 0,3 (2) W 0,4 (h 0,3 ) No-op 55

56. POE MSE Finds all paths between a wildcard receive and a matching send. If there is a path without a culprit wait in it, then does nothing. If every path contains at least one culprit wait, then the algorithm finds all ways to break paths by trying to select exactly one wait in every path. – We call this as finding minimal wait sets. – NP-Complete problem (Proved by reduction from 1-in-3 SAT). – Finding all minimal wait sets in #P-Complete. 56

57. Minimal Wait Sets Find the power-set of all the culprit waits. Sort the Power-set by size in non-decreasing order. For each subset if it breaks all paths – Delete all it supersets from the power-set. If it does not break all the paths – Delete it from the power-set 57

58. Slack/Buffering : POE MSE P0 P1 P2 W 0,2 (h 0,1 ) S 0,1 (1) S 1,1 (2) W 1,2 (h 1,1 ) R 1,3 (0) W 1,4 (h 1,3 ) R 2,3 (0) W 2,4 (h 2,3 ) W 2,2 (h 2,1 ) R 2,1 (*) S 0,3 (2) W 0,4 (h 0,3 ) 58

59. MPI TRANSITION SYSTEM 59

60. MPI State and Transitions An MPI function is in one or more of the following states: – Issued (I) – Matched (M) – Complete (C) – Returned (R) A global state is – initial state is Two kinds of transitions: – Process transitions – MPI runtime transitions 60

61. Process Transitions 61

62. MPI Runtime Book-keeping Sets No Ancestor of x is in Ready set All ancestors are matched All ancestors are Complete 62

63. IntraHB - Crooked Barrier Example P0 P1 P2 Isend(1, req) Barrier Irecv(*, req) Barrier Wait(req) Irecv(2, req1) Wait(req1) Wait(req) Isend(1, req) Wait(req) 63

64. MPI Runtime Transitions Zero Buffering 64

65. MPI Runtime Transitions Contd.. 65

66. MPI Runtime Transition Conditional Happens-Before Dynamic source re-write 66

67. Simple MPI Example 67

68. 68

69. 69

70. Dependence and Independence Properties 70

71. MPI Dependence 71

72. RESULTS – Real Benchmarks Game of Life EuroPVM / MPI 2007 versions of Gropp and Lusk done in seconds MADRE – Siegels’ Mem Aware Data Redistrib. Engine Found previously documented deadlock ParMETIS – Hypergraph Partitioner Initial run of days reduced now to seconds on a laptop MPI-BLAST – Genome sequencer using BLAST Runs to completion on small instances A few MPI Spec Benchmarks Some benchmarks exhibit interleaving explosion; others OK ADLB Initial experiments have been successful 72

73. Results Resource Leak caught in Parmetis ISP used in the development cycle of A* algorithm: – Found 3 deadlocks during various implementation phases. – All deadlocks were unintentional (not seeded) 73

74. Results Contd… Umpire Program POE Marmot any_src-can-deadlock7.cDeadlock detected 2 interleavings Deadlock Caught in 5/10 runs any_src-can-deadlock10.cDeadlock Detected 1 interleaving Deadlock caught 7/10 runs basic-deadlock10.cDeadlock detected in 1 interleaving Deadlock caught in 10/10 runs 74

75. POE OPT 75

76. POE MSE 76

77. How well did we do? Verisoft Project – Used for telephone switch software verification in Bell Labs – Available The Java Pathfinder Project – Developed at NASA for Java Control Software – On SourceForge The CHESS Project – Microsoft Research; available for academic institutions – In use within Microsoft product groups and used by academics Inspect : UV group’s unique Pthread verifier. – Available for download ISP : dynamic verification tool for MPI – Implements the dynamic verification algorithms from this dissertations. – Available for download with the PTP (Parallel Tools Platform). 77

78. CONCLUSIONS First efficient and practical dynamic reduction based algorithms for real MPI programs. Verification for portability with respect to buffering. First Happens-Before model for MPI. ISP scheduler directly based on our theory. ISP+GEM released and demoed widely. 78

79. Questions & Answers 79 THANK YOU

80. POE ALGORITHM 80

81. POE Proof of Correctness 81

82. POE Illustration 82

83. Dynamic Verification No modeling effort for the programmer Program is the model – the actual program is verified Push-button interface : easy to use. On the downside – verification is a function of input. – Most programs are fairly data independent. Dynamic verification methods are ideal for programmers! 83