Protocol Verification in Millipede Jan Bækgaard Pedersen & Alan Wagner University of British Columbia Vancouver, Canada
Verified - so what? Consider the following scenario: CPA 2001 - BRISTOL 19 September, 2001 Verified - so what? Consider the following scenario: Write a protocol specification in some formal language like CSP/SMV/Mur Verify it; if it checks out implement it in C/C++ and PVM/MPI Jan B. Pedersen - Protocol Verification in Millipede
Verified - so what? Consider the following scenario: CPA 2001 - BRISTOL 19 September, 2001 Verified - so what? Consider the following scenario: Write a protocol specification in some formal language like CSP/SMV/Mur Verify it; if it checks out implement it in C/C++ and PVM/MPI Problem: What if the implementation is WRONG? Jan B. Pedersen - Protocol Verification in Millipede
Don’t know how to! How about this scenario: CPA 2001 - BRISTOL 19 September, 2001 Don’t know how to! How about this scenario: The C/PVM programmer does not know how to use CSP/SMV/Mur What now? Jan B. Pedersen - Protocol Verification in Millipede
Debugging Parallel Message Passing Programs CPA 2001 - BRISTOL 19 September, 2001 Debugging Parallel Message Passing Programs Errors can occur at different levels: Errors in sequential code Array out of bound, Pointer errors, Arithmetic under/over flow etc. Errors in the contents of messages Wrong contents, wrong length Processes can deadlock The communication protocol can be wrong Jan B. Pedersen - Protocol Verification in Millipede
Debugging Parallel Message Passing Programs CPA 2001 - BRISTOL 19 September, 2001 Debugging Parallel Message Passing Programs Millipede Solution: Multi Level Interactive Parallel Debugger Multi Level Parallel Debugging: Tools specifically tailored to finding/correcting errors at various levels of the program Jan B. Pedersen - Protocol Verification in Millipede
Millipede Millipede is a collection of tools, some are: CPA 2001 - BRISTOL 19 September, 2001 Millipede Millipede is a collection of tools, some are: Sequential Debugging Module Extracts a process and allow sequential debugging using any sequential debugger. [CIC’2000] Las Vegas 2000 CIC/PDPTA Deadlock Detection/Correction Module Locates deadlocks/makes suggestions to changes the source code to remove deadlock. [HIPS’2001] San Francisco 2001 HIPS/IPDPS Protocol Verification Module Checks a specification against messages sent. Jan B. Pedersen - Protocol Verification in Millipede
Protocol Verification in Millipede CPA 2001 - BRISTOL 19 September, 2001 Protocol Verification in Millipede Write a protocol specification Run the program Messages violating the protocol are reported Correct the errors or/and Refine the protocol specification Go back to step number 2 Jan B. Pedersen - Protocol Verification in Millipede
CPA 2001 - BRISTOL 19 September, 2001 Preliminaries A set of processes spawned from the same pvm_spawn() is called a group. An instance is one process from a group. A line number is the number of a line with a pvm_send() or a pvm_recv(). Jan B. Pedersen - Protocol Verification in Millipede
Protocol Specification CPA 2001 - BRISTOL 19 September, 2001 Protocol Specification A protocol specification consists of a number of lines of the form: pgname1[e1]{e2}(e3) pgname2[e4]{e5}(e6) Each line followed by 0 or more quantifiers: id : RelationalExpression Jan B. Pedersen - Protocol Verification in Millipede
Protocol Lines pgname1[e1]{e2}(e3) pgname2[e4]{e5}(e6) Sender CPA 2001 - BRISTOL 19 September, 2001 Protocol Lines pgname1[e1]{e2}(e3) pgname2[e4]{e5}(e6) Sender program name Receiver program name Jan B. Pedersen - Protocol Verification in Millipede
Protocol Lines Sender group no. Receiver group no. CPA 2001 - BRISTOL 19 September, 2001 Protocol Lines Sender group no. Receiver group no. pgname1[e1]{e2}(e3) pgname2[e4]{e5}(e6) Sender program name Receiver program name Jan B. Pedersen - Protocol Verification in Millipede
Protocol Lines Sender group no. Sender instance Receiver group no. CPA 2001 - BRISTOL 19 September, 2001 Protocol Lines Sender group no. Sender instance Receiver group no. Receiver instance pgname1[e1]{e2}(e3) pgname2[e4]{e5}(e6) Sender program name Receiver program name Jan B. Pedersen - Protocol Verification in Millipede
Protocol Lines Sender group no. Sender instance Receiver group no. CPA 2001 - BRISTOL 19 September, 2001 Protocol Lines Sender group no. Sender instance Receiver group no. Receiver instance pgname1[e1]{e2}(e3) pgname2[e4]{e5}(e6) Sender program name Sender line number Receiver program name Receiver line number Jan B. Pedersen - Protocol Verification in Millipede
Protocol Lines Sender Receiver CPA 2001 - BRISTOL 19 September, 2001 Protocol Lines Sender Receiver pgname1[e1]{e2}(e3) pgname2[e4]{e5}(e6) Can be: Omitted [] A constant [c] An identifier [i] Jan B. Pedersen - Protocol Verification in Millipede
Protocol Lines Sender Receiver CPA 2001 - BRISTOL 19 September, 2001 Protocol Lines Sender Receiver pgname1[e1]{e2}(e3) pgname2[e4]{e5}(e6) -transformation Can be: Omitted [] A constant [c] An identifier [i] ei:=i & Q=Qi: true ei:=i & Q=Qi: i=c ei:=i Jan B. Pedersen - Protocol Verification in Millipede
Protocol Lines pgname1[e1]{e2}(e3) pgname2[e4]{e5}(e6) Can be: CPA 2001 - BRISTOL 19 September, 2001 Protocol Lines pgname1[e1]{e2}(e3) pgname2[e4]{e5}(e6) Can be: Omitted [] Constant [c] Identifier [i] Expression [i+1] Jan B. Pedersen - Protocol Verification in Millipede
Protocol Lines pgname1[e1]{e2}(e3) pgname2[e4]{e5}(e6) Can be: CPA 2001 - BRISTOL 19 September, 2001 Protocol Lines pgname1[e1]{e2}(e3) pgname2[e4]{e5}(e6) Can be: Omitted [] Constant [c] Identifier [i] Expression [i+1] No -transformation - the evaluated expression is compared to the values of the actual message Jan B. Pedersen - Protocol Verification in Millipede
Quantifiers id : RelationalExpression CPA 2001 - BRISTOL 19 September, 2001 Quantifiers id : RelationalExpression Introduces a new variable to be used in e1…e6 (1,2 ,3) Example: n : (0 <≥ n) && (n <= 10) Jan B. Pedersen - Protocol Verification in Millipede
Messages A message in a message passing system: CPA 2001 - BRISTOL 19 September, 2001 Messages A message in a message passing system: M = (PS, PR, (GS, IS, LS), (GR, IR, LR), NS, NR) Program names The program name is the name of the source file that generated the executable. Example: PS = Master.c PR = Slave.c Jan B. Pedersen - Protocol Verification in Millipede
Messages A message in a message passing system: CPA 2001 - BRISTOL 19 September, 2001 Messages A message in a message passing system: M = (PS, PR, (GS, IS, LS), (GR, IR, LR), NS, NR) Program name Group number Each time a process creation takes place a new group is created. Example: GS = 0 GR = 3 Jan B. Pedersen - Protocol Verification in Millipede
Messages A message in a message passing system: CPA 2001 - BRISTOL 19 September, 2001 Messages A message in a message passing system: M = (PS, PR, (GS, IS, LS), (GR, IR, LR), NS, NR) Program name Group number Instance number Within each group of processes spawned together each process has an instance number Example: IS = 0 IR = 1 Jan B. Pedersen - Protocol Verification in Millipede
Messages A message in a message passing system: CPA 2001 - BRISTOL 19 September, 2001 Messages A message in a message passing system: M = (PS, PR, (GS, IS, LS), (GR, IR, LR), NS, NR) Program name Group number Instance number Line number LS is the line number of the send, LR is the line number of the receive. Example: LS = 72 LR = 83 Jan B. Pedersen - Protocol Verification in Millipede
Messages A message in a message passing system: CPA 2001 - BRISTOL 19 September, 2001 Messages A message in a message passing system: M = (PS, PR, (GS, IS, LS), (GR, IR, LR), NS, NR) Program name Group number Instance number Line number Total number of processes in groups NS is the number of processes in group number GS of program PS. Same holds for NR, GR & PR Example: NS = 1 NR = 10 Jan B. Pedersen - Protocol Verification in Millipede
Messages Example of a message in Millipede: CPA 2001 - BRISTOL 19 September, 2001 Messages Example of a message in Millipede: M = (Master.c, Slave.c, (0, 0, 72), (3, 1, 83), 1, 10) Master.c Slave.c 72: Send(……) 83:Receive(……) Group 0 Inst 0 Line 72 Group 3 Inst 1 Line 83 Jan B. Pedersen - Protocol Verification in Millipede
Semantics L = [e1]{e2}(e3) [e4]{e5}(e6) :: Q; CPA 2001 - BRISTOL 19 September, 2001 Semantics L = [e1]{e2}(e3) [e4]{e5}(e6) :: Q; M = (PS, PR, (GS, IS, LS), (GR, IR, LR), NS, NR) To check a message M against a line L: Apply the -transformation to e1 , e2 , e3 Check PS= and PR= Check -quantifiers of Q Check remaining quantifiers of Q Check E [e4]=Gr, E [e5]=Ir, E [e6]=Lr Where E [ ] is a semantic function. Jan B. Pedersen - Protocol Verification in Millipede
E [ ] & R [ ] E [number] = Number E [id] = (id) CPA 2001 - BRISTOL 19 September, 2001 E [ ] & R [ ] E [number] = Number E [id] = (id) E [e1*e2] = E [e1] * E [e2] … R [true] = true R [false] = false R [e1 < e2] = E [e1] < E[e2] Jan B. Pedersen - Protocol Verification in Millipede
E [ ] & R [ ] E [number] = Number E [id] = (id) Symbol table lookup CPA 2001 - BRISTOL 19 September, 2001 E [ ] & R [ ] E [number] = Number E [id] = (id) E [e1*e2] = E [e1] * E [e2] … R [true] = true R [false] = false R [e1 < e2] = E [e1] < E[e2] Symbol table lookup is a symbol table containing values from the message. Jan B. Pedersen - Protocol Verification in Millipede
Protocol Specification CPA 2001 - BRISTOL 19 September, 2001 Protocol Specification A protocol specification in Millipede specifies who may send to whom Level of refinement is variable: Can start out very general Can incrementally become more complicated Jan B. Pedersen - Protocol Verification in Millipede
[]{}() []{}(); Example 1 The smallest protocol possible: CPA 2001 - BRISTOL 19 September, 2001 Example 1 The smallest protocol possible: []{}() []{}(); Any process can send to any other process regardless of group, instance or line number. Jan B. Pedersen - Protocol Verification in Millipede
Example 1 []{}() []{}(); processes may communicate CPA 2001 - BRISTOL 19 September, 2001 Example 1 []{}() []{}(); processes may communicate with other process regardless of group number. Jan B. Pedersen - Protocol Verification in Millipede
Example 1 []{}() []{}(); Any instance may communicate CPA 2001 - BRISTOL 19 September, 2001 Example 1 []{}() []{}(); Any instance may communicate with any other instance. 1 2 3 Jan B. Pedersen - Protocol Verification in Millipede
Example 1 []{}() []{}(); Any send in any line can send to CPA 2001 - BRISTOL 19 September, 2001 Example 1 []{}() []{}(); Any send in any line can send to any receive in any line. send receive send receive 1 2 3 Jan B. Pedersen - Protocol Verification in Millipede
Example 1.5 This protocol an be specialized to only CPA 2001 - BRISTOL 19 September, 2001 Example 1.5 This protocol an be specialized to only allow ring communication; process number i sends to process number i+1 mod n. []{}() []{}() :: i: 0<= i <= n-1; Jan B. Pedersen - Protocol Verification in Millipede
Example 1.5 This protocol an be specialized to only CPA 2001 - BRISTOL 19 September, 2001 Example 1.5 This protocol an be specialized to only allow ring communication; process number i sends to process number i+1 mod n. []{i}() []{(i+1)%n}() :: i: 0<= i <= n-1; Jan B. Pedersen - Protocol Verification in Millipede
Example 2 One master process, n slave processes. Slave Master CPA 2001 - BRISTOL 19 September, 2001 Example 2 One master process, n slave processes. Slave Master Slaves communicate among themselves Jan B. Pedersen - Protocol Verification in Millipede
Example 2 Master: Send parameters Receive results Slave Master CPA 2001 - BRISTOL 19 September, 2001 Example 2 Master: Send parameters Receive results Slave Master Jan B. Pedersen - Protocol Verification in Millipede
Example 2 Slave: Receive parameters Loop n times { CPA 2001 - BRISTOL 19 September, 2001 Example 2 Slave: Receive parameters Loop n times { if id>0 send to id-1 if id < n-1 send to id+1 if id > 0 receive from id-1 if id < n-1 receive from id+1 Calculate } Send results Slave 9 Slave 8 Slave 0 Slave 1 Jan B. Pedersen - Protocol Verification in Millipede
Example 2 A general, very simple protocol could be: P1: CPA 2001 - BRISTOL 19 September, 2001 Example 2 A general, very simple protocol could be: P1: 1: Master[]{}() Slave[]{}(); 2: Slave[]{}() Master[]{}(); 3: Slave[]{}() Slave[]{}(); Any slave can send to any other slave Jan B. Pedersen - Protocol Verification in Millipede
Example 2 A general, very simple protocol could be: P1: CPA 2001 - BRISTOL 19 September, 2001 Example 2 A general, very simple protocol could be: P1: 1: Master[]{}() Slave[]{}(); 2: Slave[]{}() Master[]{}(); 3: Slave[]{}() Slave[]{}(); There is only one Master group with one instance, and only one Slave group. Jan B. Pedersen - Protocol Verification in Millipede
Example 2 We can add this information to the protocol. P1’: CPA 2001 - BRISTOL 19 September, 2001 Example 2 We can add this information to the protocol. P1’: 1: Master[0]{0}() Slave[0]{}(); 2: Slave[0]{}() Master[0]{0}(); 3: Slave[0]{}() Slave[0]{}(); There is only one Master group with one instance, and only one Slave group. Jan B. Pedersen - Protocol Verification in Millipede
Example 2 Adding information about the CPA 2001 - BRISTOL 19 September, 2001 Example 2 Adding information about the communication pattern of the slaves: a.) Slave number i can send to i+1 if i<n b.) Slave number i can send to i-1 if i>0 Slave 0 Slave 1 Slave 8 Slave 9 Jan B. Pedersen - Protocol Verification in Millipede
Example 2 P2: 1: Master[0]{0}() Slave[0]{}(); CPA 2001 - BRISTOL 19 September, 2001 Example 2 P2: 1: Master[0]{0}() Slave[0]{}(); 2: Slave[0]{}() Master[0]{0}(); 3a: Slave[0]{i}() Slave[0]{i+1}():: i: i<n-1; 3b: Slave[0]{i}() Slave[0]{i-1}() :: i: 0<i; Line 3a: slave sends to it’s ‘right’ neighbour Line 3b: slave sends to it’s ‘left’ neighbour Jan B. Pedersen - Protocol Verification in Millipede
Example 2 We know which sends can send to which receives: id = i+1 CPA 2001 - BRISTOL 19 September, 2001 Example 2 We know which sends can send to which receives: send to id-1 send to id+1 recv from id-1 recv from id+1 id = i+1 send to id-1 send to id+1 recv from id-1 recv from id+1 id = i Jan B. Pedersen - Protocol Verification in Millipede
Example 2 We can add line labels of the form: CPA 2001 - BRISTOL 19 September, 2001 Example 2 We can add line labels of the form: /* PROTOCOL(<id>) */ Slave: Receive parameters /* PROTOCOL(SR)*/ Loop n times { if id>0 send to id-1 /* PROTOCOL(S1) */ if id < n-1 send to id+1 /* PROTOCOL(S2) */ if id > 0 receive from id-1 /* PROTOCOL(R1) */ if id < n-1 receive from id+1 /* PROTOCOL(R2) */ Calculate } Send results /* PROTOCOL(SS) */ Jan B. Pedersen - Protocol Verification in Millipede
Example 2 By adding line number information we arrive CPA 2001 - BRISTOL 19 September, 2001 Example 2 By adding line number information we arrive at more restrictive version of the protocol: P3: 1: Master[0]{0}(MS) Slave[0]{}(SR); 2: Slave[0]{}(SS) Master[0]{0}(MR); 3a: Slave[0]{i}(S1) Slave[0]{i+1}(R1):: i: i<n-1; 3b: Slave[0]{i}(S2) Slave[0]{i-1}(R2):: i: 0<i; Jan B. Pedersen - Protocol Verification in Millipede
Example 2 The final version of the protocol is fully quantified: CPA 2001 - BRISTOL 19 September, 2001 Example 2 The final version of the protocol is fully quantified: 1: Master[0]{0}(MS) Slave[0]{i}(SR):: i: (0<=i) && (i<n); 2: Slave[0]{i}(SS) Master[0]{0}(MR):: i: (0<=i) && (i<n); 3a: Slave[0]{i}(S1) Slave[0]{i+1}(R1):: i: (0<=i) && (i<n-1); 3b: Slave[0]{i}(S2) Slave[0]{i-1}(R2):: i: (0<i) && (i<n); This is the final version of the master slave protocol. Jan B. Pedersen - Protocol Verification in Millipede
Protocol Checking There are 2 different modes: On-line: CPA 2001 - BRISTOL 19 September, 2001 Protocol Checking There are 2 different modes: On-line: Messages are checked when they are sent Millipede intercepts messages Off-line: Messages are read from log-file Log-files were written when program ran Jan B. Pedersen - Protocol Verification in Millipede
Protocol Prediction If a protocol specification is fully quantified CPA 2001 - BRISTOL 19 September, 2001 Protocol Prediction If a protocol specification is fully quantified a table can be computed showing all valid communications Master[0]{0}(MS) -> Slave[0]{0}(SR) Slave[0]{2}(SS) -> Master[0]{0}(MR) -> Slave[0]{1}(SR) Slave[0]{2}(S1) -> Slave[0]{3}(R1) -> Slave[0]{2}(SR) Slave[0]{2}(S2) -> Slave[0]{1}(R2) -> Slave[0]{3}(SR) Slave[0]{3}(SS) -> Master[0]{0}(MR) Slave[0]{0}(SS) -> Master[0]{0}(MR) Slave[0]{3}(S2) -> Slave[0]{2}(R2) Slave[0]{0}(S1) -> Slave[0]{1}(R1) Slave[0]{1}(SS) -> Master[0]{0}(MR) Slave[0]{1}(S1) -> Slave[0]{2}(R1) Slave[0]{1}(S2) -> Slave[0]{0}(R2) Prediction Table Jan B. Pedersen - Protocol Verification in Millipede
Implementation Runtime system intercepts messages CPA 2001 - BRISTOL 19 September, 2001 Implementation Runtime system intercepts messages Writes to log-files Passes to verification module Protocol specification read from file Parse tree built Expressions evaluated dynamically Symbol table based on message Jan B. Pedersen - Protocol Verification in Millipede
Future Work Make use of the message tags: Allow constructions like CPA 2001 - BRISTOL 19 September, 2001 Future Work Make use of the message tags: []{}()<tag> []{}()<tag> Allow constructions like e [v1,v2,…,vn] Passing state from the program protocol(x); pvm_send(…) Makes the value x available for the protocol specification Jan B. Pedersen - Protocol Verification in Millipede
Conclusion Easy to use protocol specification language CPA 2001 - BRISTOL 19 September, 2001 Conclusion Easy to use protocol specification language No complicated CSP stuff (not necessarily) Can be applied offline to crashed programs or online to running programs Jan B. Pedersen - Protocol Verification in Millipede
CPA 2001 - BRISTOL 19 September, 2001 The average person after debugging parallel message passing programs for a day WITHOUT Millipede Jan B. Pedersen - Protocol Verification in Millipede