1. Protocol Verification in Millipede
Jan Bækgaard Pedersen &
Alan Wagner
University of British Columbia
Vancouver, Canada

2. Verified - so what? Consider the following scenario:
CPA BRISTOL 19 September,
Verified - so what?
Consider the following scenario:
Write a protocol specification in some formal language like CSP/SMV/Mur
Verify it; if it checks out implement it in C/C++ and PVM/MPI
Jan B. Pedersen - Protocol Verification in Millipede

3. Verified - so what? Consider the following scenario:
CPA BRISTOL 19 September,
Verified - so what?
Consider the following scenario:
Write a protocol specification in some formal language like CSP/SMV/Mur
Verify it; if it checks out implement it in C/C++ and PVM/MPI
Problem: What if the implementation is
WRONG?
Jan B. Pedersen - Protocol Verification in Millipede

4. Don’t know how to! How about this scenario:
CPA BRISTOL 19 September,
Don’t know how to!
How about this scenario:
The C/PVM programmer does not know how to use CSP/SMV/Mur
What now?
Jan B. Pedersen - Protocol Verification in Millipede

5. Debugging Parallel Message Passing Programs
CPA BRISTOL 19 September,
Debugging Parallel Message Passing Programs
Errors can occur at different levels:
Errors in sequential code
Array out of bound, Pointer errors, Arithmetic under/over flow etc.
Errors in the contents of messages
Wrong contents, wrong length
Processes can deadlock
The communication protocol can be wrong
Jan B. Pedersen - Protocol Verification in Millipede

6. Debugging Parallel Message Passing Programs
CPA BRISTOL 19 September,
Debugging Parallel Message Passing Programs
Millipede
Solution:
Multi Level Interactive Parallel Debugger
Multi Level Parallel Debugging: Tools
specifically tailored to finding/correcting
errors at various levels of the program
Jan B. Pedersen - Protocol Verification in Millipede

7. Millipede Millipede is a collection of tools, some are:
CPA BRISTOL 19 September,
Millipede
Millipede is a collection of tools, some are:
Sequential Debugging Module
Extracts a process and allow sequential debugging using any sequential debugger.
[CIC’2000] Las Vegas 2000 CIC/PDPTA
Deadlock Detection/Correction Module
Locates deadlocks/makes suggestions to changes the source code to remove deadlock.
[HIPS’2001] San Francisco 2001 HIPS/IPDPS
Protocol Verification Module
Checks a specification against messages sent.
Jan B. Pedersen - Protocol Verification in Millipede

8. Protocol Verification in Millipede
CPA BRISTOL 19 September,
Protocol Verification in Millipede
Write a protocol specification
Run the program
Messages violating the protocol are reported
Correct the errors
or/and
Refine the protocol specification
Go back to step number 2
Jan B. Pedersen - Protocol Verification in Millipede

9. CPA 2001 - BRISTOL 19 September, 2001
Preliminaries
A set of processes spawned from the same pvm_spawn() is called a group.
An instance is one process from a group.
A line number is the number of a line with a pvm_send() or a pvm_recv().
Jan B. Pedersen - Protocol Verification in Millipede

10. Protocol Specification
CPA BRISTOL 19 September,
Protocol Specification
A protocol specification consists of a
number of lines of the form:
pgname1[e1]{e2}(e3)  pgname2[e4]{e5}(e6)
Each line followed by 0 or more quantifiers:
 id : RelationalExpression
Jan B. Pedersen - Protocol Verification in Millipede

11. Protocol Lines pgname1[e1]{e2}(e3)  pgname2[e4]{e5}(e6) Sender
CPA BRISTOL 19 September,
Protocol Lines
pgname1[e1]{e2}(e3)  pgname2[e4]{e5}(e6)
Sender
program
name
Receiver
program
name
Jan B. Pedersen - Protocol Verification in Millipede

12. Protocol Lines Sender group no. Receiver group no.
CPA BRISTOL 19 September,
Protocol Lines
Sender
group no.
Receiver
group no.
pgname1[e1]{e2}(e3)  pgname2[e4]{e5}(e6)
Sender
program
name
Receiver
program
name
Jan B. Pedersen - Protocol Verification in Millipede

13. Protocol Lines Sender group no. Sender instance Receiver group no.
CPA BRISTOL 19 September,
Protocol Lines
Sender
group no.
Sender
instance
Receiver
group no.
Receiver
instance
pgname1[e1]{e2}(e3)  pgname2[e4]{e5}(e6)
Sender
program
name
Receiver
program
name
Jan B. Pedersen - Protocol Verification in Millipede

14. Protocol Lines Sender group no. Sender instance Receiver group no.
CPA BRISTOL 19 September,
Protocol Lines
Sender
group no.
Sender
instance
Receiver
group no.
Receiver
instance
pgname1[e1]{e2}(e3)  pgname2[e4]{e5}(e6)
Sender
program
name
Sender
line
number
Receiver
program
name
Receiver
line
number
Jan B. Pedersen - Protocol Verification in Millipede

15. Protocol Lines Sender Receiver
CPA BRISTOL 19 September,
Protocol Lines
Sender
Receiver
pgname1[e1]{e2}(e3)  pgname2[e4]{e5}(e6)
Can be:
Omitted []
A constant [c]
An identifier [i]
Jan B. Pedersen - Protocol Verification in Millipede

16. Protocol Lines Sender Receiver
CPA BRISTOL 19 September,
Protocol Lines
Sender
Receiver
pgname1[e1]{e2}(e3)  pgname2[e4]{e5}(e6)
-transformation
Can be:
Omitted []
A constant [c]
An identifier [i] 
ei:=i & Q=Qi: true
ei:=i & Q=Qi: i=c
ei:=i
Jan B. Pedersen - Protocol Verification in Millipede

17. Protocol Lines pgname1[e1]{e2}(e3)  pgname2[e4]{e5}(e6) Can be:
CPA BRISTOL 19 September,
Protocol Lines
pgname1[e1]{e2}(e3)  pgname2[e4]{e5}(e6)
Can be:
Omitted []
Constant [c]
Identifier [i]
Expression [i+1]
Jan B. Pedersen - Protocol Verification in Millipede

18. Protocol Lines pgname1[e1]{e2}(e3)  pgname2[e4]{e5}(e6) Can be:
CPA BRISTOL 19 September,
Protocol Lines
pgname1[e1]{e2}(e3)  pgname2[e4]{e5}(e6)
Can be:
Omitted []
Constant [c]
Identifier [i]
Expression [i+1]
No -transformation -
the evaluated expression
is compared to the values
of the actual message
Jan B. Pedersen - Protocol Verification in Millipede

19. Quantifiers  id : RelationalExpression
CPA BRISTOL 19 September,
Quantifiers
 id : RelationalExpression
Introduces a new variable to be used
in e1…e6 (1,2 ,3)
Example:
 n : (0 <≥ n) && (n <= 10)
Jan B. Pedersen - Protocol Verification in Millipede

20. Messages A message in a message passing system:
CPA BRISTOL 19 September,
Messages
A message in a message passing system:
M = (PS, PR, (GS, IS, LS), (GR, IR, LR), NS, NR)
Program
names
The program name is the name of the source file that generated the executable.
Example: PS = Master.c
PR = Slave.c
Jan B. Pedersen - Protocol Verification in Millipede

21. Messages A message in a message passing system:
CPA BRISTOL 19 September,
Messages
A message in a message passing system:
M = (PS, PR, (GS, IS, LS), (GR, IR, LR), NS, NR)
Program
name
Group
number
Each time a process creation
takes place a new group is
created.
Example: GS = 0
GR = 3
Jan B. Pedersen - Protocol Verification in Millipede

22. Messages A message in a message passing system:
CPA BRISTOL 19 September,
Messages
A message in a message passing system:
M = (PS, PR, (GS, IS, LS), (GR, IR, LR), NS, NR)
Program
name
Group
number
Instance
number
Within each group of processes
spawned together each process
has an instance number
Example: IS = 0
IR = 1
Jan B. Pedersen - Protocol Verification in Millipede

23. Messages A message in a message passing system:
CPA BRISTOL 19 September,
Messages
A message in a message passing system:
M = (PS, PR, (GS, IS, LS), (GR, IR, LR), NS, NR)
Program
name
Group
number
Instance
number
Line
number
LS is the line number of the
send, LR is the line number
of the receive.
Example: LS = 72
LR = 83
Jan B. Pedersen - Protocol Verification in Millipede

24. Messages A message in a message passing system:
CPA BRISTOL 19 September,
Messages
A message in a message passing system:
M = (PS, PR, (GS, IS, LS), (GR, IR, LR), NS, NR)
Program
name
Group
number
Instance
number
Line
number
Total number
of processes
in groups
NS is the number of processes
in group number GS of program
PS. Same holds for NR, GR & PR
Example: NS = 1
NR = 10
Jan B. Pedersen - Protocol Verification in Millipede

25. Messages Example of a message in Millipede:
CPA BRISTOL 19 September,
Messages
Example of a message in Millipede:
M = (Master.c, Slave.c, (0, 0, 72), (3, 1, 83), 1, 10)
Master.c
Slave.c
72: Send(……)
83:Receive(……)
Group 0
Inst 0
Line 72
Group 3
Inst 1
Line 83
Jan B. Pedersen - Protocol Verification in Millipede

26. Semantics L = [e1]{e2}(e3)  [e4]{e5}(e6) :: Q;
CPA BRISTOL 19 September,
Semantics
L = [e1]{e2}(e3)  [e4]{e5}(e6) :: Q;
M = (PS, PR, (GS, IS, LS), (GR, IR, LR), NS, NR)
To check a message M against a line L:
Apply the -transformation to e1 , e2 , e3
Check PS= and PR=
Check -quantifiers of Q
Check remaining quantifiers of Q
Check E [e4]=Gr, E [e5]=Ir, E [e6]=Lr Where E [ ] is a semantic function.
Jan B. Pedersen - Protocol Verification in Millipede

27. E [ ] & R [ ] E [number] = Number E [id] = (id)
CPA BRISTOL 19 September,
E [ ] & R [ ]
E [number] = Number
E [id] = (id)
E [e1*e2] = E [e1] * E [e2] …
R [true] = true
R [false] = false
R [e1 < e2] = E [e1] < E[e2]
Jan B. Pedersen - Protocol Verification in Millipede

28. E [ ] & R [ ] E [number] = Number E [id] = (id) Symbol table lookup
CPA BRISTOL 19 September,
E [ ] & R [ ]
E [number] = Number
E [id] = (id)
E [e1*e2] = E [e1] * E [e2] …
R [true] = true
R [false] = false
R [e1 < e2] = E [e1] < E[e2]
Symbol table lookup
 is a symbol table
containing values from
the message.
Jan B. Pedersen - Protocol Verification in Millipede

29. Protocol Specification
CPA BRISTOL 19 September,
Protocol Specification
A protocol specification in Millipede specifies who may send to whom
Level of refinement is variable:
Can start out very general
Can incrementally become more complicated
Jan B. Pedersen - Protocol Verification in Millipede

30. []{}()  []{}(); Example 1 The smallest protocol possible:
CPA BRISTOL 19 September,
Example 1
The smallest protocol possible:
[]{}()  []{}();
Any  process can send to any other 
process regardless of group, instance
or line number.
Jan B. Pedersen - Protocol Verification in Millipede

31. Example 1 []{}()  []{}();   processes may communicate
CPA BRISTOL 19 September,
Example 1
[]{}()  []{}(); 
 processes may communicate
with other  process regardless
of group number.
Jan B. Pedersen - Protocol Verification in Millipede

32. Example 1 []{}()  []{}(); Any instance may communicate
CPA BRISTOL 19 September,
Example 1
[]{}()  []{}();
Any instance
may communicate
with any other
instance.  1 2 3
Jan B. Pedersen - Protocol Verification in Millipede

33. Example 1 []{}()  []{}(); Any send in any line can send to
CPA BRISTOL 19 September,
Example 1
[]{}()  []{}();
Any send in any
line can send to
any receive in
any line.
send
receive
send
receive  1 2 3
Jan B. Pedersen - Protocol Verification in Millipede

34. Example 1.5 This protocol an be specialized to only
CPA BRISTOL 19 September,
Example 1.5
This protocol an be specialized to only
allow ring communication; process number
i sends to process number i+1 mod n.
[]{}()  []{}() ::  i: 0<= i <= n-1;
Jan B. Pedersen - Protocol Verification in Millipede

35. Example 1.5 This protocol an be specialized to only
CPA BRISTOL 19 September,
Example 1.5
This protocol an be specialized to only
allow ring communication; process number
i sends to process number i+1 mod n.
[]{i}()  []{(i+1)%n}() ::  i: 0<= i <= n-1;
Jan B. Pedersen - Protocol Verification in Millipede

36. Example 2 One master process, n slave processes. Slave Master
CPA BRISTOL 19 September,
Example 2
One master process, n slave processes.
Slave
Master
Slaves communicate
among themselves
Jan B. Pedersen - Protocol Verification in Millipede

37. Example 2 Master: Send parameters Receive results Slave Master
CPA BRISTOL 19 September,
Example 2
Master:
Send parameters
Receive results
Slave
Master
Jan B. Pedersen - Protocol Verification in Millipede

38. Example 2 Slave: Receive parameters Loop n times {
CPA BRISTOL 19 September,
Example 2
Slave:
Receive parameters
Loop n times {
if id>0 send to id-1
if id < n-1 send to id+1
if id > 0 receive from id-1
if id < n-1 receive from id+1
Calculate }
Send results
Slave 9
Slave 8
Slave 0
Slave 1
Jan B. Pedersen - Protocol Verification in Millipede

39. Example 2 A general, very simple protocol could be: P1:
CPA BRISTOL 19 September,
Example 2
A general, very simple protocol could be:
P1:
1: Master[]{}()  Slave[]{}();
2: Slave[]{}()  Master[]{}();
3: Slave[]{}()  Slave[]{}();
Any slave can send to any other slave
Jan B. Pedersen - Protocol Verification in Millipede

40. Example 2 A general, very simple protocol could be: P1:
CPA BRISTOL 19 September,
Example 2
A general, very simple protocol could be:
P1:
1: Master[]{}()  Slave[]{}();
2: Slave[]{}()  Master[]{}();
3: Slave[]{}()  Slave[]{}();
There is only one Master group with one
instance, and only one Slave group.
Jan B. Pedersen - Protocol Verification in Millipede

41. Example 2 We can add this information to the protocol. P1’:
CPA BRISTOL 19 September,
Example 2
We can add this information to the protocol.
P1’:
1: Master[0]{0}()  Slave[0]{}();
2: Slave[0]{}()  Master[0]{0}();
3: Slave[0]{}()  Slave[0]{}();
There is only one Master group with one
instance, and only one Slave group.
Jan B. Pedersen - Protocol Verification in Millipede

42. Example 2 Adding information about the
CPA BRISTOL 19 September,
Example 2
Adding information about the
communication pattern of the slaves:
a.) Slave number i can send to i+1 if i<n
b.) Slave number i can send to i-1 if i>0
Slave 0
Slave 1
Slave 8
Slave 9
Jan B. Pedersen - Protocol Verification in Millipede

43. Example 2 P2: 1: Master[0]{0}()  Slave[0]{}();
CPA BRISTOL 19 September,
Example 2
P2:
1: Master[0]{0}()  Slave[0]{}();
2: Slave[0]{}()  Master[0]{0}();
3a: Slave[0]{i}()  Slave[0]{i+1}():: i: i<n-1;
3b: Slave[0]{i}()  Slave[0]{i-1}() :: i: 0<i;
Line 3a: slave sends to it’s ‘right’ neighbour
Line 3b: slave sends to it’s ‘left’ neighbour
Jan B. Pedersen - Protocol Verification in Millipede

44.  Example 2 We know which sends can send to which receives: id = i+1
CPA BRISTOL 19 September,
Example 2
We know which sends can send to which receives:
send to id-1
send to id+1
recv from id-1
recv from id+1 
id = i+1
send to id-1
send to id+1
recv from id-1
recv from id+1
id = i
Jan B. Pedersen - Protocol Verification in Millipede

45. Example 2 We can add line labels of the form:
CPA BRISTOL 19 September,
Example 2
We can add line labels of the form:
/* PROTOCOL(<id>) */
Slave:
Receive parameters /* PROTOCOL(SR)*/
Loop n times {
if id>0 send to id /* PROTOCOL(S1) */
if id < n-1 send to id /* PROTOCOL(S2) */
if id > 0 receive from id /* PROTOCOL(R1) */
if id < n-1 receive from id+1 /* PROTOCOL(R2) */
Calculate }
Send results /* PROTOCOL(SS) */
Jan B. Pedersen - Protocol Verification in Millipede

46. Example 2 By adding line number information we arrive
CPA BRISTOL 19 September,
Example 2
By adding line number information we arrive
at more restrictive version of the protocol:
P3:
1: Master[0]{0}(MS)  Slave[0]{}(SR);
2: Slave[0]{}(SS)  Master[0]{0}(MR);
3a: Slave[0]{i}(S1)  Slave[0]{i+1}(R1):: i: i<n-1;
3b: Slave[0]{i}(S2)  Slave[0]{i-1}(R2):: i: 0<i;
Jan B. Pedersen - Protocol Verification in Millipede

47. Example 2 The final version of the protocol is fully quantified:
CPA BRISTOL 19 September,
Example 2
The final version of the protocol is fully
quantified:
1: Master[0]{0}(MS)  Slave[0]{i}(SR)::  i: (0<=i) && (i<n);
2: Slave[0]{i}(SS)  Master[0]{0}(MR)::  i: (0<=i) && (i<n);
3a: Slave[0]{i}(S1)  Slave[0]{i+1}(R1)::  i: (0<=i) && (i<n-1);
3b: Slave[0]{i}(S2)  Slave[0]{i-1}(R2)::  i: (0<i) && (i<n);
This is the final version of the master
slave protocol.
Jan B. Pedersen - Protocol Verification in Millipede

48. Protocol Checking There are 2 different modes: On-line:
CPA BRISTOL 19 September,
Protocol Checking
There are 2 different modes:
On-line:
Messages are checked when they are sent
Millipede intercepts messages
Off-line:
Messages are read from log-file
Log-files were written when program ran
Jan B. Pedersen - Protocol Verification in Millipede

49. Protocol Prediction If a protocol specification is fully quantified
CPA BRISTOL 19 September,
Protocol Prediction
If a protocol specification is fully quantified
a table can be computed showing all valid
communications
Master[0]{0}(MS) -> Slave[0]{0}(SR) Slave[0]{2}(SS) -> Master[0]{0}(MR)
-> Slave[0]{1}(SR) Slave[0]{2}(S1) -> Slave[0]{3}(R1)
-> Slave[0]{2}(SR) Slave[0]{2}(S2) -> Slave[0]{1}(R2)
-> Slave[0]{3}(SR)
Slave[0]{3}(SS) -> Master[0]{0}(MR)
Slave[0]{0}(SS) -> Master[0]{0}(MR) Slave[0]{3}(S2) -> Slave[0]{2}(R2)
Slave[0]{0}(S1) -> Slave[0]{1}(R1)
Slave[0]{1}(SS) -> Master[0]{0}(MR)
Slave[0]{1}(S1) -> Slave[0]{2}(R1)
Slave[0]{1}(S2) -> Slave[0]{0}(R2)
Prediction Table
Jan B. Pedersen - Protocol Verification in Millipede

50. Implementation Runtime system intercepts messages
CPA BRISTOL 19 September,
Implementation
Runtime system intercepts messages
Writes to log-files
Passes to verification module
Protocol specification read from file
Parse tree built
Expressions evaluated dynamically
Symbol table based on message
Jan B. Pedersen - Protocol Verification in Millipede

51. Future Work Make use of the message tags: Allow constructions like
CPA BRISTOL 19 September,
Future Work
Make use of the message tags:
[]{}()<tag>  []{}()<tag>
Allow constructions like
e [v1,v2,…,vn]
Passing state from the program
protocol(x);
pvm_send(…)
Makes the value x available for the protocol specification
Jan B. Pedersen - Protocol Verification in Millipede

52. Conclusion Easy to use protocol specification language
CPA BRISTOL 19 September,
Conclusion
Easy to use protocol specification language
No complicated CSP stuff (not necessarily)
Can be applied offline to crashed programs or online to running programs
Jan B. Pedersen - Protocol Verification in Millipede

53. CPA 2001 - BRISTOL 19 September, 2001
The average person after debugging parallel message passing programs for a day WITHOUT Millipede
Jan B. Pedersen - Protocol Verification in Millipede