1. Breakpoints and Halting in Distributed Systems
Presented by
Abhishek Saxena
CS 739 Distributed Systems
Spring 2002

2. References
Detecting Relational Global Predicates in Distributed Systems by Alexander I. Tomlinson and Vijay K. Garg, 1993
Breakpoints and Halting in Distributed Programs by Barton P. Miller and Jong-Deok Choi, 1992
Restoring Consistent Global States of Distributed Computations by Goldberg et al., 1991

3. Presentation Layout Introduction Motivation
Halting in Distributed Systems
Detecting Breakpoints for:
Conjunctive/Disjunctive/Linked Predicates
Relational Predicates
Applications to Research
Relevance to papers read
Conclusions

4. Introduction General problems of: Halting distributed programs
Detecting breakpoints
Validating resource conflicts
Recording, restoration and replay of program sequences

5. Motivation Why halt? Interactive debugging
Issues in distributed systems:
No single global notion of time
Unpredictable communication delays
How to issue instant command to all processes?
Command to simultaneously reach all processes?

6. Halting 2 pertinent questions: How to halt a distributed program?
Halting Algorithm
When to halt?
Breakpoint Detection

7. Halting Algorithm Extends Chandy & Lamport’s algorithm Sending rule:
Increments last_halt_id
Send halt marker containing this value to outgoing channels
Receiving rule:
Compare the halt_id with its last_halt_id & update
Send halt marker like sender

8. The Halting Algorithm Sending process P Process R Halt marker
Process S
Process U

9. The Halting Algorithm
Intuitive extension to Chandy & Lamport’s Algorithm[1]
Leads to a global consistent state since:
Process states same as recorded process states in [1]
Undelivered messages same as recorded channels states in [1]

10. Problems with this Algorithm
Processes that infrequently interact with other computation processes
Long halting time
Acyclic network connection
Consumer
Producer P Q
Communication Channel

11. A Solution…
Centralized debugger process:
Debugger process d q p

12. Problems with this solution
Communication overheads
Possible change in execution of program
Complex to build

13. Detecting Breakpoints
Breakpoints & Predicates
Predicate satisfaction = breakpoint detection
Distributed processes’ system needs:
Simple predicates
Disjunctive predicates
Linked predicates…interesting!
Conjunctive predicates…very interesting!

14. Simple Predicates Encapsulate single process behavior
Detect simple events:
Entered procedure
Message sent / received
Channel created / destroyed
Process created / destroyed

15. Disjunctive predicates
Form:
DP ::= SP [ U SP ]*
Satisfied when any SP is satisfied
Initiate halting when DP is true

16. Linked Predicates Specify sequences of events Form:
LP ::= DP [ ->DP ]*
Debugger process sends the LP {DP1->...} to processes involved in DP1
Upon DP1, strip off DP1 & send stripped LP to processes involved in DP2

17. Linked predicates’ implementation
Process Q
Processes
involved in DP2
Processes involved in DP1
Process S
Debugger process
Process P
Start halting
DP2
DP2
DP1->DP2
DP1->DP2
DP1->DP2
Process T
Start halting
Start Halting
Process R

18. Conjunctive Predicates
Form:
CP ::= SP [ ∩ SP ]*
Hardest to detect!
No single time reference across machines
Interpretation based on virtual time:
Consider processes P1, P2 with virtual time axes T1, T2
Define
SCP = { (t1, t2) | t1ε T1, t2ε T2, SP(t1) ∩ SP(T2) }

19. Conjunctive predicates
Split SCP into:
Ordered-SCP:
{ (t1, t2) | (t1, t2)ε SCP, ((SP1) i -> (SP2) j) U ((SP2) i ->(SP1) j) }
Unordered-SCP:
{ (t1, t2) | (t1, t2)ε SCP, (t1, t2) € ordered-SCP }

20. Conjunctive Predicates
ordered-SCP pair
t12
t22
unordered- SCP pair
t23
t13

21. Conjunctive Predicates
Detecting unordered-SCP events difficult
Requires:
Global information gathering process
Time delay!
Cannot preserve meaningful process states

22. Detecting Relational Global Predicates
Resource conflict validation problems undetectable by earlier predicate classes
Form:
( x0 +…+ xn > C )
xi: resource usage at Pi
C: total resource available
Undecomposable into earlier classes of predicates

23. How to detect such predicates?
2 algorithms:
Decentralized: runs concurrently
Centralized: decoupled from the target program

24. Model & Notation
Partial ordering on S = { S0, …, Sn } where, Si <= Sj, for 0 <= i,j <= n
Happens-before relation: “->”
pred.u.i: Intuitively, is the state just preceding u in process i
succ.u.i: The state just succeeding u in process i

25. Concurrent States & IntervalsQ P 9 2
State Interval 10 3
Receive Interval 11 4
Deterministic event
Local state
Non-deterministic event

26. Concurrent Intervals 1, lo1 1, j 1, hi1 P1 0, lo0 P0 0, i 0, hi0 KEY
pred relation

27. Concurrent Intervals Intervals (0,i) & (1, j) concurrent iff
KEY exists in P0 or P1 s.t.,
lo0 < i <= hi0 & lo1 < j <= hi1,
where,
the lo0, lo1, hi0, hi1 as defined by the previous diagram

28. Overview of algorithms
Gather information
What?
How?
Consider 2 processes P0 & P1
Gather concurrent interval sequences:
{ lo0 to hi0 } at P0 & { lo1 to hi1 } at P1
Check resource violations at all possible pairs of states in these sequences!!

29. Algorithms contd… Representation of (0, lo0) (0, hi0)
as a 2x2 Matrix clock
Row i of Pi’s matrix clock = Pi’s vector clock
Current interval at Pk = (k, Mk[ , ])
Row k of Mk…pred() of current interval at Pk
Row i<>k…pred.pred() of current interval at Pk

30. Maintaining Matrix Clocks
Initialize
Initialize matrix to 0
If k=0 or k=1 Mk[k, k] ++
Send message tagged with Mk[., .] ; Increment Mk[k,k] for k=0 V 1
Upon message receive update matrix clock; Increment Mk[k,k] ;
Mk[k, ]= diagonal(Mk)

31. Matrix Clock Example
3 1
0 1
1 0
0 0
2 1
0 1 P0
2 1
0 1
0 0
0 1
0 0
0 1
0 0
0 2
2 1
2 3 P1

32. Decentralized Algorithm
Consider process P0
Upon mesg receive evaluate lo0, lo1, hi0, hi1
Find min value of resource(x) at P0
Send debug mesg (min_x0, lo1, hi1) to P1
P1 detects the predicate :
(min_x0 + min_x1 > C)

33. Overheads & Complexity at P0
Message overheads:
(# of receive intervals at P0)* sizeof ( 3 integers)………………..Debug mesgs
Sizeof(4 integers)…………Application mesgs
Memory:
# intervals at P0; min_x for each interval
Computation:
(# intervals at P0)*( # debug mesgs sent + received)

34. Centralized Algorithm
Checker process runs concurrently or, post-mortem
Consider the latter: processes P0 & P1
Processes keep trace files containing:
min_x for each interval
an array of {lo0, lo1, hi0, hi1} for each interval
Runs a check algorithm
Builds heaps by inserting the min_x values for all concurrent interval sequences at P0 & P1
Use these heap-tops to detect the predicate

35. Overheads & Complexity for P0
Memory:
4 integers for matrix clock each application process
Computation:
Monitor local variables
Rest offloaded to checker
O(R0 + M0logM0 + M1logM1)
Where, R0 & M0 = # rec intervals & total intervals at P0

36. Major Practical Problems
Reduced complexity from exp to O(nlogn) but still…
Large overheads even for 2 processes
Lots of messages!
Lots of memory space!
Lots of computation!

37. Applications to Research
Development of distributed debugging environment
Recording of execution sequences
Rollback
Replay
Exploration of new execution scenarios
Command of mission-control distributed systems

38. Relevance to Papers Read
The S/Net’s Linda kernel:
Debugging distributed tuple space
Detecting race conditions, deadlocks, probe effects
Chandy & Lamport’s paper explores the detection of stable predicates and Garg’s paper explores unstable predicate detection

39. Conclusions Distributed debugging still challenging
No efficient algorithm
Hard to do away with overheads
Need for efficient event monitoring & manipulation tools
Message sequence chart generators
Program flow analysis for more independent program splitting