1. Chien-Liang Fok liang@cse.wustl.edu
Distribution Seminar
Chien-Liang Fok

2. Topic One Clocks, Order, and Mutual Exclusion in a Distributed System
Leslie Lamport, "Time, Clocks, and the Ordering of Events in a Distributed System", Communications of the ACM, 21(7), pp , July 1978.
January 18, 2019
CS 687 SP03 Chien-Liang Fok

3. What is a Distributed System?
Process
Process
Process
Process
Process
Message transmission delay is not negligible relative to time between events in a process.
A collection of processes communicating through message passing.
January 18, 2019
CS 687 SP03 Chien-Liang Fok

4. Representation of a Process
A process is a sequence of events
It may not be possible to tell which of two events in different processes occurred first.
“Ex occurs before Ey” is only a partial order
P1: E1 E2 E3 … En
P2: E1 E2 E3 … En
Pm: E1 E2 E3 … En
Within a process, there is a total ordering of events.
Events may include the sending and receiving of a message.
January 18, 2019
CS 687 SP03 Chien-Liang Fok

5. The ab relation “ab” denotes “a occurs before b”
We want to develop an algorithm that uses this relation to determine a total ordering of events in our system
Real clocks are not easily synchronized
Must define truth of ab w/o using physical clocks
January 18, 2019
CS 687 SP03 Chien-Liang Fok

6. Truth of A  B P0: P1: P2: AB is true when:
A occurs before B in the same process
A and B in different processes; A sends message while B receives it
AC and CB … A B
P0: … A … … …
P1: … … … B …
P2:
January 18, 2019
CS 687 SP03 Chien-Liang Fok

7. Concurrency Events a and b are concurrent iff:
This implies that a cannot causally affect b and vice versa
Note that for any event, e, (ee) is true
 is an irreflexive partial ordering on the set of all events in the system
(AB)  (BA)
January 18, 2019
CS 687 SP03 Chien-Liang Fok

8. Concurrency (Continued)
Given the space time diagram on right:
P1P2, Q1Q3
P1Q1, Q3  P3
(P2Q2)  (Q2P2)
P2 and Q2 are concurrent! P3 Q3 Q2 P2 Q1 P1
Time
Process P
Process Q
Event
Sent Message
January 18, 2019
CS 687 SP03 Chien-Liang Fok

9. Logical Clocks
Assign a number to each event such that if ab then C(a) < C(b) where C(x) denotes the number assigned to event x.
This is known as the clock condition
Note: the converse is not true
Since the clock condition is defined in terms of , it is clearly true when A and B are in the same process and A occurs before B, or when A sends a message that B receives.
January 18, 2019
CS 687 SP03 Chien-Liang Fok

10. Clock “tick” Occurs between events on a process
Let a and b be two events such that c(a)=4 and c(b)=7
Ticks 5, 6, and 7 occur between a and b
A tick line connects like-times between two or more processes
Must be between any two events on a process
Every message line must cross a tick line
January 18, 2019
CS 687 SP03 Chien-Liang Fok

11. Tick Lines: Visually Rep.
Process P P3
Process Q P2 P1 Q3 Q1 Q2
Time =
Event
Sent Message
Tick Line
January 18, 2019
CS 687 SP03 Chien-Liang Fok

12. Clock Implementation
How to implement a clock that satisfies the clock condition??
Add a register to each process
Increment it after each event
Augment each message with a timestamp Tm
When received message, increase the clock register’s value to be greater than both its current value and Tm
January 18, 2019
CS 687 SP03 Chien-Liang Fok

13. Obtaining Total Order
Using system of clocks, total ordering of all events in the system is easy
Order events by the time they occur
Break ties by using an arbitrary ordering of processes
Let ab be true if c(a)<c(b) or c(a)=c(b) and a’s process is preferred over b’s
The  relation is NOT unique!
January 18, 2019
CS 687 SP03 Chien-Liang Fok

14. Using Total Order to Solve the Mutual Exclusion Problem
Process
Process
Process
Resource
Only one process can be granted the resource at a time
Processes must be granted the resource in the order they requested for it
Every process that is granted the resource will eventually release it
This is NOT a trivial problem because the order in which processes should be granted the resource is the order in which they REQUESTED, not the order in which the central manager received the request!!!
Assumptions: FIFO, no message loss, everyone can send a message to everyone else
January 18, 2019
CS 687 SP03 Chien-Liang Fok

15. Mutual Exclusion Algorithm (1/4)
Implement the system of clocks
Each process has a clock register
Augment each process with a request queue rq
Initialize all rq’s to contain message t0:p0
t0 is smaller than all clock registers
p0 is the process initially granted the resource
January 18, 2019
CS 687 SP03 Chien-Liang Fok

16. Mutual Exclusion Algorithm (2/4)
Process pi
Process
Process
Resource
Acknowledge
Request
pi requests resource by sending tm:pi to every other process and putting tm:pi in rqi
When pj receives tm:pi, add to rqj and send acknowledgement to pi
January 18, 2019
CS 687 SP03 Chien-Liang Fok

17. Mutual Exclusion Algorithm (3/4)
Process pi
Process
Process
Resource
Request
pi releases resource by sending a request to everyone and removing tm:pi from rqi
When pj receives the request, it removes tm:pi from rqj
January 18, 2019
CS 687 SP03 Chien-Liang Fok

18. Mutual Exclusion Algorithm (4/4)
pi is granted the resource when both:
tm:pi  {all other requests in rqi}
pi received a message from everyone else with time stamp greater than tm
Note: this can be determined locally
-- First condition ensures that resource is granted in the order it was requested
-- Second condition ensures that pi has learned about all requests that preceded it
-- The only process that can erase a request is the process that requested it, meaning only one process can access it at a time
January 18, 2019
CS 687 SP03 Chien-Liang Fok

19. Vulnerabilities Two vulnerabilities:
If just one process deadlocks, the whole system dies
Anomalous behavior if system model does not match real-life model
January 18, 2019
CS 687 SP03 Chien-Liang Fok

20. Anomalous Behavior User B User A Resource
Hay!!
Request it!
User B
User A
Resource
User b’s message may have a lower time stamp than a
January 18, 2019
CS 687 SP03 Chien-Liang Fok

21. Addressing Anomalous Behavior
Introduce strong clock condition, ab
For any two events a and b in the system, if ab then c(a)<c(b)
This is not supported by our clock system!
January 18, 2019
CS 687 SP03 Chien-Liang Fok

22. Solution to Anomalous Behavior
Use real clocks, let
Ci(t) = reading of clock i at time t
 = maximum clock error rate, # sec off/sec
 = maximum error between clocks, sec
 = maximum message transmission delay
To prevent anomalous behavior, ensure:
Ci(t + ) - Cj(t) > 0
For the above to be true,
/(1-)  
See end of paper for derivation/proof.
January 18, 2019
CS 687 SP03 Chien-Liang Fok

23. Topic 2
Vector Clocks
Friedemann Mattern, Virtual Time and Global States of Distributed Systems.
Cosnard M. et al. (Eds): Proc. Workshop on Parallel and Distributed Algorithms, North-Holland / Elsevier, pp , 1989
Reprinted in: Z. Yang, T.A. Marsland (Eds.), "Global States and Time in Distributed Systems", IEEE, 1994, pp )
January 18, 2019
CS 687 SP03 Chien-Liang Fok

24. Motivation
Previously, partially ordered events were mapped into a total order
Two simultaneous events are treated as if one occurred first
Useful information is lost
Linear ordering of events is sometimes not good enough
January 18, 2019
CS 687 SP03 Chien-Liang Fok

25. The Nature of Time Time has the following properties Transitivity
Irreflexivity
Linearity
Eternity
goes on forever
Density
can be divided into infinitesimally small units
January 18, 2019
CS 687 SP03 Chien-Liang Fok

26. The BIG Question
Can we develop a virtual clock system that preserves more of the system’s timing properties?
Lamport’s algorithm does not preserve causal independence
January 18, 2019
CS 687 SP03 Chien-Liang Fok

27. Vector time Process P Process Q Process R
Suppose a deity can observe all clocks and store their values in a vector
Process P
Process Q
Process R
Time 1 1 1 1 2 1 2 1 3 2 2 3
January 18, 2019
CS 687 SP03 Chien-Liang Fok

28. Goal
Create algorithm so each process gets an optimal approximation of this vector
January 18, 2019
CS 687 SP03 Chien-Liang Fok

29. Technique Replace the register in each process with a vector.
Size of vector = number of processes
Update value in vector of own time like usual
When receive message, update estimated time of other process based on
current time
timestamp vector within message
January 18, 2019
CS 687 SP03 Chien-Liang Fok

30. Using our Technique Process P Process Q Process R 1 2 1 2 1 1 3 2 12
Process P 1 2 1 1 3
Process Q 2 1
Perfect local information, imperfect global info.
Process R
Time 1 1 1 1 2 1 2 1 3 2 2 3
Ideal
January 18, 2019
CS 687 SP03 Chien-Liang Fok

31. Things to Note Each process has perfect local information
Given two time vectors u, v
u ≤ v iff i : u[ i ] ≤ v[ i ]
u < v iff (u ≤ v)  (u ≠ v)
u || v iff (u < v)  (v < u)
u ≤ v and u < v are partial orders
u || v is the concurrent relation
It is NOT transitive!!
u < v – all elements in v is at least as big as u, and at least one in v is larger than u
u || v - at least one element in v is less than u, and at least one element in u is less than v
Ex: of || not being transitive: u = (1 2 4), v = (1 6 3), w = (1 5 4)
here u || v, v || w, but not u || w
January 18, 2019
CS 687 SP03 Chien-Liang Fok

32. Cuts Process P Process Q Process R
Time
Consistent Cut
Inconsistent Cut
Event
Sent Message
Cut Event
Cut Line
A timing diagrams with consistent cuts can be modified so the cuts are straight lines.
January 18, 2019
CS 687 SP03 Chien-Liang Fok

33. Time at a Cut Process P Process Q Process R
Event
Sent Message
Cut Event
Cut Line
The time vector of a consistent cut is the time of each individual process at the cut
January 18, 2019
CS 687 SP03 Chien-Liang Fok

34. Key Point For any set of events: {the lattice of consistent cuts} =
{the lattice of possible time vectors}
Thus, two events e and e’ are causally related iff TimeVector(e) < TimeVector(e’)
January 18, 2019
CS 687 SP03 Chien-Liang Fok

35. Applications
When do we care which event in a set of concurrent events occurred first??
Distributed debugging
Must consider causal relationship between events in the system
Performance analysis
Vector clocks allows one to calculate potential concurrency
January 18, 2019
CS 687 SP03 Chien-Liang Fok

36. Topic 3
Global Snapshot
K. Mani Chandy and Leslie Lamport, "Distributed Snapshots: Determining Global States of Distributed Systems", ACM Transactions on Computer Systems, 3(1), pp , February 1985.
January 18, 2019
CS 687 SP03 Chien-Liang Fok

37. Motivation Knowing the global state of the system is useful
January 18, 2019
CS 687 SP03 Chien-Liang Fok

38. Restriction Cannot pause the system
Cannot prevent system from performing regular processing
January 18, 2019
CS 687 SP03 Chien-Liang Fok

39. Problem
Given the restrictions, it is impossible to obtain global state
Cannot halt system
System is constantly changing
What can we do?
January 18, 2019
CS 687 SP03 Chien-Liang Fok

40. Goal Find a possible global state
Can determine value of stable system properties
deadlock
Termination
By definition if a possible global state satisfies a stable property, the system will forever satisfy this property.
January 18, 2019
CS 687 SP03 Chien-Liang Fok

41. The Model Process Process Process Process Process
Channel
Process
Process
Channels are FIFO and no message loss
Processes send and receive messages through channels
January 18, 2019
CS 687 SP03 Chien-Liang Fok

42. Global State Consists of: Process states Channel states
What’s the state of the process?
Channel states
What messages are being sent through it?
January 18, 2019
CS 687 SP03 Chien-Liang Fok

43. The Algorithm Sender process p: Receiver process q: Record state
Send marker on all outgoing channels
Receiver process q:
If q has not recorded state
record state
record channel state as empty
If q has recorded state
record channel state as all messaged received after recording, but before receiving marker
January 18, 2019
CS 687 SP03 Chien-Liang Fok

44. The Algorithm (2/2)
Assuming strongly connected graph, all process will have recorded
Process state
Channel state for each incoming channel
Consolidate these records to form global snapshot
The paper proves this snapshot is a reachable from one that actually existed.
Enough to prove stable properties of system
January 18, 2019
CS 687 SP03 Chien-Liang Fok

45. Conclusions
Virtual clocks are necessary since physical clocks are not easily synchronized
Vector clocks improve upon Lamport’s virtual clock since it preserves causal independence
Global snapshots of possible states is useful to determine stable properties of the system
January 18, 2019
CS 687 SP03 Chien-Liang Fok