Chien-Liang Fok liang@cse.wustl.edu Distribution Seminar Chien-Liang Fok liang@cse.wustl.edu

Topic One Clocks, Order, and Mutual Exclusion in a Distributed System Leslie Lamport, "Time, Clocks, and the Ordering of Events in a Distributed System", Communications of the ACM, 21(7), pp. 558-565, July 1978. January 18, 2019 CS 687 SP03 Chien-Liang Fok

What is a Distributed System? Process Process Process Process Process Message transmission delay is not negligible relative to time between events in a process. A collection of processes communicating through message passing. January 18, 2019 CS 687 SP03 Chien-Liang Fok

Representation of a Process A process is a sequence of events It may not be possible to tell which of two events in different processes occurred first. “Ex occurs before Ey” is only a partial order P1: E1 E2 E3 … En P2: E1 E2 E3 … En Pm: E1 E2 E3 … En Within a process, there is a total ordering of events. Events may include the sending and receiving of a message. January 18, 2019 CS 687 SP03 Chien-Liang Fok

The ab relation “ab” denotes “a occurs before b” We want to develop an algorithm that uses this relation to determine a total ordering of events in our system Real clocks are not easily synchronized Must define truth of ab w/o using physical clocks January 18, 2019 CS 687 SP03 Chien-Liang Fok

Truth of A  B P0: P1: P2: AB is true when: A occurs before B in the same process A and B in different processes; A sends message while B receives it AC and CB … A B P0: … A … … … P1: … … … B … P2: January 18, 2019 CS 687 SP03 Chien-Liang Fok

Concurrency Events a and b are concurrent iff: This implies that a cannot causally affect b and vice versa Note that for any event, e, (ee) is true  is an irreflexive partial ordering on the set of all events in the system (AB)  (BA) January 18, 2019 CS 687 SP03 Chien-Liang Fok

Concurrency (Continued) Given the space time diagram on right: P1P2, Q1Q3 P1Q1, Q3  P3 (P2Q2)  (Q2P2) P2 and Q2 are concurrent! P3 Q3 Q2 P2 Q1 P1 Time Process P Process Q Event Sent Message January 18, 2019 CS 687 SP03 Chien-Liang Fok

Logical Clocks Assign a number to each event such that if ab then C(a) < C(b) where C(x) denotes the number assigned to event x. This is known as the clock condition Note: the converse is not true Since the clock condition is defined in terms of , it is clearly true when A and B are in the same process and A occurs before B, or when A sends a message that B receives. January 18, 2019 CS 687 SP03 Chien-Liang Fok

Clock “tick” Occurs between events on a process Let a and b be two events such that c(a)=4 and c(b)=7 Ticks 5, 6, and 7 occur between a and b A tick line connects like-times between two or more processes Must be between any two events on a process Every message line must cross a tick line January 18, 2019 CS 687 SP03 Chien-Liang Fok

Tick Lines: Visually Rep. Process P P3 Process Q P2 P1 Q3 Q1 Q2 Time = Event Sent Message Tick Line January 18, 2019 CS 687 SP03 Chien-Liang Fok

Clock Implementation How to implement a clock that satisfies the clock condition?? Add a register to each process Increment it after each event Augment each message with a timestamp Tm When received message, increase the clock register’s value to be greater than both its current value and Tm January 18, 2019 CS 687 SP03 Chien-Liang Fok

Obtaining Total Order Using system of clocks, total ordering of all events in the system is easy Order events by the time they occur Break ties by using an arbitrary ordering of processes Let ab be true if c(a)<c(b) or c(a)=c(b) and a’s process is preferred over b’s The  relation is NOT unique! January 18, 2019 CS 687 SP03 Chien-Liang Fok

Using Total Order to Solve the Mutual Exclusion Problem Process Process Process Resource Only one process can be granted the resource at a time Processes must be granted the resource in the order they requested for it Every process that is granted the resource will eventually release it This is NOT a trivial problem because the order in which processes should be granted the resource is the order in which they REQUESTED, not the order in which the central manager received the request!!! Assumptions: FIFO, no message loss, everyone can send a message to everyone else January 18, 2019 CS 687 SP03 Chien-Liang Fok

Mutual Exclusion Algorithm (1/4) Implement the system of clocks Each process has a clock register Augment each process with a request queue rq Initialize all rq’s to contain message t0:p0 t0 is smaller than all clock registers p0 is the process initially granted the resource January 18, 2019 CS 687 SP03 Chien-Liang Fok

Mutual Exclusion Algorithm (2/4) Process pi Process Process Resource Acknowledge Request pi requests resource by sending tm:pi to every other process and putting tm:pi in rqi When pj receives tm:pi, add to rqj and send acknowledgement to pi January 18, 2019 CS 687 SP03 Chien-Liang Fok

Mutual Exclusion Algorithm (3/4) Process pi Process Process Resource Request pi releases resource by sending a request to everyone and removing tm:pi from rqi When pj receives the request, it removes tm:pi from rqj January 18, 2019 CS 687 SP03 Chien-Liang Fok

Mutual Exclusion Algorithm (4/4) pi is granted the resource when both: tm:pi  {all other requests in rqi} pi received a message from everyone else with time stamp greater than tm Note: this can be determined locally -- First condition ensures that resource is granted in the order it was requested -- Second condition ensures that pi has learned about all requests that preceded it -- The only process that can erase a request is the process that requested it, meaning only one process can access it at a time January 18, 2019 CS 687 SP03 Chien-Liang Fok

Vulnerabilities Two vulnerabilities: If just one process deadlocks, the whole system dies Anomalous behavior if system model does not match real-life model January 18, 2019 CS 687 SP03 Chien-Liang Fok

Anomalous Behavior User B User A Resource Hay!! Request it! User B User A Resource User b’s message may have a lower time stamp than a January 18, 2019 CS 687 SP03 Chien-Liang Fok

Addressing Anomalous Behavior Introduce strong clock condition, ab For any two events a and b in the system, if ab then c(a)<c(b) This is not supported by our clock system! January 18, 2019 CS 687 SP03 Chien-Liang Fok

Solution to Anomalous Behavior Use real clocks, let Ci(t) = reading of clock i at time t  = maximum clock error rate, # sec off/sec  = maximum error between clocks, sec  = maximum message transmission delay To prevent anomalous behavior, ensure: Ci(t + ) - Cj(t) > 0 For the above to be true, /(1-)   See end of paper for derivation/proof. January 18, 2019 CS 687 SP03 Chien-Liang Fok

Topic 2 Vector Clocks Friedemann Mattern, Virtual Time and Global States of Distributed Systems. Cosnard M. et al. (Eds): Proc. Workshop on Parallel and Distributed Algorithms, North-Holland / Elsevier, pp. 215-226, 1989 Reprinted in: Z. Yang, T.A. Marsland (Eds.), "Global States and Time in Distributed Systems", IEEE, 1994, pp. 123-133.) January 18, 2019 CS 687 SP03 Chien-Liang Fok

Motivation Previously, partially ordered events were mapped into a total order Two simultaneous events are treated as if one occurred first Useful information is lost Linear ordering of events is sometimes not good enough January 18, 2019 CS 687 SP03 Chien-Liang Fok

The Nature of Time Time has the following properties Transitivity Irreflexivity Linearity Eternity goes on forever Density can be divided into infinitesimally small units January 18, 2019 CS 687 SP03 Chien-Liang Fok

The BIG Question Can we develop a virtual clock system that preserves more of the system’s timing properties? Lamport’s algorithm does not preserve causal independence January 18, 2019 CS 687 SP03 Chien-Liang Fok

Vector time Process P Process Q Process R Suppose a deity can observe all clocks and store their values in a vector Process P Process Q Process R Time 1 1 1 1 2 1 2 1 3 2 2 3 January 18, 2019 CS 687 SP03 Chien-Liang Fok

Goal Create algorithm so each process gets an optimal approximation of this vector January 18, 2019 CS 687 SP03 Chien-Liang Fok

Technique Replace the register in each process with a vector. Size of vector = number of processes Update value in vector of own time like usual When receive message, update estimated time of other process based on current time timestamp vector within message January 18, 2019 CS 687 SP03 Chien-Liang Fok

Using our Technique Process P Process Q Process R 1 2 1 2 1 1 3 2 1 2 Process P 1 2 1 1 3 Process Q 2 1 Perfect local information, imperfect global info. Process R Time 1 1 1 1 2 1 2 1 3 2 2 3 Ideal January 18, 2019 CS 687 SP03 Chien-Liang Fok

Things to Note Each process has perfect local information Given two time vectors u, v u ≤ v iff i : u[ i ] ≤ v[ i ] u < v iff (u ≤ v)  (u ≠ v) u || v iff (u < v)  (v < u) u ≤ v and u < v are partial orders u || v is the concurrent relation It is NOT transitive!! u < v – all elements in v is at least as big as u, and at least one in v is larger than u u || v - at least one element in v is less than u, and at least one element in u is less than v Ex: of || not being transitive: u = (1 2 4), v = (1 6 3), w = (1 5 4) here u || v, v || w, but not u || w January 18, 2019 CS 687 SP03 Chien-Liang Fok

Cuts Process P Process Q Process R Time Consistent Cut Inconsistent Cut Event Sent Message Cut Event Cut Line A timing diagrams with consistent cuts can be modified so the cuts are straight lines. January 18, 2019 CS 687 SP03 Chien-Liang Fok

Time at a Cut Process P Process Q Process R Event Sent Message Cut Event Cut Line The time vector of a consistent cut is the time of each individual process at the cut January 18, 2019 CS 687 SP03 Chien-Liang Fok

Key Point For any set of events: {the lattice of consistent cuts} = {the lattice of possible time vectors} Thus, two events e and e’ are causally related iff TimeVector(e) < TimeVector(e’) January 18, 2019 CS 687 SP03 Chien-Liang Fok

Applications When do we care which event in a set of concurrent events occurred first?? Distributed debugging Must consider causal relationship between events in the system Performance analysis Vector clocks allows one to calculate potential concurrency January 18, 2019 CS 687 SP03 Chien-Liang Fok

Topic 3 Global Snapshot K. Mani Chandy and Leslie Lamport, "Distributed Snapshots: Determining Global States of Distributed Systems", ACM Transactions on Computer Systems, 3(1), pp. 63-75, February 1985. January 18, 2019 CS 687 SP03 Chien-Liang Fok

Motivation Knowing the global state of the system is useful January 18, 2019 CS 687 SP03 Chien-Liang Fok

Restriction Cannot pause the system Cannot prevent system from performing regular processing January 18, 2019 CS 687 SP03 Chien-Liang Fok

Problem Given the restrictions, it is impossible to obtain global state Cannot halt system System is constantly changing What can we do? January 18, 2019 CS 687 SP03 Chien-Liang Fok

Goal Find a possible global state Can determine value of stable system properties deadlock Termination By definition if a possible global state satisfies a stable property, the system will forever satisfy this property. January 18, 2019 CS 687 SP03 Chien-Liang Fok

The Model Process Process Process Process Process Channel Process Process Channels are FIFO and no message loss Processes send and receive messages through channels January 18, 2019 CS 687 SP03 Chien-Liang Fok

Global State Consists of: Process states Channel states What’s the state of the process? Channel states What messages are being sent through it? January 18, 2019 CS 687 SP03 Chien-Liang Fok

The Algorithm Sender process p: Receiver process q: Record state Send marker on all outgoing channels Receiver process q: If q has not recorded state record state record channel state as empty If q has recorded state record channel state as all messaged received after recording, but before receiving marker January 18, 2019 CS 687 SP03 Chien-Liang Fok

The Algorithm (2/2) Assuming strongly connected graph, all process will have recorded Process state Channel state for each incoming channel Consolidate these records to form global snapshot The paper proves this snapshot is a reachable from one that actually existed. Enough to prove stable properties of system January 18, 2019 CS 687 SP03 Chien-Liang Fok

Conclusions Virtual clocks are necessary since physical clocks are not easily synchronized Vector clocks improve upon Lamport’s virtual clock since it preserves causal independence Global snapshots of possible states is useful to determine stable properties of the system January 18, 2019 CS 687 SP03 Chien-Liang Fok