Lecture 12 Synchronization

EECE 411: Design of Distributed Software Applications Summary so far … A distributed system is: a collection of independent computers that appears to its users as a single coherent system Components need to: Communicate Cooperate => support needed Naming – enables some resource sharing Synchronization

EECE 411: Design of Distributed Software Applications Last time Physical clocks Two applications Provide at-most-once semantics Global Positioning Systems ‘Logical clocks’ Where only ordering of events matters Other coordination primitives Leader election: How do I choose a coordinator? Mutual exclusion

EECE 411: Design of Distributed Software Applications Efficient at-most-once message delivery Issues 1: How long to maintain transaction data? 2: How to deal with server failures? (Minimize state that is persistently stored)

EECE 411: Design of Distributed Software Applications Efficient at-most-once message delivery (II) Issue1: How long to maintain transaction data? Solution: Client: Sends transaction id and physical timestamp Client (or network) may resend messages Server: Discards messages with duplicate id and messages that have been generates too far in the past Mechanism Maintains: G = T current - MaxLifeTime - MaxClockSkew Discards messages with timestamps older than G Ignores (or delays) message that arrive in the future (Maintains transaction data only for the interval G--T now

EECE 411: Design of Distributed Software Applications Issue 2: What to persistently store across server failures? Solution: Current Time (CT) is written to disk every ΔT At recovery G failure is recomputed after a crash from saved CT After recovery: discard messages with timestamp older than G failure + ΔT Process messages with timestamp newer than G failure + ΔT [Note: the formulas above ignore clock skew!] Efficient at-most-once message delivery (III)

EECE 411: Design of Distributed Software Applications Uses of (synchronized) physical clocks NTP Using physical clocks to implement at-most- once semantics Global Positioning Systems

EECE 411: Design of Distributed Software Applications GPS – Global Positioning Systems (1) Basic idea: Estimate signal propagation time between satellite and receiver to estimate distance Principle: Problem: Assumes that the clocks of the satellites and receiver are accurate and synchronized: The receiver’s clock is definitely out of synch with the satellite

EECE 411: Design of Distributed Software Applications GPS – Global Positioning Systems (2) X r, Y r, Z r, are unknown coordinates of the receiver. T i is the timestamp on a message from satellite i ∆I i = (T now – T i ) is the measured delay of the message sent by satellite i. Distance to satellite i can be estimated in two ways Propagation time: d i = c x ∆I i Real distance: 3 satellites  3 equations in 3 unknowns So far I assumed receiver clock is synchronized! What if it needs to be adjusted? ∆I = (T now – T i ) + ∆r collect one more measurement frm one more satellite!

EECE 411: Design of Distributed Software Applications Computing position in wired networks Observation: a node P needs at least k + 1 landmarks to compute its own position in a k-dimensional space. Consider two-dimensional case: Solution: P needs to solve three equations in two unknowns (x P,y P ):

EECE 411: Design of Distributed Software Applications Computing Position (…cont) Problems : measured latencies to landmarks fluctuate computed distances will not even be consistent Solution: Measure latencies to L landmarks and let each node P minimize where denotes the actual distance to landmark b i, given a computed coordinate for P.

EECE 411: Design of Distributed Software Applications Logical clocks -- Time Revisited What’s important? What precise time an event occurred? The order in which events occur?

EECE 411: Design of Distributed Software Applications “Happens-before” relation Problem: We first need to introduce a notion of ordering before we can order anything. The happened-before relation on the set of events in a distributed system: if a and b in the same process, and a occurs before b, then a → b if a is an event of sending a message by a process, and b receiving same message by another process then a → b Property: transitive Notation: a → b, when all participants agree that b occurs after a. Two events are concurrent if nothing can be said about the order in which they happened (partial order)

EECE 411: Design of Distributed Software Applications Logical clocks Problem: How do we maintain a global view on the system’s behavior that is consistent with the ‘happened-before’ relation? Solution: attach a timestamp C(e) to each event e, satisfying the following properties: P1: If a and b are two events in the same process, and a → b, then we demand that C(a) < C(b). P2: If a corresponds to sending a message m, and b to the receipt of that message, then also C(a) < C(b). Note: C must only increase Problem: Need to attach timestamps to all events in the system (consistent with the rules above) when there’s no global clock maintain a consistent set of logical clocks, one per process.

EECE 411: Design of Distributed Software Applications Logical clocks (cont) -- Lamport’s Approach Solution: Each process P i maintains a local counter C i and adjusts this counter according to the following rules: (1) For any two successive events that take place within process P i, the counter C i is incremented by 1. (2) Each time a message m is sent by process P i the message receives a timestamp ts(m) = C i (3) Whenever a message m is received by a process P j, P j adjusts its local counter C j to max{C j, ts(m)}; then executes step 1 before passing m to the application. Property P1 is satisfied by (1); Property P2 by (2) and (3). Note: it can still occur that two events happen at the same time. Avoid this by breaking ties through process IDs.

EECE 411: Design of Distributed Software Applications Example

EECE 411: Design of Distributed Software Applications Architectural view Middleware layer in charge of: Stamping messages with clock times, reading timestamps Local management of logical clocks Message ordering (if necessary)

EECE 411: Design of Distributed Software Applications Totally ordered group communication Example Initial state: $100 account balance Update 1: add $100 Update 2: add 1% monthly interest Q: What’s the result if updates are performed in different order at the two replica?

EECE 411: Design of Distributed Software Applications Totally ordered group communication (cont) Solution: Each message is timestamped with local logical time then multicasted When multicasted, also message logically sent to the sender and queued using timestamp order When receiving, the middleware layer Adds message to queue Acknowledges (using multicst) the message Delivers from queue to application only when all acks are received

EECE 411: Design of Distributed Software Applications Totally Ordered Multicast – Algorithm Process P i sends timestamped message msg i to all others. The message itself is put in a local queue queue i. Any incoming message at P k is queued in queue k, according to its timestamp, and acknowledged to every other process. P k passes a message msg i to its application if: msg i is at the head of queue k for each process P x, there is a message msg x in queue k with a larger timestamp. Note: We are assuming that communication is reliable and FIFO ordered. Guarantee: all multicasted messages in the same order at all destination. Nothing is guaranteed about the actual order!

EECE 411: Design of Distributed Software Applications So far … Physical clocks Two applications Provide at-most-once semantics Global Positioning Systems ‘Logical clocks’ Where only ordering of events matters