1. Distributed Systems CS 15-440
Synchronization – Part III
Lecture 10, October 5, 2016
Mohammad Hammoud

2. Today… Last Session: Logical Clocks Today’s Session:
Distributed Mutual Exclusion
Election Algorithms
Announcements:
Midterm exam is on Wednesday, Oct 12 (it is open book, open notes)
PS3 is out. It is due on Monday, Oct 17
Project II is out. It is due on Oct 23 by midnight

3. Where do We Stand in the Synchronization Chapter?
Previous two lectures
Time Synchronization
Physical Clock Synchronization (or, simply, Clock Synchronization)
Here, actual time on the computers are synchronized
Logical Clock Synchronization
Computers are synchronized based on the relative ordering of events
Mutual Exclusion
How to coordinate between processes that access the same resource?
Election Algorithms
Here, a group of entities elect one entity as the coordinator for solving a problem
Today’s lecture

4. Summary of Lamport’s Clock
Lamport advocated using logical clocks
Processes synchronize based on the time values of their logical clocks rather than the absolute time values of their physical clocks
Which applications in DS need logical clocks?
Applications with provable ordering of events
Perfect physical clock synchronization is hard to achieve in practice. Hence we cannot provably order the events
Applications with rare events
Events are rarely generated, and physical clock synchronization overhead is not justified
However, Lamport’s clock cannot guarantee perfect ordering of events by just observing the time values of two arbitrary events

5. Limitation of Lamport’s Clock
Lamport’s Clock ensures that if ab, then C(a) < C(b)
However, it does not say anything about any two arbitrary events a and b by only comparing their time values
For any two arbitrary events a and b, C(a) < C(b) does not mean that ab
Example: P1 P2 P3
Compare m1 and m3
m1:6 6 8 10
m2:20
P2 can infer that m1m3 12 16 20 18 24 30
m3:32 24 32 40
Compare m1 and m2 30 40 50
P2 cannot infer that m1m2 or m2m1 36 48 60 42 61 56 70 48 64 80 54 54 72 90 60 80
100

6. Logical Clocks We will study two types of logical clocks
Lamport’s Clock
Vector Clocks

7. Vector Clocks
Vector Clocks was proposed to overcome the limitation of Lamport’s clock (i.e., C(a)< C(b) does not mean that ab)
The property of inferring that a occurred before b is known as the causality property
A Vector clock for a system of N processes is an array of N integers
Every process Pi stores its own vector clock VCi
Lamport’s time values for events are stored in VCi
VCi(a) is assigned to an event a
If VCi(a) < VCi(b), then we can infer that ab (or more precisely, that event a causally precedes event b)

8. Updating Vector Clocks
Vector clocks are constructed by the following two properties:
VCi[i] is the number of events that have occurred at process Pi so far
VCi[i] is the local logical clock at process Pi
If VCi[j]= k, then Pi knows that k events have occurred at Pj
VCi[j] is Pi’s knowledge of the local time at Pj
Increment VCi whenever a new event occurs
Pass VCj along with the message

9. Vector Clock Update Algorithm
Whenever there is a new event at Pi, increment VCi[i]
When a process Pi sends a message m to Pj:
Increment VCi[i]
Set m’s timestamp ts(m) to the vector VCi
When message m is received process Pj :
VCj[k] = max(VCj[k], ts(m)[k]) ; (for all k)
Increment VCj[j]
VC0=(0,0,0)
VC0=(1,0,0)
VC0=(2,0,0) P0
m:(2,0,0)
VC1=(0,0,0)
VC1=(2,1,0) P1
VC2=(0,0,0) P2

10. Inferring Events with Vector Clocks
Let a process Pi send a message m to Pj with timestamp ts(m), then:
Pj knows the number of events at the sender Pi that causally precede m
(ts(m)[i] – 1) denotes the number of events at Pi
Pj also knows the minimum number of events at other processes Pk that causally precede m
(ts(m)[k] – 1) denotes the minimum number of events at Pk
VC0=(0,0,0)
VC0=(1,0,0)
VC0=(2,0,0) P0
m:(2,0,0)
VC1=(0,0,0)
VC1=(0,1,0)
VC1=(2,2,0)
VC1=(2,3,0) P1
m’:(2,3,0)
VC2=(0,0,0)
VC2=(2,3,1) P2

11. Enforcing Causal Communication
Assume that messages are multicast within a group of processes, P0, P1 and P2
To enforce causally-ordered multicasting, the delivery of a message m sent from Pi to Pj can be delayed until the following two conditions are met:
ts(m)[i] = VCj[i] + 1 (Condition I)
ts(m)[k] <= VCj[k] for all k != i (Condition II)
Assuming that Pi only increments VCi[i] upon sending m and adjusts VCi[k] to max{VCi[k], ts(m)[k]} for each k upon receiving a message m’
VC0=(0,0,0)
VC0=(1,0,0)
VC0=(1,1,0) P0
m:(1,0,0)
m:(1,1,0)
VC1=(0,0,0)
VC1=(1,0,0)
VC1=(1,1,0) P1
VC2=(0,0,0)
VC2=(1,0,0)
VC2=(1,1,0) P2
VC2=(1,1,0)
Condition II does not hold  Delay delivery

12. Summary – Logical Clocks
Logical Clocks are employed when processes have to agree on relative ordering of events, but not necessarily actual time of events
Two types of Logical Clocks
Lamport’s Logical Clocks
Supports relative ordering of events across different processes by using the happened-before relationship
Vector Clocks
Supports causal ordering of events

13. Overview Time Synchronization Mutual Exclusion Election Algorithms
Clock Synchronization
Logical Clock Synchronization
Mutual Exclusion
Election Algorithms

14. Need for Mutual Exclusion
Distributed processes need to coordinate to access shared resources
Example: Writing a file in a Distributed File System
Client A
Server
Client C
Read from file abc.txt
Write to file abc.txt P1 P3
Distributed
File
abc.txt
Client B P2
Write to file abc.txt
In uniprocessor systems, mutual exclusion to a shared resource is provided through operating system support.
However, such support is insufficient to enable mutual exclusion of distributed entities
In Distributed Systems, processes coordinate accesses to a shared resource by passing messages to enforce distributed mutual exclusion

15. Types of Distributed Mutual Exclusion
Mutual exclusion algorithms are classified into two categories
Permission-based Approaches
A process, which wants to access a shared resource, requests the permission from one or more coordinators
Token-based Approaches
Each shared resource has a token
Token is circulated among all the processes
A process can access the resource if it has the token
Request to access
Coordinator C1
Client 1
Grant P1
Server
Access
Resource
Server
Resource
Access
Access
Access
Client 1
Client 2
Client 3 P1 P2 P3
Token
Token
Token

16. Overview Time Synchronization Mutual Exclusion Election Algorithms
Clock Synchronization
Logical Clock Synchronization
Mutual Exclusion
Permission-based Approaches
Token-based Approaches
Election Algorithms

17. Permission-based Approaches
There are two types of permission-based mutual exclusion algorithms
Centralized Algorithms
Decentralized Algorithms
We will study an example of each type of algorithms

18. a. A Centralized Algorithm
One process is elected as a coordinator (C) for a shared resource
Coordinator maintains a Queue of access requests
Whenever a process wants to access the resource, it sends a request message to the coordinator to access the resource
When the coordinator receives the request:
If no other process is currently accessing the resource, it grants the permission to the process by sending a “grant” message
If another process is accessing the resource, the coordinator queues the request, and may not reply to the request
The process releases the exclusive access after accessing the resource
The coordinator will then send the “grant” message to the next process in the queue P0 P1 P2
Rel
Req
Grant
Access
Req
Grant
Access
Resource C P2 P1
Queue

19. Discussion about Centralized Algorithm
Blocking vs. Non-blocking Requests
The coordinator can block the requesting process until the resource is free
Otherwise, the coordinator can send a “permission-denied” message back to the process
The process can poll the coordinator at a later time, or
The coordinator queues the request. Once the resource is released, the coordinator will send an explicit “grant” message to the process
The algorithm guarantees mutual exclusion, and is simple to implement
Fault-Tolerance:
Centralized algorithm is vulnerable to a single-point of failure (at coordinator)
Processes cannot distinguish between dead coordinator and request blocking
Performance Bottleneck:
In a large system, single coordinator can be overwhelmed with requests

20. b. A Decentralized Algorithm
To avoid the drawbacks of the centralized algorithm, Lin et al. [1] advocated a decentralized mutual exclusion algorithm
Assumptions:
Distributed processes are in a Distributed Hash Table (DHT) based system
Each resource is replicated n times
The ith replica of a resource rname is named as rname-i
Every replica has its own coordinator for controlling access
The coordinator for rname-i is determined by using a hash function
Approach:
Whenever a process wants to access the resource, it will have to get a majority vote from m > n/2 coordinators
If a coordinator does not want to vote for a process (because it has already voted for another process), it will send a “permission-denied” message to the process

21. A Decentralized Algorithm – An Example
If n=10 and m=7, then a process needs at-least 7 votes to access the resource C1
rname-1
Req
Access C2
rname-2 5 6 7 3 4 2 1 P0 OK C3
rname-3 C4
rname-4
Req C5
rname-5 1 3 2 P1 OK C6
rname-6
Deny C7
rname-7
Deny C8
rname-8 C9
rname-9
C10
rname-10 Pi n
= Number of votes gained
= Process i Cj
= Coordinator j
rname-x
xth replica
= of a resource
rname

22. Fault-tolerance in Decentralized Algorithm
The decentralized algorithm assumes that the coordinator recovers quickly from a failure
However, the coordinator would have reset its state after recovery
Coordinator could have forgotten any vote it had given earlier
Hence, the coordinator may incorrectly grant permission to the processes
Mutual exclusion cannot be deterministically guaranteed
But, the algorithm probabilistically guarantees mutual exclusion

23. Probabilistic Guarantees in the Decentralized Algorithm
What is the minimum number of coordinators who should fail for violating mutual exclusion?
At least n-m+1 coordinators should fail
Let the probability of violating mutual exclusion be Pv
Derivation of Pv
Let T be the lifetime of the coordinator
Let p=Δt/T be the probability that a coordinator crashes during time-interval Δt
Let P[k] be the probability that k out of m coordinators crash during the same interval
We compute the mutual exclusion violation probability Pv by:
In practice, this probability should be very small
For T=3 hours, Δt=10 s, n=32, and m=0.75n : Pv =10-40

24. Overview Time Synchronization Mutual Exclusion Election Algorithms
Clock Synchronization
Logical Clock Synchronization
Mutual Exclusion
Permission-based Approaches
Token-based Approaches
Election Algorithms

25. A Token Ring Algorithm
In the Token Ring algorithm, each resource is associated with a token
The token is circulated among the processes
The process with the token can access the resource
How the token is circulated among processes?
Resource
Access T T
All processes form a logical ring where each process knows its next process
One process is given the token to access the resource
The process with the token has the right to access the resource
If the process has finished accessing the resource OR does not want to access the resource:
it passes the token to the next process in the ring P0 P1 P2 P3 P4 P5 P6 P7 T T T T T T

26. Discussion about Token Ring
Token ring approach provides deterministic mutual exclusion
There is one token, and the resource cannot be accessed without a token
Token ring approach avoids starvation
Each process will receive the token
Token ring has a high-message overhead
When no processes need the resource, the token circulates at a high-speed
If the token is lost, it must be re-generated
Detecting the loss of the token is difficult (especially if the amount of time between successive appearances of the token is unbounded)
Dead processes must be purged from the ring
ACK based token delivery can assist in purging dead processes

27. Comparison of Mutual Exclusion Algorithms
Delay before a process can access the resource (in message times)
Number of messages required for a process to access and release the shared resource
Problems
Centralized
Decentralized
Token Ring 2 3
Coordinator crashes
2mk
2mk + m; k=1,2,…
Large number of messages
Token may be lost
Ring can cease to exist since processes crash
0 to (n-1)
1 to ∞
Assume that:
n = Number of processes in the distributed system
For the Decentralized algorithm:
m = minimum number of coordinators who have to agree for a process to access a resource
k = average number of requests made by the process to a coordinator to request for a vote

28. Overview Time Synchronization Mutual Exclusion Election Algorithms
Clock Synchronization
Logical Clock Synchronization
Mutual Exclusion
Permission-based Approaches
Token-based Approaches
Election Algorithms

29. Election in Distributed Systems
Many distributed algorithms require one process to act as a coordinator
Typically, it does not matter which process is elected as the coordinator
Time server
Client 1
Server
Resource P1
Coordinator C1
Home Node Selection in Naming
Berkeley Clock Synchronization Algorithm
A Centralized Mutual Exclusion Algorithm

30. Election Process
Any process Pi in the DS can initiate the election algorithm that elects a new coordinator
At the termination of the election algorithm, the elected coordinator process should be unique
Every process may know the process ID of every other process, but it does not know which processes have crashed
Generally, we require that the coordinator is the process with the largest process ID
The idea can be extended to elect best coordinator
Example: Election of a coordinator with least computational load
If the computational load of process Pi denoted by loadi, then coordinator is the process with highest 1/loadi. Ties are broken by sorting process ID.

31. Election Algorithms We will study two election algorithms
Bully Algorithm
Ring Algorithm

32. 1. Bully Algorithm
A process initiates election algorithm when it notices that the existing coordinator is not responding
Process Pi calls for an election as follows:
Pi sends an “Election” message to all processes with higher process IDs
When process Pj with j>i receives the message, it responds with a “Take-over” message. Pi no more contests in the election
Process Pj re-initiates another call for election. Steps 1 and 2 continue
If no one responds, Pi wins the election. Pi sends “Coordinator” message to every process 1 5 6 3 7 4 2
Take-Over
Election
Election
Take-Over
Coordinator
Election
Election
Take-over
Election
Election X

33. 2. Ring Algorithm
This algorithm is generally used in a ring topology
When a process Pi detects that the coordinator has crashed, it initiates an election algorithm
Pi builds an “Election” message (E), and sends it to its next node. It inserts its ID into the Election message
When process Pj receives the message, it appends its ID and forwards the message
If the next node has crashed, Pj finds the next alive node
When the message gets back to the process that started the election:
it elects process with highest ID as coordinator, and
changes the message type to “Coordination” message (C) and circulates it in the ring
E: 5,6,0
C: 6
E: 5,6,0,1
C: 6 1 2 3 4 5 6 7
E: 5,6,0,1,2
C: 6 X
E: 5,6
C: 6
E: 5,6,0,1,2,3
C: 6
C: 6
E: 5
C: 6
E: 5,6,0,1,2,3,4

34. Comparison of Election Algorithms
Number of Messages for Electing a Coordinator
Problems
Bully Algorithm
Ring Algorithm
O(n2)
Large message overhead 2n
An overlay ring topology is necessary
Assume that:
n = Number of processes in the distributed system

35. Summary of Election Algorithms
Election algorithms are used for choosing a unique process that will coordinate certain activities
At the end of the election algorithm, all nodes should uniquely identify the coordinator
We studied two algorithms for election
Bully algorithm
Processes communicate in a distributed manner to elect a coordinator
Ring algorithm
Processes in a ring topology circulate election messages to choose a coordinator

36. Election in Large-Scale Networks
Bully Algorithm and Ring Algorithm scale poorly with the size of the network
Bully Algorithm needs O(n2) messages
Ring Algorithm requires maintaining a ring topology and requires 2n messages to elect a leader
In large networks, these approaches do not scale well
We discuss a scalable election algorithm for large-scale peer-to-peer networks

37. Election in Large-Scale Peer-to-Peer Networks
Many P2P networks have a hierarchical architecture for balancing the advantages between centralized and distributed networks
Typically, P2P networks are neither completely unstructured nor completely centralized
Centralized networks are efficient and, they easily facilitate locating entities and data
Flat unstructured peer-to-peer networks are robust, autonomous and balances load between all peers

38. Super-peers
In large unstructured Peer-to-Peer Networks, the network is organized into peers and super-peers
A super-peer is an entity that does not only participate as a peer, but also carries on an additional role of acting as a leader for a set of peers
Super-peer acts as a server for a set of client peers
All communication from and to a regular peer proceeds through a super-peer
It is expected that super-peers are long-lived nodes with high-availability
Super Peer
Regular Peer
Super-Peer Network

39. Super-Peers – Election Requirements
In a hierarchical P2P network, several nodes have to be selected as super-peers
Traditionally, only one node is selected as a coordinator
Requirements for a node being elected as a super-peer
Super-peers should be evenly distributed across the overlay network
There should be a predefined proportion of super-peers relative to the number of regular peers
Each super-peer should not need to serve more than a fixed number of regular peers

40. Election of Super-peers in a DHT-based systemm k

41. Next Class
Consistency and Replication- Part I

42. References
[1] Shi-Ding Lin, Qiao Lian, Ming Chen and Zheng Zhang, “A Practical Distributed Mutual Exclusion Protocol in Dynamic Peer-to-Peer Systems”, Lecture Notes in Computer Science, 2005, Volume 3279/2005, 11-21, DOI:  / _2
[2]