1. Chapter 18 Distributed Control Algorithms
Copyright © 2008

2. Introduction Operation of Distributed Control Algorithms
Correctness of Distributed Control Algorithms
Distributed Mutual Exclusion
Distributed Deadlock Handling
Distributed Scheduling Algorithms
Distributed Termination Detection
Election Algorithms
Practical Issues in Using Distributed Control Algorithms
Operating Systems, by Dhananjay Dhamdhere
Operating Systems, by Dhananjay Dhamdhere Copyright © 2008 2

3. OS Control Functions in a Distributed Environment
Special features of distributed OS control functions
Mutual exclusion
Involves synchronization of processes in different computers
Deadlock handling
Deadlocks may involve use of resources in different hosts
Scheduling
Perform load balancing for comparable loading of computers
Termination detection
Check whether all processes of a computation, which may operate in different computers, have completed
Election
Elect coordinator for a privileged function
Operating Systems, by Dhananjay Dhamdhere Copyright © 2008
Operating Systems, by Dhananjay Dhamdhere 3

4. Operation of Distributed Control Algorithms (continued)
A distributed control algorithm operates in parallel with its clients, so that it can respond readily to events related to its service
Each process has a control part and a basic part 4

5. Correctness of Distributed Control Algorithms
Algorithm correctness has two facets:
Liveness: eventually performs correct actions
i.e., without indefinite delays
Safety: does not perform wrong actions
Proving correctness of a distributed algorithm is a complex task
Operating Systems, by Dhananjay Dhamdhere
Operating Systems, by Dhananjay Dhamdhere Copyright © 2008 5

6. Correctness of Distributed Control Algorithms (continued)
Operating Systems, by Dhananjay Dhamdhere
Operating Systems, by Dhananjay Dhamdhere Copyright © 2008 6

7. Distributed Mutual Exclusion
A Permission-Based Algorithm
Token-Based Algorithms
Operating Systems, by Dhananjay Dhamdhere
Operating Systems, by Dhananjay Dhamdhere Copyright © 2008 7

8. A Permission-Based Algorithm
Ricart and Agrawala algorithm: CS entry in FCFS order
Requires 2 x (n – 1 ) messages per CS entry

10. Maekawa Algorithm
Each process has a request set of processes; it seeks the permission of only processes in the request set (Ri represents the request set of process Pi)
Correctness is ensured through the following rules:
For all Pi : Pi is included in Ri
For all Pi, Pj: Ri ∩ Rj is non-null
The algorithm requires 2 x √n messages per CS entry
Operating Systems, by Dhananjay Dhamdhere Copyright © 2008
Operating Systems, by Dhananjay Dhamdhere 10

11. Token-Based Algorithms for Mutual Exclusion
A token represents the privilege to use a CS
Only a process possessing token can enter CS
Safety of algorithm follows from this rule
Liveness: must ensure token eventually reaches a (requesting) process
These algorithms use abstract system models
Edges represent the paths used to pass control messages
For example:
Abstract ring and tree topologies
Operating Systems, by Dhananjay Dhamdhere
Operating Systems, by Dhananjay Dhamdhere Copyright © 2008 11

12. An Algorithm Employing the Ring Topology

13. Raymond’s Token-Based Algorithm
Features of the algorithm
The algorithm uses an abstract inverted tree to reduce the number of messages. It has three invariants
Pholder, the process holding the token, is at root of the tree
Each process in the system belongs to the tree
Each process other than the Pholder has only one edge, which points to its parent in the tree
Thus, each process has a path that ends on Pholder
Each process has a local request queue
When it receives a request, it puts the requestor’s id in the queue
When it makes a request, it puts its own id in the queue
Operating Systems, by Dhananjay Dhamdhere Copyright © 2008
Operating Systems, by Dhananjay Dhamdhere 13

14. Token is transferred to P3
Raymond’s Algorithm
Uses an abstract inverted tree as the system model
Token is transferred to P3 14

16. Distributed Deadlock Handling
Deadlock detection, prevention, and avoidance approaches studied earlier use state information
Problems arise in extending these approaches to a distributed system
Approaches in distributed systems:
Detection: centralized and distributed
Prevention
No special techniques for distributed deadlock avoidance have been discussed in OS literature
Model used in this section:
SISR model of resource allocation
Operating Systems, by Dhananjay Dhamdhere
Operating Systems, by Dhananjay Dhamdhere Copyright © 2008 16

17. Problems in Centralized Deadlock Detection
Steps in centralized deadlock detection:
Collect WFGs of all nodes at a central node
Superimpose them to form a merged WFG
Use a conventional deadlock detection algorithm
Problem: can lead to phantom deadlocks
Violates safety property in deadlock detection
Operating Systems, by Dhananjay Dhamdhere
Operating Systems, by Dhananjay Dhamdhere Copyright © 2008 17

18. Distributed Deadlock Detection
Key issues in deadlock detection:
A cycle indicates a deadlock in an SISR system
A knot indicates a deadlock in an MISR system
Distributed deadlock detection approach:
Cycles and knots detected through joint actions of nodes in system
Every node can detect and declare a deadlock
Two such algorithms:
Diffusion computation-based algorithm
Edge chasing algorithm
Operating Systems, by Dhananjay Dhamdhere
Operating Systems, by Dhananjay Dhamdhere Copyright © 2008 18

19. Diffusion Computation-Based Deadlock Detection
Diffusion computation: used to collect info about nodes
Diffusion phase
Computation that has originated in one node, spreads to other nodes
A control message called a query is sent along each edge
The first query received by a node is called an engaging query. On receiving it, the node sends queries along all its out-edges
Information collection phase
Each node sends information in response to each query
It sends a dummy reply for a non-engaging query
It collects information from all replies it received, adds its own information, and sends the result as the reply to the engaging query. We call it an engaging reply
Operating Systems, by Dhananjay Dhamdhere Copyright © 2008
Operating Systems, by Dhananjay Dhamdhere 19

20. Diffusion Computation-Based Algorithm
Algorithm 18.4 was proposed by Chandy, Misra, and Haas (1983)
Works for SISR and MISR systems
P2, P3 are blocked. P1 becomes blocked and sends a query but does
not receive a reply because P4 is not blocked
P4 requests a resource held by P1, becomes blocked and sends a query.
It would receive a reply and declare a deadlock
Operating Systems, by Dhananjay Dhamdhere Copyright © 2008
Operating Systems, by Dhananjay Dhamdhere 20

21. Diffusion Computation-Based Algorithm (continued)

22. Mitchell–Merritt Algorithm for Distributed Deadlock Detection
It is an edge chasing algorithm—control messages are sent over WFG edges to detect cycles
A provision is made to ensure that the cycle has not been broken before it was detected
Each process is assigned a public label and a private label
The labels are identical when a process is created
The public label of a process changes when it gets blocked on a resource request
It also changes when it waits for a process having a larger public label
A wait-for edge with a specific relation between public and private labels of its source and destination processes indicates presence of a deadlock
Operating Systems, by Dhananjay Dhamdhere Copyright © 2008
Operating Systems, by Dhananjay Dhamdhere 22

23. Mitchell-Merritt Algorithm
Rules of the algorithm:
public label
private label
Operating Systems, by Dhananjay Dhamdhere
Operating Systems, by Dhananjay Dhamdhere Copyright © 2008 23

24. Distributed Deadlock Prevention
Cycles are prevented as follows:
A pair (local time, node id) is used to time-stamp creation of a process
When process Pi requests a resource allocated to Pj, time-stamps of Pi, Pj are used to decide whether Pi can wait for Pj
Two approaches
Wait-or-die
Pi is allowed to wait if older than Pj; otherwise, it is killed
Would-or-wait
Pi is allowed to wait if younger than Pj; otherwise Pj is killed
A killed process retains original timestamp if restarted
Operating Systems, by Dhananjay Dhamdhere Copyright © 2008
Operating Systems, by Dhananjay Dhamdhere 24

25. Distributed Scheduling Algorithms
A distributed scheduling algorithm balances computational loads in the nodes
Uses process migration
Apply a threshold to decide if a node is heavily or lightly loaded
Operating Systems, by Dhananjay Dhamdhere
Operating Systems, by Dhananjay Dhamdhere Copyright © 2008 25

26. Distributed Scheduling Algorithms (continued)
Migration may be preemptive or nonpreemptive
Stability is an important issue
An unstable algorithm may lead to a situation similar to thrashing
A process is transferred very often; does not make progress
Algorithms can be sender- or receiver-initiated
A heavily loaded node is a sender
A lightly loaded node is a receiver
A symmetrically initiated algorithm contains features of both
Operating Systems, by Dhananjay Dhamdhere
Operating Systems, by Dhananjay Dhamdhere Copyright © 2008 26

27. Distributed Scheduling Algorithms (continued)
Operating Systems, by Dhananjay Dhamdhere
Operating Systems, by Dhananjay Dhamdhere Copyright © 2008 27

28. Distributed Scheduling Algorithms (continued)
Operating Systems, by Dhananjay Dhamdhere
Operating Systems, by Dhananjay Dhamdhere Copyright © 2008 28

29. Distributed Termination Detection
When a process terminates, OS frees its resources
This approach is not adequate for distributed systems
Processes of a distributed computation execute in different nodes of a distributed system. They should be terminated when all of them have completed their tasks
These processes perform work assigned to them
A process is active when it is performing work, and passive when it has no work
Work is assigned to a process through a message
Passive process becomes active on receiving a message
Distributed termination condition (DTC):
All processes of a distributed computation are passive
No basic messages are in transit
Operating Systems, by Dhananjay Dhamdhere
Operating Systems, by Dhananjay Dhamdhere Copyright © 2008 29

30. Distributed Termination Detection (continued)
Credit-distribution-based termination detection
Every process is assigned a numerical credit
It sends some of its credit in each message
When a process becomes passive, it sends its credit to collector process
The distributed computation is known to have terminated when credit accumulated by collector is C
Diffusion computation-based termination detection
Each process that becomes passive initiates a diffusion computation to determine if DTC holds
Operating Systems, by Dhananjay Dhamdhere
Operating Systems, by Dhananjay Dhamdhere Copyright © 2008 30

31. Distributed Termination Detection (continued)
Operating Systems, by Dhananjay Dhamdhere
Operating Systems, by Dhananjay Dhamdhere Copyright © 2008 31

32. Election Algorithms
Critical system functions are assigned to a coordinator (for the function)
E.g., replacing lost token in a token-based algorithm
Coordinator is typically the highest-priority process in set
A process that finds that coordinator is not responding assumes it has failed
Initiates election algorithm
Algorithm chooses highest-priority nonfailed process as new coordinator
Then, announces its id to all nonfailed processes
Operating Systems, by Dhananjay Dhamdhere
Operating Systems, by Dhananjay Dhamdhere Copyright © 2008 32

33. Election Algorithms for Unidirectional Ring Topologies
Links in ring are assumed to be FIFO channels
Assumption: control part of failed process continues to function and forwards received messages along out-edge
Election performed by obtaining ids of all nonfailed processes and electing highest-priority process
Two types of messages:
“elect me” and “new coordinator”
O(n2) messages or 3n−1 (worst case) in refined version
Operating Systems, by Dhananjay Dhamdhere
Operating Systems, by Dhananjay Dhamdhere Copyright © 2008 33

34. Election Algorithms Algorithm for the ring topology
Process Pi initiates by sending (“elect me”, Pi) message
Process Pj receiving an (“elect me”, Pi) message sends an (“elect me”, Pj) message and then forwards Pi’s message
Pi receives back its own message after receiving message of every other process; it elects the highest priority process as leader
It sends a “new coordinator” message to inform others
Algorithm 2: Refinement of algorithm 1
In Step 2, Pj sends its own message if its priority is higher than Pi’s; otherwise, it sends Pi’s message
Only highest priority process would get back its own message
Operating Systems, by Dhananjay Dhamdhere Copyright © 2008
Operating Systems, by Dhananjay Dhamdhere 34

35. Election Algorithm Bully algorithm
Initiator Pi sends (“elect me”, Pi) messages to all higher priority processes and starts a time-out interval T1
If a time-out occurs, it sends a “new coordinator” message to lower priority processes
If it receives a “don’t you dare” message from a higher priority process Pj, it starts another time-out interval T2
If a time-out occurs, it starts a new election
If a process Pj receives an “elect me” message from a lower priority process
It sends a “don’t you dare” message to its sender
Starts a new election by sending (“elect me”, Pj) messages
Requires O(n2) messages per election
Operating Systems, by Dhananjay Dhamdhere Copyright © 2008
Operating Systems, by Dhananjay Dhamdhere 35

36. Practical Issues in Using Distributed Control Algorithms
Resource Management
Process Migration
Operating Systems, by Dhananjay Dhamdhere
Operating Systems, by Dhananjay Dhamdhere Copyright © 2008 36

37. Resource Management
Name server in each node must be updated when resources are added
Solution: use an arrangement of name servers as in the domain name service (DNS)
Only the name server of a domain needs to be updated when a resource is added
Operating Systems, by Dhananjay Dhamdhere
Operating Systems, by Dhananjay Dhamdhere Copyright © 2008 37

38. Process Migration Reasons for process migration:
To achieve load balancing
To reduce network traffic involved in utilizing a remote resource
To provide availability of services when a node has to be shut down for maintenance
Operating Systems, by Dhananjay Dhamdhere
Operating Systems, by Dhananjay Dhamdhere Copyright © 2008 38

39. Process Migration Difficulties
Process state is distributed in various data structures of the OS
Process id’s may change due to migration
Process id’s are used in interprocess communication
Solution: Use global process ids as in Sun cluster
Delivery of messages requires a special provision
A node receiving a message would redirect it if the destination process has migrated out of it
This residual state causes poor reliability
Alternatively, all processes may be informed when a process is migrated
Requires a complex protocol
Operating Systems, by Dhananjay Dhamdhere Copyright © 2008
Operating Systems, by Dhananjay Dhamdhere 39

40. Summary
Actions of a distributed control algorithms are performed in many nodes of the system
Two aspects of correctness are liveness and safety
Distributed system models: physical and logical
Examples of distributed control algorithms:
Distributed mutual exclusion: e.g., token based
Distributed deadlock detection: diffusion computation
Distributed scheduling (to balance load)
Distributed termination: e.g., credit-based
Election (highest priority process wins)
Operating Systems, by Dhananjay Dhamdhere
Operating Systems, by Dhananjay Dhamdhere Copyright © 2008 40