1. CSS490 Replication & Fault Tolerance
Textbook Ch9 (p440 – 484)
Instructor: Munehiro Fukuda
These slides were compiled from the course textbook and the reference books.
Hello! Everyone,
My name is Shinya Kobayashi. Today, I am going to present our paper titled “Inter-Cluster Job Coordination Using Mobile Agents” on behalf of the first author, Munehiro Fukuda. Munehiro was hoping to show up and present the paper at AMS2001, however he got to wait in Japan until he will get an H1B visa. Since I received the presentation materials from him quite recently, please allow me to present this paper using this script.
I can respond to your questions as far as I know, however you can also ask Munehiro by . His address is on the title page of our paper.
(time 1:05)
Winter, 2004
CSS490 Fault Tolerance

2. File Replication Concepts
Difference between replication and caching
A replica is associated with a server, whereas a cache with client.
A replicate focuses on availability, while a cache on locality
A replicate is more persistent than a cache is
A cache is contingent upon a replica
Advantages
Increased availability/reliability
Performance enhancement (response time and network traffic)
Scalability and autonomous operation
Requirements
Naming: no need to be aware of multiple replicas.
Consistency: data consistency among replicated files.
Replication control: explicit v.s. implicit/lazy replication
ACID: Atomicity, Consistency, Isolation, and Durability
Winter, 2004
CSS490 Fault Tolerance

3. File Replication Basic Architectural Model
Request: send a client request to a server.
Coordination: deliver the request to each replica manger in some order.
Execution: process a client request but not permanently commit it.
Agreement: agree if the execution will be committed
Response: respond to the front end
Client
Replica
Manger
Front
End
Replica
Manger
Client
Front
End
Replica
Manger
Ex: DNS Web server
Winter, 2004
CSS490 Fault Tolerance

4. Group Communication Group membership service
Create and destroy a group.
Add or withdraw a replica manager to/from a group.
Detect a failure.
Notify members of group membership changes.
Provide clients with a group address.
Message delivery
Absolute ordering
Consistent ordering
Replica
Manger
Replica
Manger
Client
Replica
Manger
Replica
Manger
group
Winter, 2004
CSS490 Fault Tolerance

5. Absolute Ordering Linearizability
Rule:
Mi must be delivered before mj if Ti < Tj
Implementation:
A clock synchronized among machines
A sliding time window used to commit message delivery whose timestamp is in this window.
Example:
Distributed simulation
Drawback
Too strict constraint
No absolute synchronized clock
No guarantee to catch all tardy messages
Ti < Tj Ti mi Tj mi mj mj
Winter, 2004
CSS490 Fault Tolerance

6. Consistent (Total) Ordering Sequential Consistency
Rule:
Messages received in the same order (regardless of their timestamp).
Implementation:
A message sent to a sequencer, assigned a sequence number, and finally multicast to receivers
A message retrieved in incremental order at a receiver
Example:
Replicated database update
Drawback:
A centralized algorithm
Ti < Tj Ti Tj mj mj mi mi
Winter, 2004
CSS490 Fault Tolerance

7. Two-Phase Commit Protocol
INIT
WAIT
ABORT
COMMIT
Commit
Vote-request
Vote-abort
Global-abort
Vote-commit
Global-commit
Coordinator
INIT
READY
ABORT
COMMIT
Vote-request
Vote-commit
Global-commit
Ack
Global-abort
Vote-abort
Worker 1
INIT
READY
ABORT
COMMIT
Vote-request
Vote-commit
Global-commit
Ack
Global-abort
Vote-abort
Worker 2
Another possible cases:
The coordinator didn’t receive all vote-commits. → Time out and send a global-abort.
A worker didn’t receive a vote-request. → All workers eventually receive a global-abort.
A worker didn’t receive a global-commit. → Time out and check the other work’s status.
Winter, 2004
CSS490 Fault Tolerance

8. Multi-copy Update Problem
Read-only replication
Allow the replication of only immutable files.
Primary backup replication
Designate one copy as the primary copy and all the others as secondary copies.
Active backup replication
Access any or all of replicas
Read-any-write-all protocol
Available-copies protocol
Quorum-based consensus
Winter, 2004
CSS490 Fault Tolerance

9. Primary-Copy Replication
Request: The front end sends a request to the primary replica.
Coordination:. The primary takes the request atomically.
Execution: The primary executes and stores the results.
Agreement: The primary sends the updates to all the backups and receives an ask from them.
Response: reply to the front end.
Advantage: an easy implementation, linearizable, coping with n-1 crashes.
Disadvantage: large overhead especially if the failing primary must be replaced with a backup.
Client
Replica
Manger
Front
End
Primary
Backup
Replica
Manger
Client
Front
End
Replica
Manger
Backup
Winter, 2004
CSS490 Fault Tolerance

10. Active Replication Request: The front end multicasts to all replicas.
Coordination:. All replica take the request in the sequential order.
Execution: Every replica executes the request.
Agreement: No agreement needed.
Response: Each replies to the front.
Advantage: achieve sequential consistency, cope with (n/2 – 1) byzantine failures
Disadvantage: no more linearizable
Client
Replica
Manger
Front
End
Replica
Manger
Client
Front
End
Replica
Manger
Winter, 2004
CSS490 Fault Tolerance

11. Read-Any-Write-All Protocol
Read from any one of them
Read
Lock any one of replicas for a read
Write
Lock all of replicas for a write
Sequential consistency
Intolerable for even 1 failing replica upon a write.
Client
Replica
Manger
Front
End
Write to all of them
Replica
Manger
Client
Front
End
Replica
Manger
Replica
Manger
Winter, 2004
CSS490 Fault Tolerance

12. Available-Copies Protocol
Read
Lock any one of replicas for a read
Write
Lock all available replicas for a write
Recovering replica
Bring itself up to date by coping from other servers before accepting any user request.
Better availability
Cannot cope with network partition. (Inconsistency in two sub-divided network groups)
Read from any one of them
Client
Replica
Manger
Front
End
Write to all available replicats
Replica
Manger X
Client
Front
End
Replica
Manger
Replica
Manger
Winter, 2004
CSS490 Fault Tolerance

13. Quorum-Based Protocols
#replicas in read quorum + #replicas in write quorum > n
Read
Retrieve the read quorum
Select the one with the latest version.
Perform a read on it
Write
Retrieve the write quorum.
Find the latest version and increment it.
Perform a write on the entire write quorum.
If a sufficient number of replicas from read/write quorum, the operation must be aborted.
Client
Replica
Manger
Front
End
Read quorum
Write quorum
Read-any-write-all: r = 1, w = n
Winter, 2004
CSS490 Fault Tolerance

14. ISIS System Process group: see page 4 of this ppt file Group view
Reliable multicast
Causal multicast: see pages 5 & 6 of MPI ppt file
Atomic broadcast: see page 7 of this ppt file p1 p2 p3 p4
Joins the group
multicast
crashed
Partially multicast messages
must be discarded
Multicast to
available processes
rejoins
Winter, 2004
CSS490 Fault Tolerance

15. Gossip Architecture If (Tj > Tk) update RMk else
discard the gossip message
RMk
Gossip
RMi
(Ti)
RMj
(Tj)
Update, Tf
Query, Tf
Value, Ti
Update id
If (Tf < Ti)
return value
else {
waits for RMi to be updated or
query RMj/RMk}
If (Tf > Tj)
update RMj
else {
update Client or
ignore and update RMj} FE
(Tf) FE
Query
Value
Update
Client
Client
Winter, 2004
CSS490 Fault Tolerance

16. Bayou System Committed Tentative To make a tentative update committed:
Primary C0 C1 C2 CN T0 T1 T2 T3 Tn
Tn+1 RM RM
To make a tentative update committed:
Perform a dependency check
Check conflicts
Check priority
Merge Procedure
Cancel tentative updates
Change tentative updates
Sent first
Sent later FE FE FE FE Tn T0 T1 T3
Client
Client
Client
Client
Secretary and other employees:
book 3pm
Executive: book 3pm
Winter, 2004
CSS490 Fault Tolerance

17. Coda File System Normal case: Read-any, write-all protocol
Whenever a client writes back its file, it increments the file version at each server.
Version[1,1,1] W
Version[2,2,2]
Network disconnection:
A client writes back its file to only available servers.
Version conflicts are detected and resolved automatically when network is reconnected W
Version[2,2,3]
Version[3,3,2]
Client disconnection:
A client caches as many files as possible (in hoard walking).
A client works in local if disconnected (in emulation mode).
A client writes back updated files to servers (in reintegration mode).
hoard
reintegration
emulation
Server 2
Server 3
Server 1
Winter, 2004
CSS490 Fault Tolerance

18. Paper Review by Students
ISIS System
Gossip Architecture
Bayou System
Coda
Winter, 2004
CSS490 Fault Tolerance