CSS490 Replication & Fault Tolerance Textbook Ch9 (p440 – 484) Instructor: Munehiro Fukuda These slides were compiled from the course textbook and the reference books. Hello! Everyone, My name is Shinya Kobayashi. Today, I am going to present our paper titled “Inter-Cluster Job Coordination Using Mobile Agents” on behalf of the first author, Munehiro Fukuda. Munehiro was hoping to show up and present the paper at AMS2001, however he got to wait in Japan until he will get an H1B visa. Since I received the presentation materials from him quite recently, please allow me to present this paper using this script. I can respond to your questions as far as I know, however you can also ask Munehiro by email. His email address is on the title page of our paper. (time 1:05) Winter, 2004 CSS490 Fault Tolerance

File Replication Concepts Difference between replication and caching A replica is associated with a server, whereas a cache with client. A replicate focuses on availability, while a cache on locality A replicate is more persistent than a cache is A cache is contingent upon a replica Advantages Increased availability/reliability Performance enhancement (response time and network traffic) Scalability and autonomous operation Requirements Naming: no need to be aware of multiple replicas. Consistency: data consistency among replicated files. Replication control: explicit v.s. implicit/lazy replication ACID: Atomicity, Consistency, Isolation, and Durability Winter, 2004 CSS490 Fault Tolerance

File Replication Basic Architectural Model Request: send a client request to a server. Coordination: deliver the request to each replica manger in some order. Execution: process a client request but not permanently commit it. Agreement: agree if the execution will be committed Response: respond to the front end Client Replica Manger Front End Replica Manger Client Front End Replica Manger Ex: DNS Web server Winter, 2004 CSS490 Fault Tolerance

Group Communication Group membership service Create and destroy a group. Add or withdraw a replica manager to/from a group. Detect a failure. Notify members of group membership changes. Provide clients with a group address. Message delivery Absolute ordering Consistent ordering Replica Manger Replica Manger Client Replica Manger Replica Manger group Winter, 2004 CSS490 Fault Tolerance

Absolute Ordering Linearizability Rule: Mi must be delivered before mj if Ti < Tj Implementation: A clock synchronized among machines A sliding time window used to commit message delivery whose timestamp is in this window. Example: Distributed simulation Drawback Too strict constraint No absolute synchronized clock No guarantee to catch all tardy messages Ti < Tj Ti mi Tj mi mj mj Winter, 2004 CSS490 Fault Tolerance

Consistent (Total) Ordering Sequential Consistency Rule: Messages received in the same order (regardless of their timestamp). Implementation: A message sent to a sequencer, assigned a sequence number, and finally multicast to receivers A message retrieved in incremental order at a receiver Example: Replicated database update Drawback: A centralized algorithm Ti < Tj Ti Tj mj mj mi mi Winter, 2004 CSS490 Fault Tolerance

Two-Phase Commit Protocol INIT WAIT ABORT COMMIT Commit Vote-request Vote-abort Global-abort Vote-commit Global-commit Coordinator INIT READY ABORT COMMIT Vote-request Vote-commit Global-commit Ack Global-abort Vote-abort Worker 1 INIT READY ABORT COMMIT Vote-request Vote-commit Global-commit Ack Global-abort Vote-abort Worker 2 Another possible cases: The coordinator didn’t receive all vote-commits. → Time out and send a global-abort. A worker didn’t receive a vote-request. → All workers eventually receive a global-abort. A worker didn’t receive a global-commit. → Time out and check the other work’s status. Winter, 2004 CSS490 Fault Tolerance

Multi-copy Update Problem Read-only replication Allow the replication of only immutable files. Primary backup replication Designate one copy as the primary copy and all the others as secondary copies. Active backup replication Access any or all of replicas Read-any-write-all protocol Available-copies protocol Quorum-based consensus Winter, 2004 CSS490 Fault Tolerance

Primary-Copy Replication Request: The front end sends a request to the primary replica. Coordination:. The primary takes the request atomically. Execution: The primary executes and stores the results. Agreement: The primary sends the updates to all the backups and receives an ask from them. Response: reply to the front end. Advantage: an easy implementation, linearizable, coping with n-1 crashes. Disadvantage: large overhead especially if the failing primary must be replaced with a backup. Client Replica Manger Front End Primary Backup Replica Manger Client Front End Replica Manger Backup Winter, 2004 CSS490 Fault Tolerance

Active Replication Request: The front end multicasts to all replicas. Coordination:. All replica take the request in the sequential order. Execution: Every replica executes the request. Agreement: No agreement needed. Response: Each replies to the front. Advantage: achieve sequential consistency, cope with (n/2 – 1) byzantine failures Disadvantage: no more linearizable Client Replica Manger Front End Replica Manger Client Front End Replica Manger Winter, 2004 CSS490 Fault Tolerance

Read-Any-Write-All Protocol Read from any one of them Read Lock any one of replicas for a read Write Lock all of replicas for a write Sequential consistency Intolerable for even 1 failing replica upon a write. Client Replica Manger Front End Write to all of them Replica Manger Client Front End Replica Manger Replica Manger Winter, 2004 CSS490 Fault Tolerance

Available-Copies Protocol Read Lock any one of replicas for a read Write Lock all available replicas for a write Recovering replica Bring itself up to date by coping from other servers before accepting any user request. Better availability Cannot cope with network partition. (Inconsistency in two sub-divided network groups) Read from any one of them Client Replica Manger Front End Write to all available replicats Replica Manger X Client Front End Replica Manger Replica Manger Winter, 2004 CSS490 Fault Tolerance

Quorum-Based Protocols #replicas in read quorum + #replicas in write quorum > n Read Retrieve the read quorum Select the one with the latest version. Perform a read on it Write Retrieve the write quorum. Find the latest version and increment it. Perform a write on the entire write quorum. If a sufficient number of replicas from read/write quorum, the operation must be aborted. Client Replica Manger Front End Read quorum Write quorum Read-any-write-all: r = 1, w = n Winter, 2004 CSS490 Fault Tolerance

ISIS System Process group: see page 4 of this ppt file Group view Reliable multicast Causal multicast: see pages 5 & 6 of MPI ppt file Atomic broadcast: see page 7 of this ppt file p1 p2 p3 p4 Joins the group multicast crashed Partially multicast messages must be discarded Multicast to available processes rejoins Winter, 2004 CSS490 Fault Tolerance

Gossip Architecture If (Tj > Tk) update RMk else discard the gossip message RMk Gossip RMi (Ti) RMj (Tj) Update, Tf Query, Tf Value, Ti Update id If (Tf < Ti) return value else { waits for RMi to be updated or query RMj/RMk} If (Tf > Tj) update RMj else { update Client or ignore and update RMj} FE (Tf) FE Query Value Update Client Client Winter, 2004 CSS490 Fault Tolerance

Bayou System Committed Tentative To make a tentative update committed: Primary C0 C1 C2 CN T0 T1 T2 T3 Tn Tn+1 RM RM To make a tentative update committed: Perform a dependency check Check conflicts Check priority Merge Procedure Cancel tentative updates Change tentative updates Sent first Sent later FE FE FE FE Tn T0 T1 T3 Client Client Client Client Secretary and other employees: book 3pm Executive: book 3pm Winter, 2004 CSS490 Fault Tolerance

Coda File System Normal case: Read-any, write-all protocol Whenever a client writes back its file, it increments the file version at each server. Version[1,1,1] W Version[2,2,2] Network disconnection: A client writes back its file to only available servers. Version conflicts are detected and resolved automatically when network is reconnected W Version[2,2,3] Version[3,3,2] Client disconnection: A client caches as many files as possible (in hoard walking). A client works in local if disconnected (in emulation mode). A client writes back updated files to servers (in reintegration mode). hoard reintegration emulation Server 2 Server 3 Server 1 Winter, 2004 CSS490 Fault Tolerance

Paper Review by Students ISIS System Gossip Architecture Bayou System Coda Winter, 2004 CSS490 Fault Tolerance