Global States and Checkpoints CS 271

Distributed Checkpoints and Rollback Recovery Fault tolerance is achieved by periodically using stable storage to save the processes’ states during the failure-free execution. Upon a failure, a failed process rolls back from one of its saved states, thereby reducing the amount of lost computation. Each of the saved states is called a checkpoint CS 271

Checkpoint based Recovery Uncoordinated checkpointing: Each process takes its checkpoints independently Coordinated checkpointing: Process coordinate their checkpoints in order to save a system-wide consistent state. Communication-induced checkpointing: It forces each process to take checkpoints based on information piggybacked on the application messages it receives from other processes. CS 271

Domino effect: example Recovery Line P0 m7 m2 m5 m0 m3 P1 m6 m4 m1 P2 Domino Effect: Cascaded rollback which causes the system to roll back to too far in the computation (even to the beginning), in spite of all the checkpoints CS 271

Global State Chandy and Lamport—TOCS 1985 Global state of a distributed system Local state of each process Messages sent but not received Many applications need the state of the system Failure recovery, distributed deadlock detection Detect stable properties. Problem: how can you figure out the state of a distributed system? Each process is independent Network does not have any processing power. Distributed snapshot: a consistent global state CS 271

Global State A consistent cut An inconsistent cut CS 271

Distributed Snapshot Algorithm Assume each process communicates with another process using unidirectional FIFO point-to-point channels (e.g, TCP connections) Any process can initiate the algorithm Checkpoint local state Send MARKER on every outgoing channel On receiving a first marker on a channel: Process checkpoints local state and Send markers on all outgoing channels, and save messages on all other channels. On receiving subsequent marker on a channel: stop saving messages for that channel Saved messages are the state of the channel CS 271

Distributed Snapshot A process finishes when It receives a marker on each incoming channel and processes them all State: local state plus state of all channels Send state to initiator Any process can initiate snapshot Multiple snapshots may be in progress Each is distinguished by tagging the marker with the initiator ID (and sequence number) CS 271

Snapshot Algorithm Example Organization of a process and channels for a distributed snapshot CS 271

Snapshot Algorithm Example Process Q receives a marker for first time and records local state Q records all incoming message Q receives a marker for its incoming channel and finishes recording the state of the incoming channel CS 271

Execution Example Sp0 Sp1 Sp2 Sp3 p m1 m2 m3 q Sq0 Sq1 Sq2 Sq3 CS 271

Execution Example q records state as Sq1 , sends marker to p Sp0 Sp1 CS 271

Execution Example p records state as Sp2, channel state as empty Sp0 q Sq0 Sq1 Sq2 Sq3 CS 271

Execution Example q records channel state as m3 Sp0 Sp1 Sp2 Sp3 p m1 Sq0 Sq1 Sq2 Sq3 CS 271

Execution Example Recorded Global State = ((Sp2, Sq1), (0,m3) ) Sp0 CS 271

Take home Message (Snapshot and global states) General solution for global state detection. Causality based detection of stable properties. Simple efficient protocol, uses Markers and FIFO properties. Don’t forget Channel States. Foundation for Distributed Checkpointing and Rollback Recovery CS 271