1. Operating System Reliability Andy Wang COP 5611 Advanced Operating Systems

2. Some Axioms Some simple systems, designed from scratch, sometimes work A complex system that works is invariably found to have evolved from a simple system that works A complex system, designed from scratch never works

3. Failure-Mode Theorems Complex system usually operate in failure mode A system should have safe behaviors when encountering failures When a “fail-safe” system fails, it fails by failing to fail safe

4. Some definitions Failure of a system occurs when the system does not perform its services in the manner specified  Sometimes failures are subtle (e.g., performance fault) Fault is anomalous physical condition  Includes system specification/implementation mistakes Error is part of system state that differs from its intended value

5. Classification of Failures Process failures System failures Secondary storage failures Communication medium failures

6. Process Failures Examples  Computation results in incorrect outcome  System state deviates from specification  Process fails to progress Errors leading to failure  Deadlock, timeout, protection violation  Bad input, consistency violation Ignoring malicious behavior

7. System Failures Processor fails to execute  Software error, hardware error (CPU, bus, etc.) Fail-stop behavior assumed Failure types  Amnesia  Partial-amnesia  Pause  Halting

8. Secondary Storage Failures Stored data inaccessible  Parity error  Head crash  Contaminated medium Reconstructable from archive + log, maybe Mirrored disks (independent failure mode)

9. Communication Medium Failures Site can’t communicate with another site Causes  Switching node failure Hardware failure Software failure Congestion  Link failure Hardware  Implementation failure Network partitions can result

10. Recovery Restart process/processor Reclaim resources Undo/finish incomplete transactions Concurrency makes things harder

11. Forward Error Recovery Goal: To restore system from erroneous state to error-free state If nature of error is completely known  Remove error from state  Proceed with execution from error-free state Rarely possible to do

12. Backward Error Recovery When error source unknown  Restore state to previous error-free state; restart  Independent of fault, errors causing fault  Problems Performance penalty No guarantee fault will not reoccur Possible unrecoverable component of state Recovery point: state used to replace error

13. Backward Error Recovery Basic approaches  Operation-based Logs  Update-in-place  Write-ahead-log  State-based

14. Update-in-Place Every update to object also records the log  Name of object  Old and new states of object Recoverable update operation implements as  Do, undo, redo operations

15. Write-ahead Log Update-in-place has problem if crash occurs between update and log recorded to stable storage Update object only after undo log recorded Before committing updates, record both redo and undo logs Expensive to write log to stable storage

16. State-Based Recovery Save entire process state at recovery point  Recovery point called checkpoint  Rolling back process: restoring to checkpoint  Tradeoff: frequent checkpoints vs. completion delay Shadow pages  Save unmodified page copy on stable storage  Update only volatile copy; discard on rollback

17. Concurrent Systems Recovery Rollback issues  Orphan messages  Domino effect  Lost messages  Livelocks

18. Orphan Messages x1x2 X[[ y1 m y2 Y[[ z1z2 Z[[ [ recovery point

19. Domino Effect Suppose Y rolls back to y2  m is orphan message  Process Y must rollback to y1 Suppose Z rolls back to z2  Y rolls back to y1  Forcing Z to roll back to z1

20. Lost Messages x1 X[ m z1 Z[ failure [ recovery point

21. Live Locks x1 X[ z1 Z[ repeated failure [ recovery point

22. Concurrent Recovery Coordination required at either time of establishing checkpoints Beginning of recovery

23. Checkpoint Assumptions Communication via messages Unreliable FIFO channels  Higher-level end-to-end protocols assumed  Subsumes rollback-caused message loss No network partitions from communication failures

24. Checkpoint Algorithm Concepts Permanent and tentative checkpoints  Saved on stable storage  Permanent: part of known consistent global checkpoint  Tentative: until successful termination of checkpoint algorithm  Rolls back only to permanent checkpoints

25. Synchronous Checkpoint Algorithms Two-phase commit Problems:  Message overhead for synchronizations  Synchronization delays  Costly when failures are rare

26. Asynchronous Checkpointing Local checkpoints taken independently Log all incoming messages on stable storage  Minimizes undone computation  Allows reprocessing of messages after rollback

27. Asynchronous Checkpointing Assumptions Assumptions  Reliable FIFO communication channels  Infinite buffers  Event-driven computation A process idle until message received Processes message and change state Sends zero or more messages Can identify each event with monotonically increasing counter

28. Event-Driven Computation x1x2 X y1y2 Y z1z2 Z

29. Asynchronous Checkpointing Basic idea  Save states, messages sent at each event  Volatile logging  Each processor notes number of messages sent to others, and received from others  Use counters to determine orphan messages

30. Summary Failures caused by errors Can remove errors by forward/backward error recovery Backward error-recovery more costly, more general Synchronous checkpoints helpful, costly Asynchronous checkpoints messier, domino effects