1. Abstractions for Fault Tolerance
Kris Malfettone, Adrian Dumchus

2. Fault Tolerant System Difficult to design and understand
Must maintain control over failure free behavior as well as failure behavior
Fault Tolerance
Behavior remains well defined when components fail
Masks component failures from user

3. Service A service is a system behavior perceived by the user
Current Service State is a summary of past behavior
Services with same set of operations and allowed behaviors are of same type
Service Implementation carries out operations for a service, consists of one or more servers

4. Server
Encapsulates private state data using set of procedures (instructions or methods)
Allows user to access and change a server’s state
Servers are local or centralized - services can be distributed

5. “Depends On” relation
Servers implement services using services implemented by other servers
Service U depends on R if R is used in validating server u implementing U is correct
Server u is a user (client) of r
r is a resource of u
Resources may depend on other resources, etc.

6. Failure Classification
Server is correct if input results in behavior consistent with service specification
Failure occurs when server does not behave according to specification

7. Types of Failures
Omission Failure - does not provide response to an input
Timing Failure - response is correct, but either early or late (performance failure)
Response Failure - responds incorrectly
value failure
state transition failure

8. Types of Failures (cont.)
Crash Failure - after first omission failure, all input results in omission until restart
amnesia crash - restarts in predefined init state independent of pre-crash inputs
partial-amnesia crash - part of state is same as before crash, rest is init
pause-crash - restart in state prior to crash
halting-crash- never restarts

9. Failure Semantics
Specification should, in addition to including failure-free semantics, include all likely failure behaviors
The more failure behaviors allowed, the weaker the failure class
Arbitrary failure semantics – weakest, any failure behavior can occur for a service
The stronger the failure semantics, the more expensive and complex the server

10. Failure Masking Server failures can be masked from users two ways:
By higher level servers
By using server groups

11. Hierarchical masking
Failure at lower level may result in different type of failure at higher level
Preferable for servers to have semantics stronger than arbitrary (omission, crash, or performance)
Exception handling – way to propagate failure info across abstraction levels and to mask lower level failures.
If masking fails at one level, failure info is propagated to next level where attempts to mask are continued

12. Group Masking
Ensures services remain available using a group of redundant, physically independent servers
If some fail, remaining servers provide the service and mask the original failure

13. Understanding Fault-Tolerance
Basic goals of fault-tolerance:
Hide component failures as long as enough redundancy is available
Provide manageable failure behavior so that users can easily recover from failures

14. Commonly used Fault-Tolerant Services
Processors with Crash failure semantics
Most processors provide amnesia-crash
Two main methods:
Error detecting codes
More complex, less reliable, increased testing and design costs
Duplication and matching
Provides better approximation of crash failure semantics

15. Commonly used Fault-Tolerant Services (cont.)
Stable storage with omission failure semantics
Error detecting/correcting codes
Ensure volatile and persistent storage
Ensure read omission failure semantics assuming no operating system crash occurs
When operating system crashes, common method is using mirrored persistent storage servers
Failure by one server is masked by another server
If masking attempt fails, omission failure reported
Higher level crashes (files, databases, etc.) use logging and recovery algorithms

16. Commonly used Fault-Tolerant Services (cont.)
Restartable servers
Mask failures of non-replicated servers by restarting them when they fail
Clients resend service requests until eventually succeed
Requires servers to periodically checkpoint their state
When server is restarted after a crash, must re-execute all logged service requests since last checkpoint

17. Commonly used Fault-Tolerant Services (cont.)
Point-to-point communication services
Error correcting codes used at lowest, physical layer for masking
Negative or positive acknoweledgements used at higher levels for masking

18. Commonly used Fault-Tolerant Services (cont.)
Distributed storage services
Clients may see omission and performance failures, but not more complex response failures
Atomic commit problem – goal is to ensure that sequence of updates is either made permanent or entirely aborted
To ensure failure masking, uses dually persistent storages devices and hierarchical techniques to use alternate paths should primary path fail
Communication failures are either masked by hierarchical techniques (retransmissions), or result in omission failures.

19. Commonly used Fault-Tolerant Services (cont.)
Restartable arbitrary distributed services
Difficulty when checkpointing in capturing global state of all local server states, as well as the state of communications between the servers

20. Commonly used Fault-Tolerant Services (cont.)
Replicated storage and servers
Use of server groups raises issues:
How to maintain consistent state
Must agree on group membership
Must agree on order of service requests and state updates within group (atomic broadcast problem)
How servers in group should communicate
How to ensure required number of servers are running

21. In Conclusion
Fundamental concepts include notions of service, server, and the ‘depends upon’ relation
Concepts which capture the goals of fault-tolerant computing – failure semantics, hierarchical failure masking, and group failure masking
Mask failures when possible
Ensure system has clearly specified, manageable failure semantics when masking not possible