© Chinese University, CSE Dept. Distributed Systems / Distributed Systems Topic 12: Recovery and Fault Tolerance Computer Science & Engineering Department The Chinese University of Hong Kong

© Chinese University, CSE Dept. Distributed Systems / Outline 1 Introduction 2 Transaction Recovery 3 Fault Tolerance 4 Hierarchical and Group Masking of Faults 5 CORBA Fault Tolerance Service 6 Summary

© Chinese University, CSE Dept. Distributed Systems / Introduction  Fault tolerance: the survival attribute of systems.  Fault-tolerant applications: –transaction based –process control  Recovery aspects of distributed transactions.  The design of real time services. –Fail-stop vs Byzantine failure.  Masking failures in a service.  CORBA fault tolerance service.

© Chinese University, CSE Dept. Distributed Systems / Basic Approaches  Fault Detection: –Push Model: Server objects send heartbeat messages to Fault Manager. –Pull Model: Fault Manager polls (or pings) server objects through their is_alive() interface.  Data Recovery: –Checkpoint and rollback: Save the server object states. Roll back to checkpointed states at recovery. –Message logging and replay: Log all messages. Replay them at recovery.

© Chinese University, CSE Dept. Distributed Systems / Transaction Recovery  Recovery concerns data durability (permanent and volatile data) and failure atomicity.  A server keeps data in volatile memory and records committed data in a recovery file.  Recovery manager –save data items in permanent storage –Restore the server’s data items after a crash –reorganize the recovery file for better performance –reclaim storage space (in the recovery file)

© Chinese University, CSE Dept. Distributed Systems / Intentions List  An intentions list of a server is a list of data item names and values altered by a transaction.  The server uses the intentions list when a transaction commits or aborts.  When a server prepares to commit, it must have saved the intentions list in its recovery file.  The recovery files contain sufficient information to ensure the transaction is committed by all the servers.

© Chinese University, CSE Dept. Distributed Systems / Type of entryDescription of contents of entry Data itemA value of a data item Transaction statusTransaction identifier, transaction status (prepared, committed, aborted) and others Intentions listTransaction identifier and a sequence of intentions, each of which consists of, 2 Entries in Recovery File

© Chinese University, CSE Dept. Distributed Systems / Logging  A log contains history of all the transactions performed by a server.  The recovery file contains a recent snapshot of the values of all the data items in the server followed by a history of transactions.  When a server is prepared to commit, the recover manager appends all the data items in its intentions list to the recovery file.  The recovery manager associates a unique identifier with each data item.

© Chinese University, CSE Dept. Distributed Systems / Log for Banking Service

© Chinese University, CSE Dept. Distributed Systems / Recovery by Logging  Recovery of data items –Recovery manager is responsible for restoring the server’s data items. –The most recent information is at the end of the log. –A recovery manager gets corresponding intentions list from the recovery file.  Reorganizing the recovery file –Checkpointing: the process of writing the current committed values (checkpoint) to a new recovery file. –Can be done periodically or right after recovery.

© Chinese University, CSE Dept. Distributed Systems / Shadow Versions  Shadow versions technique uses a map to locate versions of the server’s data items in a file called a version store.  The versions written by each transaction are shadows of the previous committed versions.  When prepared to commit, any changed data are appended to the version store.  When committing, a new map is made. When complete, new map replaces the old map.

© Chinese University, CSE Dept. Distributed Systems / Shadow Versions Example

© Chinese University, CSE Dept. Distributed Systems / Log and 2PC

© Chinese University, CSE Dept. Distributed Systems / Recovery of 2PC

© Chinese University, CSE Dept. Distributed Systems / Fault Tolerance  Two contrasting points on distributed systems: –The operation of a service depends on the correct operation of other services. –Joint execution of a set of servers is less likely to fail than any one of the individual components.  Designers of a service should specify its correct behavior and the way it may fail  Failure semantics: a description of the ways a service may fail. Can be used for its clients to mask its failures.

© Chinese University, CSE Dept. Distributed Systems / Class of failureSubclassDescription Omission failure A server omits to respond to a request Response failure Server responds incorrectly to a request Value failureReturn wrong value State transitionHas wrong effect on resources (for failureexample, sets wrong values in data items) Timing failure Response not within a specified time interval Crash failure Repeated omission failure: a server repeatedly fails to respond to requests until it is restarted Amnesia-crashA server starts in its initial state, having forgotten its state at the time of the crash Pause-crashA server restarts in the state before the crash Halting-crashServer never restarts 3 Characteristics of Faults

© Chinese University, CSE Dept. Distributed Systems / Fail-Stop vs Byzantine Failures  A fail-stop server is one that fails cleanly. That is, it either functions, or else it crashes.  Byzantine failure behavior is used to describe the worse possible failure semantics of a server: it fails maliciously or arbitrarily.  Byzantine agreement is intended for correct behaviors within response time requirement in the presence of faulty hardware.  It depends on if messages can be authenticated.

© Chinese University, CSE Dept. Distributed Systems / Byzantine Generals

© Chinese University, CSE Dept. Distributed Systems / Byzantine Agreement Algorithms  Byzantine agreement algorithms send more messages and use more active servers.  When messages can be authenticated, 2N+1 servers are required to tolerate N bad servers.  When messages cannot be authenticated, 3N+1 servers are required.  With enough good servers, solutions require O(N 2 ) messages with constant delay time.  Fortunately, the good news is...

© Chinese University, CSE Dept. Distributed Systems / Hierarchical Masking of Faults  We describe two approaches to masking faults: hierarchical failure masking and group failure masking.  In hierarchical failure masking, a server of higher level tries to mask faults at lower-level.  When a lower-level failure cannot be masked, it is converted to a higher level exception.  Example: Server crash is masked in RR protocol by raising an exception to the client.

© Chinese University, CSE Dept. Distributed Systems / Group Failure Masking  A service can be made fault tolerant by implementing it by a group of servers.  A group is t-fault tolerant if it can tolerate up to t member failures.  For fail-stop failures, t+1 servers are needed.  For Byzantine failures, 2t+1 servers needed.  To ensure correctness, the server program must be deterministic, and each operation must be atomic w.r.t. other operations.

© Chinese University, CSE Dept. Distributed Systems / Group Failure Masking  A group can be closely synchronized or loosely synchronized.  In a closely synchronized group of servers: –All members execute requests immediately. –Server programs are both deterministic and atomic. –Suitable for real time system and Byzantine failures.  In a loosely synchronized group of servers: –One server (primary) performs requests, others (backup) log the requests and take over if needed. –Requires less resource but takes longer to recover.

© Chinese University, CSE Dept. Distributed Systems / CORBA Fault Tolerance Service Application Objects CORBA facilities CORBA services Object Request Broker Fault Tolerance

© Chinese University, CSE Dept. Distributed Systems / Outline of Fault Tolerant CORBA  Fault Tolerance Properties  Replication Styles, Membership Styles, Consistency Styles, Fault Monitoring Styles  Infrastructure-Controlled and Application-Controlled  Object Group References and Alternative Destinations  At-Most-Once Invocation (repeated requests detected)  Fault Detection and Notification  Checkpointing and Logging

© Chinese University, CSE Dept. Distributed Systems / Architectural Overview

© Chinese University, CSE Dept. Distributed Systems / Fault Detectors, FaultNotifier, Fault Analyzer, and ReplicationManager

© Chinese University, CSE Dept. Distributed Systems / Property Management interface PropertyManager { void set_default_properties(in Properties props) raises (InvalidProperty,UnsupportedProperty); Properties get_default_properties(); void remove_default_properties(in Properties props) raises (InvalidProperty,UnsupportedProperty); void set_type_properties(in TypeId type_id, in Properties overrides) raises (InvalidProperty,UnsupportedProperty); Properties get_type_properties(in TypeId type_id); void remove_type_properties(in TypeId type_id, in Properties props) raises (InvalidProperty, UnsupportedProperty); void set_properties_dynamically(in ObjectGroup object_group, in Properties overrides) raises(ObjectGroupNotFound, InvalidProperty, UnsupportedProperty); Properties get_properties(in ObjectGroup object_group) raises(ObjectGroupNotFound); };

© Chinese University, CSE Dept. Distributed Systems / ObjectGroupManager & GenericFactory // Specification of ObjectGroupManager Interface // which ReplicationManager Inherits interface ObjectGroupManager { ObjectGroup create_member(in ObjectGroup object_group, in Location the_location, in TypeId type_id … ) ObjectGroup add_member(in ObjectGroup object_group, in Location the_location, in Object member); ObjectGroup remove_member(in ObjectGroup object_group,in Location the_location); ObjectGroup set_primary_member(in ObjectGroup object_group,in Location the_location); Locations locations_of_members(in ObjectGroup object_group); ObjectGroup get_object_group_ref(in ObjectGroup object_group); ObjectGroupId get_object_group_id(in ObjectGroup object_group); }; // Specification of GenericFactory Interface // which ReplicationManager Inherits and Application Objects Implement interface GenericFactory { typedef unsigned long long FactoryCreationId; Object create_object(in TypeId type_id, in Criteria the_criteria, out FactoryCreationId factory_creation_id); void delete_object(in FactoryCreationId factory_creation_id); };

© Chinese University, CSE Dept. Distributed Systems / Replication Management // Specification of ReplicationManager Interface interface ReplicationManager : PropertyManager, ObjectGroupManager, GenericFactory { void register_fault_notifier(in FaultNotifier fault_notifier); FaultNotifier get_fault_notifier() raises (InterfaceNotFound); };

© Chinese University, CSE Dept. Distributed Systems / Logging and Recovery // Specification of Checkpointable Interface // which Updateable and Application Objects Inherit interface Checkpointable { State get_state() raises(NoStateAvailable); void set_state(in State s) raises(InvalidState); }; // Specification of Updateable Interface // which Application Objects Inherit interface Updateable : Checkpointable { State get_update() raises(NoUpdateAvailable); void set_update(in State s) raises(InvalidUpdate); };

© Chinese University, CSE Dept. Distributed Systems / Fault Detection and Notification // Specification of PullMonitorable Interface which Application Objects Inherit interface PullMonitorable { boolean is_alive(); }; // Specification of FaultNotifier Interface interface FaultNotifier { typedef string ConsumerId; void push_structured_fault(in CosNotification::StructuredEvent event); CosNotifyFilter::Filter create_subscription_filter (in string constraint_grammar) raises (CosNotifyFilter::InvalidGrammar); ConsumerId connect_structured_fault_consumer( in CosNotifyComm::StructuredPushConsumer push_consumer, in CosNotifyFilter::Filter filter) ; void disconnect_consumer( in ConsumerId connection) raises(CosEventComm::Disconnected); … };

© Chinese University, CSE Dept. Distributed Systems / Summary  Transaction recovery –long-life application and data integrity –atomic commit protocol is the key –checkpoints and logging in a recovery file  Fault tolerance –real-time application –importance of fault semantics –primary-backup server for fail-stop failures –closely synchronized group for Byzantine failures  Emerging CORBA Fault Tolerance Service