1. Failure recovery and Checkpointing in Distributed Systems
CS 455: Introduction to Distributed Systems Computer Science Department, Colorado State University.
- Daniel Sullivan, Pasha Volchak, Tyler Decker 1

2. Why is this problem important?
High demand services need to be up 24/7 (Netflix, Facebook, Google, Amazon)
Failures are complex as it introduces problems of states, checkpoints and rollback, data loss, lost of passing messages, etc.
One system goes down so can the rest
Can cost money and lives 2

3. Problem characterization
Loss of messages including several types
Lost, delayed, orphaned or duplicate
Loss of state = very expensive
Storage/replication of data
Find the balance of keeping a file in good supply (Bottlenecking, Data loss, System failure)
Checkpointing (How much vs how little and efficiency) 3

4. Trade-off space for solutions in this area
TCP vs UDP for message passing (Network bandwidth vs certainty)
System heartbeats and checkpointing (resource cost, loss of progress)
Speed and efficiency vs risk of state
Task completion vs self sustainability
Financial cost vs new software and hardware 4

5. Dominant approaches to the problem (1)
Data replication
Scalable checkpointing systems (data­aware aggregation and compression)
Use of cloud resources (resource brokering, scheduling)
The election algorithm
Leader election
Group membership
Self stabilization 5

6. Dominant approaches to the problem (2)
Chaos Monkey
Netflix and Stack exchange
System scanning
Learning algorithms 6

7. Insights Gleaned Test the limits of your system!
practicing fault tolerance = actual fault tolerance
Adapt resources from solutions in other fields to your current problem
use of compression combined with use of checkpointing
cloud resources
Better safe than sorry (use replication !!)
Be organized with use of smart algorithms 7

8. What the problem space in the future would look like
More complexity and size of systems
Increase of access globally (device count, more people)
Technological progress
Increase of demand
Resource use
Data storage
Network bandwidth
Increasing emphasis in security 8

9. Trade-off space and solutions in the future
Chaos monkey/Limit testing
High demand of attention vs sustainability
Combining resources from multiple areas
Cutting edge technologies (McR Engine, compression, cloud resources)
This can be expensive
Hierarchical layer management
Efficiency of data storage/replication 9

10. Questions? 10