CAP THEOREM Have the rules really changed?

Tolerance to Network Partition Consistency Availability Tolerance to Network Partition Basics of CAP Theorem

Basics of CAP Any distributed system can satisfy any two of the three properties at the same time. Consistency - Having single up-to-date copy of the data at the same time with all the nodes Availability - The system would continue to operate as expected even with node failures Partition-tolerance: The system would continue to operate as expected despite network or message failures

Inconsistent Server 1 Server 2 Server 1 Server 2 Server 1 Server 2 x x y x y x y x y x write written read x z z z z z Client Client Client Client Client

Availability Client, Server Example Server 1 Server 2 Client sends a request for an operation to the server The server must eventually respond to the client z Client

Partition Tolerance Client, Server Example Server 1 Server 2 This system is not partition tolerant as S1 is updated S2 z Client

WHY is it important?? A proper understanding of CAP theorem is essential to making decisions about the future of distributed database design Misunderstanding can lead to erroneous or inappropriate design choices The notion of having at most of “2 of these 3” is no longer strict as before Choosing to have either Availability or Consistency is difficult

CAP - A popular misconception The CAP categorizations - AP, CP, CA say very little about the actual behaviour of the distributed system. How about CA? Can a distributed system really be not tolerant of partitions?

CAP is not binary Out of the three, none of the properties are binary Availability can be scaled from zero to hundred percent Consistency also has many variances between being consistent and inconsistent Though not well investigated, but network partitioning also has different degrees CAP is not binary

Facebook Example Suppose your friend finds something interesting on Facebook and shares it on your wall asking, you to check it out. Where is it? But when you login into your account you see nothing.

BUT WHY?? Facebook Example Contd. Eventually after waiting for a minute, when you go back and try to check it out again, you find the story. The actual reason – Facebook uses an eventual consistent model over the strong consistent. BUT WHY?? It is non-trivial for them to store the huge amount of data collected/generated, efficiently and reliably as they have billion of users. Using eventual consistent model offers to reduce the load and in turn improve availability.

How to Decide?

CAP confusion and Partition Decision Contrary to what people believe or understand, CAP is not a one time decision. No need to choose between consistency or availability beforehand. A system can guarantee both availability and consistency when fully connected. This decision can be left to answer up until the moment when the partition happens. It can be changed multiple times based on different actions and data.

1 2 Different Options for Partition Decision Decrease availability by cancelling the operation 1 Risk inconsistency and proceed with the operation 2

Partition Recovery

Partition and its effects At some point, communication resumes and the partition ends. A log (history) is being is being kept during the partition mode. After the partition, system knows the state and history of both sides. System can deduce operations that actually violated invariants.

The state on both sides of the partition must be made consistent Partition Recovery When the partition finally ends, a system is required to perform a so called Partition Recovery Step. Two hard problems during recovery are: The state on both sides of the partition must be made consistent 1 Undo any mistakes 2

More on Partition Recovery It is easier to fix by starting from the state at the time of partition. Rolling forward with both sets of operations is the way forward. Some systems can merge conflicts but some can’t. Designers can choose to avoid the use of certain operations at the time of partition. System can automatically merge during recovery.

Achieving Partition Recovery Using CRDTs - commutative replicated data types 1 Using compensating transactions 2

CRDTs CRDTs - class of data structures that help in maintaining commutative operations and data merging. If operations on your data types are both commutative and idempotent Nodes in a distributed system are able to arbitrary leave and rejoin a network They can send duplicate messages at any point, and Maintain eventual consistency in the whole process.

CRDTs Continued It doesn’t matter with what order the operations on the data types are performed. Even if you perform an operation multiple times it has no effect on the state In the end there is a guarantee that all the nodes achieve the same state after applying all pending operations.

Compensating Transactions Second step towards Partition Recovery Fixing any inconsistencies after a partition is resolved Mistakes made have to be corrected So-called externalized operations (actions that let out information to other systems or users that are not in control of the original system) need to be analyzed Checked for errors and compensated.

Compensating Transactions Various ways to fix invariants - Last writer wins – not the smartest, as ignores some updates Approach that can merge operations Human escalation

Example - Automatic Merging Airplane overbooking Airplane booking system allowing booking of seats in partition mode Would work out if the flight is not full If there are too many customers, some would loose their seats Having the history makes it easier to resolve the issue Customer Service can compensate in some way

Brewer’s conclusion System designers should be aware of their system closely Look at each invariant and design the system to react on the network partition He sums it up very concisely – “System Designers should not blindly sacrifice consistency or availability [...]. They can optimize both properties through careful management of invariants during partitions.”

My conclusion The publication is interesting specially coming from the inventor of CAP theorem himself Article is very good for newbies learning about the basics of CAP theorem Manages to bring in interesting thoughts and mindsets for more experienced developers Things I liked: Lays down great emphasis on the fact that CAP allows for very fine grained decisions Opposes my initial understanding of it being a binary formulation of the theorem

Shortcomings Not any groundbreaking new theory Just a summation of possibilities and technologies that have already been researched over the last decade Everything boils down to a unified model

Thank You!