1. 95-702 Distributed Systems Learning Objectives
Understand locks and be able to recognize when locks are required
Be aware of deadlocks and deadlock mitigation
Understand the importance of transaction processing systems and a TP system’s multi-tiered architecture
Be able to describe two phase locking (2PL) for concurrency control
Be able to describe the two phase commit protocol (2PC) for consensus
Understand how system availability trades off with system consistency as described by the CAP theorem
Transactions

2. Transaction Processing (TP) Systems
Historically, one of the first was American Airlines SABRE – Semi- Automated Business Research Environment - 83,000 transactions per day (1960’s)
Became IBM’s Airline Control Program
Became IBM’s Transaction Processing Facility (TPF)
Many such modern systems exist:
Oracle Tuxedo (Thousands of transactions per second)
IBM’s Customer Information Control System (CICS)
Most databases and messaging systems provide for
transactional support
JEE and Microsoft .NET both provide extensive capabilities
for creating and deploying TP applications
Transactions

3. TP System Architecture
Front end
Program takes requests from the user device
Request Controller
selects the
proper transaction to run
The Transaction Server executes the required activities
User device
This represents a multi-tiered TP application.
The user-device might be a gas pump or
retail sales terminal or browser.
Database
Adapted From "Principles Of Transaction Processing" Bernstein and Newcomer

4. Transactions (ACID)
Atomic: All or nothing. No intermediate states are visible. No possibility that only part of the transaction ran. If a transaction fails or aborts prior to committing, the TP system will undo the effects of any updates (will recover). We either commit or abort the entire process. Checkpointing and Logging and recoverable objects can be used to ensure a transaction is atomic with respect to failures.
Consistent: system invariants preserved, e.g., if there were n dollars in a bank before a transfer transaction then there will be n dollars in the bank after the transfer. This is largely in the hands of the application programmer.
Isolated: Two transactions do not interfere with each other. They appear as serial executions. This is the case even though transactions may run concurrently. Locking is often used to prevent one transaction from interfering with another.
Durable: The commit causes a permanent change to stable storage. This property may be obtained with log-based recovery algorithms. If there has been a commit but updates have not yet been completed due to a crash, the logs will hold the necessary information on recovery.
Transactions
Transactions
Transactions 4 4

5. Assume concurrent visits
private double balance;
public synchronized void deposit(double amount) throws
RemoteException {
balance = balance + amount; }
public synchronized void withdraw(double amount) throws
balance = balance – amount;
This is all that is required for many applications. But
TP middleware must do much more.
If one thread invokes
a method it acquires a
lock. Another thread
will be blocked until
the lock is released.
What happens
if we don’t
synchronize?
Transactions

6. Communicating Threads (1)
Consider a shared queue and two operations: synchronized first() { if Q is empty return false else remove and return front } synchronized append() { add to rear } Is this sufficient? No. If the queue is empty the client of first() will have to poll on the method. What’s wrong with polling? It is also potentially unfair. Why?
Transactions

7. Communicating Threads (2)
Consider again the shared queue and two
operations:
synchronized first() {
if queue is empty call wait()
remove from front }
synchronized append() {
adds to rear
call notify()
When threads can synchronize
their actions on an object by means
of wait and notify, the server holds
on to requests that cannot
immediately be satisfied and the
client waits for a reply until
another client has produced
whatever they need.
Note that both methods are
synchronized. Only one thread at
a time is allowed in.
This is a simple example.
Wait/notify gets tricky fast.
Transactions

8. Back to Transactions
A client may require that a sequence of separate requests to a single server be isolated and atomic.
- Isolated => Free from interference from other
concurrent clients.
- Atomic => Either all of the operations complete
successfully or they have no effect at all in the presence
of server crashes.
- We also want serializability => If two transactions T1
and T2 are running, we want it to appear as if T1 was
followed by T2 or T2 was followed by T1.
- But, interleaving may have occurred (we like interleaving for
performance reasons).
Transactions

9. Assume each operation on the server is synchronized - Happy Case
Client 1 Transaction T; a.withdraw(100); b.deposit(100); c.withdraw(200); b.deposit(200);
Client 2 Transaction W;
total = x.getBalance();
total = total +
y.getBalance();
z.getBalance();
Suppose both run
to completion (no
partial execution)
=> atomic.
Why are we isolated?
Transactions
Transactions 9

10. Assume each operation on the server is synchronized
Client 1 Transaction T; a.withdraw(100); b.deposit(100); c.withdraw(200); b.deposit(200);
Client 2 Transaction W;
total = a.getBalance();
total = total +
b.getBalance();
c.getBalance();
Suppose both run
to completion (no
partial execution)
=> atomic.
Are we isolated?
Transactions

11. Assume each operation on the server is synchronized
Client 1 Transaction T; a.withdraw(100); b.deposit(100); c.withdraw(200); b.deposit(200);
Client 2 Transaction W;
total = a.getBalance();
total = total +
b.getBalance();
c.getBalance();
Inconsistent retrieval!
Transactions

12. Assume each operation on the server is synchronized
Client 1 Transaction T; bal = b.getBalance(); b.setBalance(bal*1.1);
Suppose both run
to completion with
no partial
execution => Atomic.
Client 2 Transaction W;
bal = b.getBalance();
b.setBalance(bal*1.1);
But are we isolated?
Transactions

13. Assume each operation on the server is synchronized
Client 1 Transaction T; bal = b.getBalance() b.setBalance(bal*1.1);
Client 2 Transaction W;
bal = b.getBalance();
b.setBalance(bal*1.1);
Lost Update!
Transactions

14. Assume each operation on the server is synchronized
Transaction T; a.withdraw(100); b.deposit(100); c.withdraw(200); b.deposit(200);
The aim of any server that
supports transactions is to
maximize concurrency. So,
transactions are allowed to execute
concurrently if they would have
the same effect as serial execution.
Locking is the most popular
mechanism to achieve transaction
Isolation.
Each transaction is created
and managed by a coordinator.
Transactions

15. Interacting with a coordinator
Transaction T tid = openTransaction(); a.withdraw(tid,100); b.deposit(tid,100); c.withdraw(tid,200); b.deposit(tid,200); closeTransaction(tid) or abortTransaction(tid)
Coordinator Interface:
openTransaction() -> transID
closeTransaction(transID) ->
commit or abort
abortTransaction(TransID)
Transactions

16. Serially Equivalent
For two transactions to be serially equivalent, it is necessary and sufficient that all pairs of conflicting operations of the two transactions be executed in the same order at all of the objects they both access. (Coulouris)
Let r1(x) mean that transaction 1 reads x.
Let w2(x) mean that transaction 2 writes x.
r1(x) r2(x) do not conflict (may be ordered either way).
r1(x) w2(x) do conflict (order matters)
w1(x) w2(x) do conflict (order matters)
Transactions

17. Locking to Attain Serializability
With locks, each transaction reserves access to the data it uses.
There are read locks rL1(x) and write locks wL2(x).
Before reading, a read lock is set. Before writing, a write lock is set.
A transaction can obtain a lock only if no other transaction has a conflicting lock on the same data item.
Locks may be removed with wU1(y) or rU1(x)
In the next two slides, we return to the earlier examples and apply this locking scheme. They become serially equivalent.
Transactions

18. total = a.getBalance(); Get a read lock on b total = total +
Allow either one to run first - Two Phase Locking for Concurrency Control
Client 1 Transaction T; Get write lock on a a.withdraw(100); Get write lock on b b.deposit(100); Get write lock on c c.withdraw(200); b.deposit(200); Unlock a,b,c
Client 2 Transaction W;
Get a read lock on a
total = a.getBalance();
Get a read lock on b
total = total +
b.getBalance();
Get a read lock on c
c.getBalance();
Unlock a,b,c
Transactions
Transactions 18

19. Allow either one to run first - Two Phase Locking for Concurrency Control
Client 1 Transaction T; Get a read lock on b bal = b.getBalance() Upgrade to write b.setBalance(bal*1.1); Unlock b
Client 2 Transaction W;
Get a read lock on b
bal = b.getBalance();
Upgrade to write lock
b.setBalance(bal*1.1);
Unlock b
Transactions
Transactions 19

20. Locking is not enough Transaction T1 Transaction T2 rL1(x) r1(x)
rU1(x)
wL1(y)
w1(y)
wU1(y)
rL2(y)
r2(y)
wL2(x)
w2(x)
rU2(y)
wU2(x)
Transactions

21. What pairs are in conflict?
Transaction T1 Transaction T2
rL1(x)
r1(x)
rU1(x)
wL1(y)
w1(y)
wU1(y)
rL2(y)
r2(y)
wL2(x)
w2(x)
rU2(y)
wU2(x)
“all pairs of conflicting operations of the two
transactions be executed in the same order”
Coulouris
Transactions

22. What pairs are in conflict?
Transaction T1 Transaction T2
rL1(x)
r1(x)
rU1(x)
wL1(y)
w1(y)
wU1(y)
rL2(y)
r2(y)
wL2(x)
w2(x)
rU2(y)
wU2(x)
To be serially equivalent:
If r1(x) occurs before w2(x)
then w1(y) must
occur before r2(y) and
If r1(x) occurs after w2(x)
occur after r2(y).
Transactions

23. Locking is not enough Transaction T1 Transaction T2 rL1(x) r1(x)
rU1(x)
wL1(y)
w1(y)
wU1(y)
rL2(y)
r2(y)
wL2(x)
w2(x)
rU2(y)
wU2(x)
Locking alone does
not enforce the rules
on serially equivalence.
Transactions

24. Locking is not enough Transaction T1 Transaction T2 rL1(x) r1(x)
rU1(x)
wL1(y)
w1(y)
wU1(y)
We now have r1(x), r2(y), w2(x), w1(y). But we need… ,
rL2(y)
r2(y)
wL2(x)
w2(x)
rU2(y)
wU2(x)
Transactions

25. Locking is not enough
We now have r1(x), r2(y), w2(x), w1(y). But we need…
either T1 followed by T2 or T2 followed by T1.
We need either r1(x), w1(y), r2(y), w2(x) or
r2(y), w2(x), r1(x), w1(y).
How do we guarantee that?
Two phase locking (2PL) demands that all locks are obtained
for each transaction before releasing any of them!
Transactions

26. Lock and Unlock in two Phases
Transaction T1 Transaction T2
rL1(x)
r1(x)
wL1(y)
w1(y)
wU1(y)
rU1(x) ,
rL2(y)
r2(y)
wL2(x)
w2(x)
rU2(y)
wU2(x)
Now T1 and T2
are serialized.
This may lead
to deadlock. The
serialization proof
exists but is
beyond course
scope.
Transactions
Transactions 26

27. What might Lock_Item() look like?
Lock_Item(x)
B: if(Lock(x) == 0)
Lock(x) = 1
else {
wait until Lock(x) == 0 and
we are woken up.
GOTO B }
Now, a transaction is free to use x.
Not interleaved with other
code until this terminates or
waits. In java, this would be a
synchronized method.
Similar to the code above
that used a shared
queue.
Transactions

28. Master of Information System Management
And unlock_item() ?
The transaction is done using x.
Unlock_Item(x)
Lock(x) = 0
if any transactions are waiting then
wake up one of the waiting
transactions.
Not interleaved with other
code. If this were java, this
method would be synchronized.
Transactions
Master of Information System Management

29. Does this allow for any concurrency?
In reality, the
coordinator
would do the
locking.
Transaction T Transaction T2
Lock_Item(x) Lock_Item(y)
T1 uses x T2 uses y
Unlock_Item(x) Unlock_Item(y)
If x differs from y these two transactions proceed concurrently.
If both want to use x, one waits until the other completes.
Transactions
Master of Information System Management

30. Locks May Lead to Deadlock
Four Requirements for deadlock:
(1) Resources need mutual exclusion. They are not thread safe.
(2) Resources may be reserved while a process is waiting for more.
(3) Preemption is not allowed. You can't force a process to give
up a resource.
(4) Circular wait is possible. X wants what Y has and Y wants what Z
has but Z wants what X has.
Solutions (short course):
Prevention (disallow one of the four)
Avoidance (study what is required by all before beginning)
Detection (using time-outs or wait-for graphs) and recovery
Transactions

31. Deadlock
Source: G. Coulouris et al., Distributed Systems: Concepts and Design, Third Edition.
Transactions

32. Local Transactions (Single Server)
Typically handled directly by a database.
Call beginTransaction on an SQLConnection object.
Exceute SQL statements.
Call commit or rollback.
Locks may be held on the rows/tables involved.
Everything is done on a copy until the commit or rollback.
Deadlock may be detected with wait-for graphs.
In distributed transactions, deadlock may be detected with time-outs.
Transactions

33. Transactions On Objects (Single Server)
a =lookUp(“A”)
b =lookUp(“B”)
beginTran
x = a.read()
b.write(x)
closeTran or
abortTran
“A”
Lock management, recovery management
and traditional
middleware B
“B” C
a =lookUp(“A”)
beginTran
x = a.read()
closeTran or
abortTran
“C”
Recoverable objects
Transactions

34. What is a Recoverable Object?
When the server is
running it can keep all
of its objects in its
volatile memory and
records its committed
objects in a recovery
file.
A recoverable object follows the
Golden Rule of Recoverability:
“Never modify the only copy.”
The transaction make changes
to local copies of resources until
a commit or rollback.
Upon recovery, the
server can restore the
object’s latest committed
versions.
If a transaction fails, only the
tentative versions of the objects
have changed, not the
non-volatile copy.
Transactions
Transactions 34

35. Distributed Transactions (More than one server)
Begin transaction BookTrip
book a plane from Qantas
book hotel from Hilton
book rental car from Hertz
End transaction BookTrip
The Two Phase Commit Protocol is a classic solution for atomicity and
consistency.
Transactions

36. Interacting with a coordinator
Transaction T tid = openTransaction(); a.withdraw(tid,100); b.deposit(tid,100); c.withdraw(tid,200); b.deposit(tid,200); closeTransaction(tid) or abortTransaction(tid)
Coordinator Interface:
openTransaction() -> transID
closeTransaction(transID) ->
commit or abort
abortTransaction(TransID)
Think about atomicity and
consistency.
Transactions
Transactions 36

37. Client Talks to a Coordinator
Different servers
Any server
BookPlane Participant
BookTrip
Coordinator
Recoverable objects needed
to book a plane
BookHotel Participant
Recoverable objects needed
to book a hotel.
openTrans
Unique Transaction ID
TID
BookRentalCar Participant
Recoverable objects needed
to rent a car.
BookTrip Client
TID = openTransaction()
Transactions

38. Client Uses Services BookPlane Participant BookTrip Coordinator
Different servers
Any server
BookPlane Participant
BookTrip
Coordinator
Recoverable objects needed
to book a plane
BookHotel Participant
Recoverable objects needed
to book a hotel.
Call + TID
BookRentalCar Participant
Recoverable objects needed
to rent a car.
BookTrip Client
plane.bookFlight(111,”Seat32A”,TID)
Transactions

39. Participants Talk to Coordinator
The participant only
calls join if it has not
already done so.
Different servers
BookPlane Participant
BookTrip
Coordinator
Recoverable objects needed
to book a plane
join(TID,ref to participant)
BookHotel Participant
Recoverable objects needed
to book a hotel.
BookRentalCar Participant
BookTrip Client
The participant knows where the
coordinator is because that
information can be included in
the TID (eg. an IP address.)
The coordinator now has a pointer to the
participant.

40. Suppose All Goes Well (1)
Different servers
BookPlane Participant
BookTrip
Coordinator
Recoverable objects needed
to book a plane
BookHotel Participant
Recoverable objects needed
to book a hotel.
BookRentalCar Participant
BookTrip Client
Recoverable objects needed
to rent a car.
OK returned
OK returned
OK returned
Transactions

41. Suppose All Goes Well (2)
Different servers
BookPlane Participant
BookTrip
Coordinator
Recoverable objects needed
to book a plane
BookHotel Participant
Coordinator begins
2PC and this results in
a GLOBAL COMMIT
sent to each participant.
Recoverable objects needed
to book a hotel.
BookRentalCar Participant
Recoverable objects needed
to rent a car.
BookTrip Client
OK returned
OK returned
OK returned
CloseTransaction(TID) Called

42. This Time No Cars Available (1)
Different servers
BookPlane Participant
BookTrip
Coordinator
Recoverable objects needed
to book a plane
BookHotel Participant
Recoverable objects needed
to book a hotel.
BookRentalCar Participant
Recoverable objects needed
to rent a car.
BookTrip Client
OK returned
OK returned
NO CARS AVAIL
abortTransaction(TID) called

43. This Time No Cars Available (2)
Different servers
BookPlane Participant
BookTrip
Coordinator
Recoverable objects needed
to book a plane
BookHotel Participant
Coordinator sends a
GLOBAL_ABORT to all
particpants
Recoverable objects needed
to book a hotel.
BookRentalCar Participant
Recoverable objects needed
to rent a car.
BookTrip Client
OK returned
OK returned
NO CARS AVAIL
abortTransaction(TID) called

44. This Time No Cars Available (3)
Different servers
BookPlane Participant
BookTrip
Coordinator
ROLLBACK CHANGES
BookHotel Participant
abortTransaction
ROLLBACK CHANGES
Each participant
Gets a GLOBAL_ABORT
BookRentalCar Participant
ROLLBACK CHANGES
BookTrip Client
OK returned
OK returned
NO CARS AVAIL
abortTransaction(TID)

45. BookPlane Server Crashes After Returning ‘OK’ (1)
Different servers
BookPlane Participant
BookTrip
Coordinator
Recoverable objects needed
to book a plane
BookHotel Participant
Recoverable objects needed
to book a hotel.
BookRentalCar Participant
BookTrip Client
Recoverable objects needed
to rent a car.
OK returned
OK returned
OK returned
Transactions

46. BookPlane Server Crashes After Returning ‘OK’ (2)
Different servers
BookPlane Participant
BookTrip
Coordinator
Recoverable objects needed
to book a plane
BookHotel Participant
Coordinator excutes 2PC:
Ask everyone to vote.
No news from the BookPlane
Participant so multicast a
GLOBAL ABORT
Recoverable objects needed
to book a hotel.
BookRentalCar Participant
Recoverable objects needed
to rent a car.
BookTrip Client
OK returned
OK returned
OK returned
CloseTransaction(TID) Called

47. BookPlane Server Crashes after returning ‘OK’ (3)
Different servers
BookPlane Participant
BookTrip
Coordinator
Recoverable objects needed
to book a plane
BookHotel Participant
GLOBAl ABORT
ROLLBACK
BookRentalCar Participant
ROLLBACK
BookTrip Client
OK returned
OK returned
ROLLBACK
OK returned
CloseTransaction(TID) Called

48. Two-Phase Commit Protocol
BookPlane
Vote_Request
BookTrip
Coordinator
Vote_Commit
Vote Request
BookHotel
Vote Commit
Vote Request
BookRentalCar
Phase 1 BookTrip coordinator
sends a Vote_Request to each
process. Each process returns
a Vote_Commit or Vote_Abort.
Vote Commit
Transactions

49. Two-Phase Commit Protocol
BookPlane
Global Commit
BookTrip
Coordinator
ACK
BookHotel
Global Commit
ACK
Global Commit
BookRentalCar
Phase 2 BookTrip coordinator
checks the votes. If every process
votes to commit then so will the coordinator.
In that case, it will send a Global_Commit to each process.
If any process votes to abort the coordinator sends a GLOBAL_ABORT.
Each process waits for a Global_Commit message before committing its part of the
transaction.
ACK

50. 2PC Finite State Machine from Tanenbaum
BookTrip Coordinator
Participant
State has already been saved to permanent
storage.
Init
Init
Vote-request
Vote-commit
Commit
Vote-request
Vote-request
Vote-abort
Ready
wait
Vote-abort
Global-abort
Vote-commit
Global-commit
Global-commit
ACK
Global-abort
ACK
Commit
Abort
Commit
Abort
Transactions

51. 2PC Blocks in Three Places
If waiting too long for a Vote-Request
send a Vote-Abort
Init
Init
Vote-request
Vote-commit
Commit
Vote-request
Vote-request
Vote-abort
Ready
wait
Vote-abort
Global-abort
Vote-commit
Global-commit
Global-commit
ACK
Global-abort
ACK
Commit
Abort
Commit
Abort
Transactions

52. 2PC Blocks in Three Places
Init
Init
Vote-request
Vote-commit
Commit
Vote-request
If waiting too long
After Vote-request
Send a Global-Abort
Ready
Vote-request
Vote-abort
wait
Vote-abort
Global-abort
Vote-commit
Global-commit
Global-commit
ACK
Global-abort
ACK
Commit
Abort
Commit
Abort
Transactions

53. 2PC Blocks in Three Places
If waiting too long we can’t simply abort! We must wait
until the coordinator recovers. We might also make queries
on other participants.
Init
Init
Vote-request
Vote-commit
Commit
Vote-request
Vote-request
Vote-abort
Ready
wait
Vote-abort
Global-abort
Vote-commit
Global-commit
Global-commit
ACK
Global-abort
ACK
Commit
Abort
Commit
Abort
Transactions

54. 2PC Blocks in Three Places
If this process learns that another has committed then this
process is free to commit. The coordinator must have sent out
a Global-commit that did not get to this process.
Init
Init
Vote-request
Vote-commit
Commit
Vote-request
Vote-request
Vote-abort
Ready
wait
Vote-abort
Global-abort
Vote-commit
Global-commit
Global-commit
ACK
Global-abort
ACK
Commit
Abort
Commit
Abort
Transactions

55. 2PC Blocks in Three Places
If this process learns that another has aborted then it too
is free to abort.
Init
Init
Vote-request
Vote-commit
Commit
Vote-request
Vote-request
Vote-abort
Ready
wait
Vote-abort
Global-abort
Vote-commit
Global-commit
Global-commit
ACK
Global-abort
ACK
Commit
Abort
Commit
Abort
Transactions

56. 2PC Blocks in Three Places
Suppose this process learns that another
process is still in its init state. The coordinator must have
crashed while multicasting the Vote-request. It’s safe for
this process (and the queried process) to abort.
Init
Init
Vote-request
Vote-commit
Commit
Vote-request
Vote-request
Vote-abort
Ready
wait
Vote-abort
Global-abort
Vote-commit
Global-commit
Global-commit
ACK
Global-abort
ACK
Commit
Abort
Commit
Abort
Transactions

57. 2PC Blocks in Three Places
Tricky case: If the queried processes are all still in their ready
state what do we know? We have to block and wait until the
Coordinator recovers.
Init
Init
Vote-request
Vote-commit
Commit
Vote-request
Vote-request
Vote-abort
Ready
wait
Vote-abort
Global-abort
Vote-commit
Global-commit
Global-commit
ACK
Global-abort
ACK
Commit
Abort
Commit
Abort
Transactions

58. Summary 2PL and 2PC
Two phase locking (2PL) is a concurrency control protocol – guarantees transactions have the same effect as serial execution. Guarantees transactions do not interfere with each other. Does it prevent deadlock?
The two phase commit protocol (2PC) is an atomic commitment protocol – a special type of consensus protocol. Does it prevent deadlock? Typically used for distributed transactions.
A consensus protocol tries to reach agreement in the presence of crashes or failures. All agree?
Transactions
Transactions 58

59. DS Principles For Internet Scale applications (Ian Gorton SEI)
Goal: highly available and highly scalable.
Principle: Complex systems do not scale. 2PC is complex. Thus, on the internet, weak consistency is replacing strong consistency. For example, a change to a Google Doc may not be available to all immediately.
Principle: Statelessness. Any server, any time implies keeping state off the server. No routing to correct server.
Principle: Allow failures but be able to monitor system behavior.
The CAP Theorem was first announced in 2000.
Principle: Mixed approaches (consistency tradeoffs) are possible.
Transactions

60. The CAP Theorem (Brewer)
Consistency: if a value is written to node 1 then it is that value that is read from node 2. It is not the case that one node has an old value and another node has the most recent value. Brewer says “consistency (C) is equivalent to having a single up-to-date copy of the data;”
Available: The system is highly available for updates.
Partition tolerance: If a network failure occurs the system will still work.
CAP Theorem: You may only have a system with two of these three. You may have either CA, CP, or AP.
Transactions

61. The Traditional CAP Theorem (Brewer)
Essentially, the CAP theorem says that in the face of network partitions, you may either have availability or consistency but not both.
Example: Suppose there is a break in the network between an automated teller machine and the main banking database. We have a choice. We can be either unavailable for ATM use or we can allow for small transactions and be a little inconsistent. If we choose the latter we can still reconcile any inconsistencies later.
Example: Suppose we are using HTML5 local storage and do some disconnected work offline. We are choosing availability over consistency – and may only require eventual consistency.
Transactions

62. CAP Theorem Update Brewer says (2012):
“Because partitions are rare, CAP should allow perfect C and A most of the time, but when partitions are present or perceived, a strategy that detects partitions and explicitly accounts for them is in order. This strategy should have three steps: detect partitions, enter an explicit partition mode that can limit some operations, and initiate a recovery process to restore consistency and compensate for mistakes made during a partition.”
Transactions