1. CSCI5570 Large Scale Data Processing Systems
NewSQL
James Cheng
CSE, CUHK
Slide Ack.: modified based on the slides from Lianghong Xu

2. Calvin: fast distributed transactions for partitioned database systems
Alexander Thomson, Thaddeus Diamond, Shu-Chun Weng, Kun Ren, Philip Shao, Daniel J. Abadi
SIGMOD 2012

3. High-­level Overview A transaction processing and replication layer
Based on generic, non-transactional data store
Full ACID for distributed transactions
Active, consistent replication
Horizontal scalability
Key technical feature
Scalable distributed transaction processing by a deterministic locking mechanism that eliminates distributed commit protocols

4. The Problem Distributed transactions are expensive Agreement protocol
Multiple roundtrips in 2-phase commit
Much longer than transaction logic itself
Limited scalability
Locking
Lock held during the entire transaction
Possible deadlock

5. Consistent replication
Many systems allow inconsistent replication
Replicas can diverge, but are eventually consistent
Dynamo, SimpleDB, Cassandra…
Consistent replication: emerging trend (for OLTP)
Instant failover
Increased latency, especially for geo-replication
Cost is only in latency, not throughput or contention
lock contention: This occurs whenever one process or thread attempts to acquire a lock held by another process or thread. The more fine-grained the available locks, the less likely one process/thread will request a lock held by the other. (For example, locking a row rather than the entire table, or locking a cell rather than the entire row.)
Lock contention occurs when one process or thread attempts to acquire a lock held by another process or thread

6. Goals of Calvin Eliminate 2-phase commit (scalability)
Reduce lock duration (throughput)
Provide consistent replication (consistency)
Approach:
Decoupling “transaction logic” from “heavy-lifting tasks”
replication, locking, disk access …
Deterministic concurrency control

7. Non-deterministic Database Systems
Serial ordering cannot be pre-determined given certain transaction inputs
Determined in the runtime
logically equivalent to some serial ordering
Ordering can diverge for different executions
Due to different thread scheduling, network latencies …
Abort on non-deterministic events, e.g., node failure

8. Serial Ordering in 2-phase Locking

9. Deterministic database systems
Given transactional inputs, serial ordering is pre-determined
Acquire looks in the order of the pre-determined transactional input
All replicas can be made to emulate the same serial execution order
Database states can be guaranteed not to diverge

10. Deterministic database systems
Given transactional inputs, serial ordering is pre-determined
Benefits:
No agreement protocol
No need to check node failures
Recovery from other replicas
No aborts due to non-deterministic events
Consistent replication made easier
Only need to replicate transactional inputs
Execute same transaction inputs in replicating sites
Disadvantage
Reduced concurrency due to the pre-determined ordering

11. Architecture Consistent replication Deterministic locking
Client
Requests
Replicated requests
Consistent replication
Sequencer
Sequencer
Sequencer
Sequencer
Partial Requests batch
Deterministic locking
Scheduler
Scheduler
Scheduler
Scheduler
Remote access
create, read, update and delete
Local access
“CRUD” storage interface
Storage
Storage
Storage
Storage
Partition 1
Partition 2
Partition 1
Partition 2
Replica A
Replica B

12. Sequencer and replication
Transactional inputs are divided into batches each fallen into a 10-msec epoch
Sequencer places transactions in each batch as a global sequence
Nodes are organized into replication groups, each containing all replicas of a particular partition (e.g., partition 1 in replica A and partition 1 in replica B form a replication group)
Two modes of replication
asynchronous mode
synchronous mode

13. Sequencer and replication
Only transactional inputs need to be replicated
Asynchronous replication
one replica is designated as master
master replica handles all transaction inputs
each sequencer on master propagates to slave sequencers in its replication group
low latency, but complex failure recovery
Synchronous replication
all sequencers in a replication group use Paxos (ZooKeeper) to agree on a combined batch of transactions for each epoch
larger latency but throughput not affected
contention footprint (i.e., total duration a transaction holds its locks) remains the same

14. Scheduler and deterministic locking
Each node’s scheduler only lock records stored locally
Single-threaded locking manager
Resemble 2-phase locking
Enforce transaction ordering from sequencers
if transactions A and B both request locks on record R, and A before B in the sequence, then A must request lock on R before B, and lock manager must grant lock on R to A before B
No deadlock
Question: what if A and B come from 2 sequencers, X and Y, that are not in the same replication group? How to order A and B? it seems that X and Y do not talk to each other (or also use Paxos to agree on a global order?)

15. Deterministic locking
Read A
Read B
Write A T1
Pre-determined
serial ordering
Read B
Write B T2
Read C T3
T1 must precede T2
T1: Lw(A)
T1: Lr(B)
T2: Lw(B)
T3: Lr(C)
Memory A B C
Disk

16. Worker execution A B C Read A Read B Write A T1 Pre-determined
serial ordering
Read B
Write B T2
Read C T3
Active participant: write set is stored locally
Passive participant: only read set is stored
Memory A B C
Disk
Active participant
Passive participant
Passive participant

17. Worker execution A B C Read A Read B Write A T1 Pre-determined
serial ordering
Read B
Write B T2
Read C T3
Memory A B C
Disk
Active participant

18. Problem of deterministic locking
Non-deterministic locking
Read A
Read B
Write A
Read A
Read B
Write B
Read C
Write A
Need to fetch from disk T1
Reordering
Read B
Write B T2
Read C T3
T2 cannot proceed because T1 is block waiting on disk access to A
Even if T1 does NOT acquire lock for B yet
T3 can still proceed

19. Calvin’s solution Delay forwarding transaction requests Problem
Prefetch data from disk to eliminate contention
Hope: most (99%) transactions execute without access to disk
Problem
How long to delay?
Need to estimate disk and network latency
Tracking all in-memory records
Need a scalable approach
Future work

20. Checkpointing Fault-tolerance made easier in Calvin
Instant failover
Only log transactional inputs, no REDO log
Checkpointing for fast recovery
Failure recovery from a recent state
Rather than from scratch

21. Checkpointing Naive synchronous checkpointing
Freeze one replica only for checkpointing
Difficult to bring the frozen replica up-to-date
Asynchronous variation of Zigzag
Zigzag
2 copies per record in each replica, 2 times memory footprint
1 copy is up-to-date, the other is the latest checkpoint
Calvin’s Zigzag algorithm
Specify a point in the global serial order, two versions for each record:
before: only updated by transactions preceding the point
after: only updated by transactions after the point
Once all transactions preceding the point completed, before version is immutable and checkpointed
Then before version is deleted, after version will be used for the next checkpointing

22. Scalability Calvin scales (near-) linearly
Throughput comparable to world-record
With much cheaper hardware

23. Scalability Scales linearly for low contention workloads
Figure 5: Total and per-node microbenchmark throughput, varying deployment size.
Scales linearly for low contention workloads
Scales sub-linearly when contention is high
Stragglers (slow machines, execution process skew)
Exacerbated with high contention

24. Scalability under high contention
Slowdown is measured by comparing with running a perfectly partitionable, low-contention version of the same workload
Calvin scales better than 2PC in face of high contention