EECS 498 Introduction to Distributed Systems Fall 2017 Harsha V. Madhyastha

November 29, 2017 EECS 498 – Lecture 20

November 29, 2017 EECS 498 – Lecture 20

Context for Spanner Distributed database that supports transactions BigTable: Database spread across servers within a datacenter Spanner: Database spread across data centers Employee ID Date of Joining Age Title Salary November 29, 2017 EECS 498 – Lecture 20

Support for Transactions Serializability Externally visible effects equivalent to some serial order of execution How is this different from linearizability? Serializability does not guarantee real time order Spanner offers strict serializability Serializability + Linearizability Previously considered impractical at global scale November 29, 2017 EECS 498 – Lecture 20

Example: Social Network Transaction T1: Remove supervisor from friend’s list Make a negative post about supervisor Transaction T2: Read friend list Read posts made by friends November 29, 2017 EECS 498 – Lecture 20

Executing Transactions Transaction latency = 4 rounds of wide-area communication 2 rounds for each phase of 2PL November 29, 2017 EECS 498 – Lecture 20

Executing Transactions Leader-based Paxos  each phase of 2PL in 1 round-trip Still too slow for read-only transactions November 29, 2017 EECS 498 – Lecture 20

Read-Only Txns with 2PL TC P1 P2 P3 Lock Commit November 29, 2017 EECS 498 – Lecture 20

Lock-free Read-Only Txns How to ensure T1’s read from P1 returns data from before T2? TC1 Read P1 P2 Lock Commit TC2 November 29, 2017 EECS 498 – Lecture 20

Snapshot Reads Every transaction has a commit timestamp When a row is modified by transaction, create new version Transaction’s timestamp is version number Read-only transaction reads latest version of data committed before its timestamp What property must be true for linearizability? Commit timestamps should monotonically increase November 29, 2017 EECS 498 – Lecture 20

Executing Transactions How to ensure commit timestamps monotonically increase? November 29, 2017 EECS 498 – Lecture 20

Synchronizing Clocks Accurate sources of time: GPS Atomic clocks Why not put these into every server? Spanner deploys a few of these in every DC All servers periodically sync with timemasters November 29, 2017 EECS 498 – Lecture 20

TrueTime Architecture GPS timemaster GPS timemaster GPS timemaster GPS timemaster Atomic-clock timemaster GPS timemaster Client Datacenter 1 Datacenter 2 … Datacenter n Current time = now ± ε November 29, 2017 EECS 498 – Lecture 20

TrueTime Implementation Between syncs, assume worst case clock drift ε +6ms 200 μs/sec reference uncertainty time 0sec 30sec 60sec 90sec November 29, 2017 EECS 498 – Lecture 20

TrueTime TT.now() returns a range: [earliest, latest] Current time is … … not less than earliest … not greater than latest High-level principle: Known unknowns better than unknown unknowns November 29, 2017 EECS 498 – Lecture 20

Timestamps and TrueTime Acquired locks Release locks T Pick commit timestamp s s Wait until TT.now().earliest > s = TT.now().latest average ε average ε November 29, 2017 EECS 498 – Lecture 20

Leader Leases Must ensure at most one leader in any replica group at any point in time In Spanner, leader gets a lease on how long it gets to be the leader How can group members come to consensus on leader leases to ensure leader disjointness? Assume clocks are perfectly synchronized, but network has arbitrary delays How to leverage TrueTime? November 29, 2017 EECS 498 – Lecture 20

Paxos-based Leader Leases When a replica responds to a proposer at time t, it grants lease from [t, t + 10s] t = TT.now().earliest() Proposer assumes leader lease until minimum of replica leases Lease valid until TT.now().latest > min lease November 29, 2017 EECS 498 – Lecture 20

Schema-Change Transactions Not scalable to acquire locks on all shards How can we leverage TrueTime? Schedule transaction for time in future Employee ID Date of Joining Age Title Salary Gender November 29, 2017 EECS 498 – Lecture 20

Cloud Spanner Spanner recently introduced to Google Cloud Claims to offer high availability Spanner’s communication over Google’s private network Well engineered over last decade Partitions are very uncommon Spanner unavailability << VM unavailability Don’t need app logic to deal with Spanner outage November 29, 2017 EECS 498 – Lecture 20

Causes of Spanner Outages November 29, 2017 EECS 498 – Lecture 20