Presented By Alon Adler – Based on OSDI ’12 (USENIX Association) Spanner Google’s scalable, multi-version, globally-distributed and synchronously-replicated database Presented By Alon Adler – Based on OSDI ’12 (USENIX Association)

Why Spanner born? Google had BigTable and MegaStore. Why not BigTable ? Can’t handle with complex, evolving schemes. Only eventual consistency across datacenters. Transactional scope limited to single row. Why not MegaStore ? Low performance.

So, What is Spanner? At high level of abstraction, it is a database that shards data across many set of Paxos state machines in datacenters spread all over the world. Spanner is designed to scale up to millions of machines across hundreds of datacenters and trillions of database rows. Spanner maintained multiple replicas for each data. Replication is used for global availability. Applications can use Spanner for high availability even in face of wide-area natural disasters.

So, What is Spanner? Spanner supports general-purpose transactions (ACID). Atomicity, Consistency, Isolation, Durability. Sometimes “Eventually-consistent” of BigTable isn’t good enough. Spanner provides a SQL-based query language. Which provides to the applications the ability to handle complex schemes.

A Spanner deployment is called “universe”. universemaster – status of all zones placement driver – transfers data between zones location proxies – Used by clients to locate spanservers that hold the data they need zonemaster allocates data to spanservers Thousands of spanservers per zone

Spanserver Software Stack Tables sharded across rows into tablets (like bigtable) . Tablet maps (key:string, timestamp:int64)->string. Each spanserver is responsible for. 100-1000 tablets Paxos state machine enables support synchronous replication.

Paxos State Machine Paxos state machines - to implement a consistently replicated bag of mapping. The key-value mapping state of each replica is stored in its corresponding tablet. Writes must initiate the Paxos protocol at the leader. The set of replicas is collectively a Paxos Group. Each replica can be located on different datacenter.

Spanner’s Features As a globally-distributed database , Spanner provides several interesting features. Applications can specify constraints to control which datacenters contain which data : How far data is from its users (to control read latency). How replicas are from each other (to control write latency). How many replicas are maintained (to control durability, availability and read performance). In addition, Spanner has two features that are difficult to implement in a distributed database : Externally-consistent reads and writes. Globally-consistent reads across the database at a timestamp.

Spanner’s Features Why Externally-consistent reads and writes and Globally-consistent reads across the database at a timestamp are difficult to implement in a distributed database. Because we don't have a global “Wall Clock”.

So, what we can do? Global “Wall-Clock” time == External Consistency : Commit order respects global wall-time order. So, we will transform the problem to : Timestamp order respects global wall-time order. timestamp order == commit order.

Assigning timestamps to RW transactions Transaction that write use 2PL. Each transaction T is assigned a timestamp s. Data written by T is timestamped with s. Assign timestamp while locks are held. Acquired locks Release locks T Pick s = now()

Timestamp Invariants Timestamp order == commit order Timestamp order respects global wall-time order T3 T4

TrueTime API The key enabler of these properties (previous slide) is a new TrueTime API and its implementation. The API exposes clock uncertainty, and the guarantees on Spanner’s timestamps depend on the bounds that the implementation provides. The implementation keeps uncertainty small (generally less than 10ms) by using multiple modern clock references (GPS and atomic clocks).

TrueTime “Global wall-clock time” with bounded uncertainty. TT.now() earliest latest 2*ε

Timestamps and TrueTime Acquired locks Release locks T Pick s = TT.now().latest s Wait until TT.now().earliest > s Commit wait average ε average ε

Operations Spanner supports: Read-write transaction. Read-only transaction. Snapshot reads. Read-only transaction must be pre-declared as not have any writes. Reads in read-only transactions execute at a system-chosen timestamp without locking, so that incoming writes are not blocked. Snapshot read is a read in the past that execute without locking. Client can either specify a timestamp or provide an upper bound.

Reads within read-write transactions Writes that occur in a transaction are buffered at the client until commit, as a result reads in a transaction do not see the effects of them. The client issues reads to the leader replica of the appropriate group. Acquires read locks and then reads the most recent data. While a client transaction remains open, it sends “keep-alive” messages. When a client has completed all reads and buffered all writes , write protocol begin.

RW transactions which involves one Paxos Group Start consensus Achieve consensus Notify slaves Acquired locks Release locks T Pick s Commit wait done

RW transactions which involves more than one Paxos Group – 2PC protocol Start logging Done logging Acquired locks Release locks TC Committed Notify participants of s Acquired locks Release locks TP1 Acquired locks Release locks TP2 Prepared Send s Compute s for each Commit wait done Compute overall s

Example Remove X from my friend list Risky post P TC T2 sC=6 s=8 s=15 Remove myself from X’s friend list TP sP=8 s=8 Time <8 8 15 My friends [X] [] My posts [P] X’s friends [me] []

Serving Reads at a Timestamp Each replica maintains 𝑡 𝑆𝑎𝑓𝑒 . A replica can satisfy a read at a timestamp t if t <= 𝑡 𝑆𝑎𝑓𝑒 . 𝑡 𝑆𝑎𝑓𝑒 = min( 𝑡 𝑆𝑎𝑓𝑒 𝑃𝑎𝑥𝑜𝑠 , 𝑡 𝑆𝑎𝑓𝑒 𝑇𝑀 ). 𝑡 𝑆𝑎𝑓𝑒 𝑃𝑎𝑥𝑜𝑠 is timestamp of highest-applied Paxos write. 𝑡 𝑆𝑎𝑓𝑒 𝑇𝑀 is much harder: = ∞ if no pending 2PC transaction. = minimum (s-prepare i,g ) over i prepared transactions in group g. Thus, 𝑡 𝑆𝑎𝑓𝑒 is maximum timestamp at which reads are safe.

Read-Only transactions Executes in two phases: Assign a timestamp 𝑆 𝑅𝑒𝑎𝑑 . Reads as snapshot reads at 𝑆 𝑅𝑒𝑎𝑑 . The snapshot reads can execute at any replicas that are up-to-date. The simple assignment of 𝑆 𝑅𝑒𝑎𝑑 =TT.now().latest , preservers external consistency. Such a timestamp may require the execution of data reads at 𝑆 𝑅𝑒𝑎𝑑 to block if 𝑡 𝑆𝑎𝑓𝑒 has not advanced sufficiently. To reduce the chances of blocking, Spanner should assign the oldest timestamp that preserved external consistency.

Read-Only transactions Assigning a timestamp requires a negotiation phase between all of the Paxos groups that are involved in the read. As a result , Spanner requires a scope expression that summarizes the keys that will be read. If the scope’s values are served by a single Paxos group: The client issues the read-only transaction to the group leader. The leader assign 𝑆 𝑅𝑒𝑎𝑑 = LastTS() (=the timestamp of the last committed write at Paxos). And execute the read at any up-to-date replica. If the scope’s values are served by multiple Paxos groups: 𝑆 𝑅𝑒𝑎𝑑 = TT.now().latest (which may wait for safe time to advance).

Benchmarks 50 Paxos groups, 2500 buckets, 4KB reads or writes, datacenters 1ms apart. Latency remains mostly constant as number of replicas increases because Paxos executes in parallel at a group’s replicas.

Benchmarks All leaders explicitly placed in zone Z1. Red-line – Killing non-leader no effects on read throughput. Green-line – Killing leader-soft giving the leaders time to handoff leadership. Blue-line – Killing leader-hard no warning for leaders.

Questions? Thanks!