Lecture 13: Spanner

What this lecture covers

Spanner is a rare production system that offers serializable transactions over data spread across data centers worldwide. The lecture builds on the previous lecture's two-phase locking and two-phase commit, and asks how to keep those mechanisms correct and fast when participants are themselves replicated groups spread across a continent, and when most of the workload is read-only.

It traces two central ideas: running two-phase commit over Paxos-replicated participants so a crashed coordinator no longer blocks the system, and using synchronized clocks (Spanner's TrueTime) to let read-only transactions read consistent data from a nearby replica without locks or coordination. After watching, you should be able to explain how Spanner executes a read-write transaction end to end, why plain "read the latest value" breaks serializability for read-only transactions, and how snapshot isolation plus TrueTime's start and commit-wait rules fix it.

Key ideas

• Sharding plus Paxos replication: Spanner splits data into shards by key, and each shard's replicas across data centers form an independent Paxos group with its own leader, giving both parallel throughput and fault tolerance.
• External consistency: if one transaction finishes committing before another starts in real time, the second transaction must see the first's writes; this is equivalent to linearizability applied to transactions.
• Read-write transactions: use standard two-phase locking and two-phase commit, but the participants and the transaction coordinator are each Paxos-replicated groups, not single machines.
• Why replicating the coordinator matters: if a plain two-phase commit coordinator crashes, all its transactions block with locks held; replicating it with Paxos lets a new leader take over and continue from the log, eliminating that blocking failure.
• Read-only transactions: skip locking and two-phase commit entirely and can read from a local replica, at the cost of relying on a different consistency mechanism.
• Snapshot isolation: every transaction gets a timestamp, records are kept in multiple versions, and a read returns the highest-timestamped version at or below the reader's timestamp, emulating execution in timestamp order.
• Safe time: a replica delays answering a read for a given timestamp until it has received Paxos log entries covering that timestamp, so it never returns a version older than the request implies.
• TrueTime, start rule, and commit wait: since clocks can't be perfectly synchronized, every time read returns an [earliest, latest] interval; a transaction's timestamp is the latest value at start (or commit) time, and a read-write transaction must wait until its chosen timestamp is provably in the past before it commits.

Walkthrough

Motivation and physical layout (2:02)

Spanner grew out of Google's advertising database, which was manually sharded across MySQL and Bigtable instances with no cross-shard transactions. The lecture describes the physical setup: data centers each holding shards of the keyspace, with every shard replicated across multiple data centers via a Paxos variant much like Raft. Clients are web servers in a data center, and each Paxos group has its own leader, so writes to a shard must go to that shard's leader.

Read-write transactions: two-phase locking and commit over Paxos groups (11:19)

The lecture traces a bank-transfer-style transaction across two shards held in different Paxos groups. The client reads from each shard's leader, which sets a lock and returns the value; all writes happen together near commit time. The client picks one Paxos group to act as transaction coordinator, sends writes to each shard's leader, and each leader logs a prepare through its own Paxos group before voting yes to the coordinator. Only if every leader votes yes does the coordinator log and send commit messages, after which each shard applies its write and releases locks. Because the coordinator's decision is itself replicated in a Paxos log, a new leader can take over and finish the protocol if the original coordinator fails, removing two-phase commit's classic blocking-on-coordinator-failure problem.

The cost of read-write transactions and the case for a faster read-only path (22:34)

The lecture quantifies the expense of this design: cross-country transactions can take around 100 milliseconds, versus roughly 14 milliseconds when replicas are nearby, both measured in the paper's tables. Since the advertising workload is dominated by read-only transactions, Spanner gives them a much cheaper path: reading from local replicas with no locks, no two-phase commit, and no transaction manager, producing roughly a tenfold latency improvement.

Why "just read the latest value" fails (30:40)

Before presenting the real design, the lecture shows why the obvious shortcut breaks correctness: a slow read-only transaction that reads x before a second read-write transaction commits, then reads y after it commits, can see a value of x from before that transaction and a value of y from after it. This result matches no valid serial ordering of the transactions, since it mixes data from before and after the same commit.

Snapshot isolation (35:53)

The fix, assuming synchronized clocks for now, is snapshot isolation: every transaction gets a timestamp (commit time for read-write, start time for read-only), the database keeps multiple versions of each record tagged by timestamp, and a read returns the highest version at or below the reader's timestamp. Working through the earlier example with timestamps 10, 15, and 20 shows the read-only transaction now reads both values as of timestamp 15, which corresponds to a valid serial order. The lecture also explains "safe time": a replica won't answer a read for a given timestamp until its Paxos log has caught up to that point, so a lagging replica doesn't return stale data silently.

Clock uncertainty and TrueTime (53:11)

Perfectly synchronized clocks are impossible, so Spanner uses GPS and other time sources feeding local "time masters," with genuine uncertainty from signal propagation delay and local clock drift. Every time query returns a [earliest, latest] interval rather than a single value. A read-only transaction with a too-large timestamp merely waits longer; one with a too-small timestamp risks missing recent commits, which would violate external consistency.

Start rule and commit wait (1:06:26)

Two rules from the paper fix the uncertainty problem: the start rule assigns a transaction the latest value of the current TrueTime interval as its timestamp, guaranteeing the timestamp is not yet in the past when chosen. The commit-wait rule forces a read-write transaction to keep polling the clock and delay committing until its chosen timestamp is provably earlier than the current earliest bound. A worked example with transactions T0, T1, and T2 shows how this combination guarantees that a read-only transaction starting after another transaction commits is assigned a strictly later timestamp, so it is guaranteed to see that transaction's write.

Before you watch

• Review two-phase locking and two-phase commit from the previous lecture on distributed transactions, since Spanner's read-write path reuses both directly.
• Be familiar with Raft or Paxos leader election and log replication, since Spanner replicates both shards and the transaction coordinator this way.
• Recall linearizability, since the lecture treats external consistency as equivalent to it applied to transactions.

Check your understanding

1. Why does replicating the transaction coordinator with Paxos eliminate two-phase commit's usual blocking failure mode?
2. Why does simply reading the freshest value of each record fail to produce serializable results for a read-only transaction?
3. How does snapshot isolation use per-record timestamps to make a set of reads behave as if they all happened at one point in time?
4. What problem does "safe time" solve, and why can't a replica just answer with whatever version it currently has?
5. Why are both the start rule and the commit-wait rule necessary to guarantee external consistency, when snapshot isolation alone already guarantees serializability?