Lecture 4: Primary-Backup Replication

What this lecture covers

This lecture asks how you build a service that keeps running when a single computer crashes, using replication rather than the sharded, application-level fault tolerance of GFS covered earlier in the course. It works through the VMware FT paper, which replicates an entire virtual machine, memory, registers, and all, so that any unmodified software can be made fault tolerant without understanding its internals.

By the end, you can explain the difference between state transfer and replicated state machine approaches to replication, describe how VMware FT keeps a primary and backup synchronized at the instruction level, and reason about the failure cases that make replication hard: lost acknowledgments, duplicate output, and network partitions that could split the system into two live primaries.

Key ideas

• Fail-stop failures: replication only protects against a computer cleanly stopping (power loss, network cut, crash), not against software bugs or hardware that computes wrong answers silently.
• Correlated failures: identical hardware bought in bulk, or replicas in the same building, can fail together, which defeats the independence replication relies on.
• State transfer: the primary periodically ships its whole memory image to the backup; simple but expensive to keep current.
• Replicated state machine: the primary forwards only the external inputs (packets, interrupts) that drive it; the backup replays them and stays in sync as long as execution is deterministic.
• Logging channel: the stream of inputs and nondeterministic-instruction results the primary sends the backup so it can replay the exact same instruction sequence.
• Output rule: the primary must hold a reply to a client until the backup has acknowledged the log entry that produced it, so a reply is never seen unless the backup also knows about the request.
• Test-and-set tiebreaker: an external service decides which replica is allowed to go live, preventing both primary and backup from taking over after a network partition.

Walkthrough

Fail-stop failures and the limits of replication (0:03)

The lecture opens by scoping what replication can and cannot fix. It only handles fail-stop failures, where a machine simply stops rather than computing something wrong. Bugs in the replicated software, bugs in the replication scheme itself, and hardware that miscomputes are all out of scope, since both replicas would just agree on the same wrong answer. Some hardware and software faults can be turned into fail-stop faults (checksums catching a corrupted packet, an unrelated kernel panic taking the whole machine down), but this is a bonus, not a guarantee. The lecture also stresses that replicas must fail independently: buying identical machines in bulk, or housing both replicas in one building, risks correlated failures such as a shared manufacturing defect or a city-wide power outage. Whether replication is worth its cost (double or triple the machines) is framed as an economic question, not a technical one.

State transfer vs. replicated state machines (8:15)

Two general strategies are introduced. State transfer sends the primary's full state (its memory) to the backup periodically. Replicated state machine instead sends only the external events that hit the primary, on the assumption that if two machines start identical and see the same inputs in the same order, they stay identical. Because operations are usually far smaller than the state they produce, replicated state machine is more efficient, but it depends on much stronger assumptions about deterministic execution, and it does not work cleanly on multi-core hardware because instruction interleaving across cores is not deterministic. VMware FT, the paper for the lecture, uses a replicated state machine approach and is restricted to uniprocessor virtual machines.

How VMware FT keeps two VMs in lockstep (22:28)

VMware FT builds on virtual machine monitors (hypervisors), which let one physical computer run several guest operating systems. The scheme runs an identical primary and backup virtual machine on two separate physical machines. When a client packet arrives, the primary's virtual machine monitor delivers it to the guest as a simulated interrupt and also forwards a copy to the backup's monitor, which fakes the same interrupt for the backup. Both replicas compute the same reply, but only the primary's monitor is allowed to actually send it onto the network; the backup's copy is silently dropped. If the backup stops hearing from the primary, it "goes live": it stops waiting for logged events, starts producing its own output, and claims the primary's network identity so future client traffic reaches it.

Sources of nondeterminism and the logging channel (34:36)

For the replay to work, every point where execution could diverge has to be logged. The main sources are external inputs (packet contents and the exact instruction at which the arrival interrupt occurs), and instructions whose results depend on something other than memory, such as reading the time of day, a unique processor ID, or a random number. The primary executes these and sends the instruction number and the result to the backup, which intercepts the same instruction and substitutes the logged answer instead of executing it. Special CPU support lets the monitor tell the hardware to interrupt after an exact number of instructions, which is how timer interrupts are made to land at the same instruction count on both replicas. The backup is deliberately kept from running ahead of the primary: it only executes up to the next event it has buffered from the primary.

The output rule (57:55)

A naive design lets a client see a reply from the primary before the backup has recorded the request that produced it. If the primary then crashes and that log entry is lost, the backup takes over not knowing the operation happened, and can compute a conflicting duplicate reply, an unacceptable divergence for unmodified client software. The fix is the output rule: the primary's monitor withholds any outbound packet until the backup has acknowledged the log entry for the input that generated it. This guarantees that any reply a client actually receives corresponds to a request the backup also knows about. A related wrinkle is that a failover can still cause a duplicate reply to be sent, but because primary and backup share identical TCP state, the duplicate carries the same sequence number and the client's TCP stack silently discards it.

Split brain and the test-and-set tiebreaker (1:14:03)

The lecture closes on network partitions: if the primary and backup can each reach clients but not each other, both may conclude the other is dead and try to go live, corrupting the service. VMware FT resolves this by requiring either replica to win a test-and-set operation against an external server before going live; only one of them can hold the flag. Since a computer can never distinguish "the other replica died" from "the network to it is broken," this test-and-set step is required on every failover, not just during partitions, which makes the test-and-set server itself a critical dependency that, in practice, needs its own fault tolerance.

Before you watch

• Review how GFS achieves fault tolerance through application-level replication of chunks, since this lecture contrasts that approach with VMware FT's much lower-level, whole-machine replication.
• Be comfortable with the basic idea of a virtual machine monitor (hypervisor) running one or more guest operating systems on a single physical machine.
• Know what TCP sequence numbers and duplicate detection are, since the lecture relies on them to explain why a duplicated reply after failover is harmless.

Check your understanding

1. Why can replication never protect against a bug in the software being replicated?
2. What is the key difference between state transfer and replicated state machine replication, and why does the lecture say replicated state machine is usually more efficient?
3. Why does VMware FT restrict itself to uniprocessor virtual machines rather than multi-core ones?
4. Explain the problem the output rule prevents, and how it prevents it.
5. Why must both the primary and the backup contact the test-and-set server before going live, even when there is no network partition?