Lecture 6: Fault Tolerance: Raft (1)

What this lecture covers

This lecture opens the course's multi-lecture treatment of Raft by first diagnosing a weakness shared by every replication scheme covered so far: MapReduce, GFS, and VMware FT all quietly depend on a single entity (a master or a test-and-set server) to decide who is primary, which is itself a single point of failure. It then builds up, from first principles, why majority-vote quorum systems solve this without needing an unbreakable network or a human tiebreaker, and walks through Raft's overall architecture: the log, the leader, and the request/commit flow between clients, the application layer, and the Raft library.

After watching, you can explain why split brain arises when a network partitions, why an odd number of servers and majority votes prevent two leaders from being elected in the same term, how a client request flows through Start() and the apply channel to become a committed log entry, and how randomized election timers make Raft elections converge on one winner most of the time.

Key ideas

• Hidden single point of failure: MapReduce's master, GFS's master, and VMware FT's test-and-set server are each a single decider tucked inside an otherwise replicated system.
• Split brain: if a network partition lets each side believe the other is dead, both sides can independently decide they are primary, corrupting the service.
• Majority (quorum) systems: with 2F + 1 servers, any operation requires agreement from more than half; at most one partition can ever contain a majority, and any two majorities must share at least one server.
• Raft has a leader: unlike the original Paxos, Raft elects a leader so that, in the common case, one round of RPCs is enough to replicate a client request.
• The log: it orders client operations consistently across replicas, holds entries a follower cannot yet execute because they aren't confirmed committed, lets a leader retransmit missed entries, and lets a crashed server rebuild its state on restart.
• Terms: each term has at most one leader; servers use term numbers, not leader identity, to detect when a new leader has taken over.
• Randomized election timeouts: because a fixed timeout would let servers restart elections in lockstep and split votes forever, each server picks a fresh random timeout so, usually, one candidate's timer fires first and wins before others even try.

Walkthrough

The hidden single point of failure in prior systems (0:00)

The lecture reviews MapReduce, GFS, and VMware FT and points out a common pattern: each relies on one component, a master or test-and-set server, to make the critical decision of who is primary after a failure. A lone decider cannot disagree with itself, which is convenient, but it is also a single point of failure, so the "real" fault tolerance problem has just been pushed into a smaller box rather than solved.

Why split brain happens, and why the obvious fixes fail (2:03)

Using a toy two-server, replicated test-and-set service, the lecture shows that requiring clients to always reach both replicas defeats the purpose of replication (you now need both to be up), while letting a client proceed after reaching just one replica is unsafe: a network partition can let one client reach server one and think it holds the lock while another client reaches server two and thinks the same, since each side reasonably but wrongly concludes the other server is dead. Historically this was handled either with networks engineered to essentially never fail, or by paging a human to manually decide which replica is really alive; both are expensive or slow.

Majority votes and quorum systems (10:10)

The insight that makes automated failover safe is majority voting over an odd number of servers. Because a majority is always counted against the total number of servers, not just the live ones, at most one side of any network partition can ever assemble a majority, so split brain is structurally impossible. A more subtle but equally important property is that any two majorities, drawn at different times from the same server set, must overlap in at least one server, which is what lets a new leader learn about everything a previous majority agreed to. A system with 2F + 1 servers tolerates F failures. This idea, independently arrived at around 1990 in Paxos and View Stamped Replication, underlies Raft, which the lecture notes is architecturally closer to View Stamped Replication than to Paxos.

Raft's architecture: log, leader, and the client request flow (18:16)

Raft is a library linked into each replica alongside application code (for example a key-value server). A client sends a request to the application layer of the current leader, which hands it to Raft via Start() rather than executing it immediately. Raft's leader sends AppendEntries RPCs to replicate the entry; once it hears back from a majority, the entry is committed, and only then does Raft notify the application layer (through an apply channel) so it can execute the command and reply to the client. Followers cannot execute a received entry until a later message tells them it is committed, since it might still be discarded. The log serves several roles beyond ordering: it holds tentative, uncommitted entries on followers, lets a leader resend entries a follower missed, and lets a crashed-and-restarted server replay its persisted log to rebuild state.

Leader election and randomized timeouts (48:04)

Each server keeps an election timer; if it hears nothing from a current leader before the timer expires, it assumes the leader is dead, increments the term, votes for itself, and requests votes from the others. Because each server casts only one vote per term, and winning requires a majority, at most one candidate can win a given term. A won election is communicated only implicitly: seeing an AppendEntries RPC for a new term tells other servers that somebody won it, and receiving it resets their election timers. A fixed timeout is dangerous because all followers of a dead leader would restart elections simultaneously and split the vote every round; Raft instead has each server pick a fresh random timeout on every reset, so with high probability one candidate's timer fires meaningfully before the others', letting it collect a full round of votes before a competing election starts. The lecture discusses tuning this timeout: the minimum must clear several heartbeat intervals to tolerate dropped packets, and the range must be wide enough that the gap between the first and second timers reliably exceeds one RPC round trip.

Reconciling divergent logs after a crash (1:08:21)

The hard cases in Raft arise specifically around crashes, since a live leader with a majority simply dictates the log and followers obey. The lecture works through concrete log diagrams (recording term numbers per slot) to determine which configurations of divergent logs across replicas are even reachable through valid sequences of crashes and elections, and which are not. For entries known to be present on a majority of replicas, Raft cannot discard them, since they might already have been committed and reported to a client. For entries present only on a former leader that crashed before confirming a majority, Raft is free to overwrite or drop them, since there is no way they could have been reported as committed. The general rule that emerges is that a new leader, once elected with a majority, forces all replicas' logs to match its own from that point onward, which is safe precisely because that leader's majority is guaranteed to overlap with any previous committed majority.

Before you watch

• Review the primary-backup and test-and-set mechanisms from the VMware FT lecture, since this lecture explicitly builds on the weakness that scheme has as motivation for majority voting.
• Be familiar with the general shape of the Raft paper's Figure 2 (state each server keeps, and the rules for RequestVote and AppendEntries), since the lecture refers to it directly when discussing persistence and log rules.
• Understand basic RPC request/response mechanics, since the whole discussion of timing and timeouts assumes familiarity with round-trip communication between servers.

Check your understanding

1. Why is a single test-and-set server or master, even inside an otherwise replicated system, still a single point of failure?
2. Explain why requiring a client to always reach both replicas, and why letting it proceed after reaching just one, both fail to prevent split brain.
3. Why does a majority always have to be counted against the total number of servers rather than just the live ones?
4. Walk through the client request flow from Start() to the apply channel: at what point is a command considered committed, and why can't a follower execute it earlier?
5. Why does Raft use randomized rather than fixed election timeouts, and what could go wrong with a fixed timeout?