Lecture 7: Fault Tolerance: Raft (2)

What this lecture covers

This is the second Raft lecture, continuing directly from the previous one's discussion of divergent replica logs. It works through the mechanics of how a new leader forces followers' logs to match its own, why Raft restricts who is allowed to become leader (ruling out a "longest log wins" scheme), and a faster way to back up disagreeing logs than one entry at a time. It then covers what state a Raft server must persist to disk and why, how log compaction with snapshots keeps a long-running system's storage and restart time bounded, and closes with a formal definition of linearizability, the correctness standard a replicated service must meet.

After watching, you can explain how AppendEntries's consistency check and backtracking repair a follower's log, state and justify Raft's election restriction, list which pieces of Raft state must be persisted and why, describe how snapshots let a leader stop storing its entire history, and determine whether a given execution history of a service is linearizable.

Key ideas

• Log consistency check: AppendEntries includes the previous entry's index and term; a follower rejects the RPC if its own log disagrees there, forcing the leader to back up and resend.
• Next index and backing up: the leader tracks a nextIndex per follower and decrements it on rejection until it finds a point where the logs agree, then overwrites everything after that point.
• Election restriction: a candidate can win a vote only if its last log entry has a term at least as high as the voter's, or the same term with a log at least as long, which prevents a leader with a shorter but stale log from overwriting already-committed entries.
• Faster backup: a rejecting follower returns the conflicting entry's term (XTerm), the first index of that term (XIndex), and its log length, so the leader can skip back roughly a whole disagreeing term at once instead of one entry per RPC.
• Persistent vs. volatile state: the log, currentTerm, and votedFor must survive a crash because they are the only record of committed operations and of which term already had a leader or a vote cast; commitIndex, lastApplied, and leader-only indices can be safely rebuilt after a restart.
• Log compaction (snapshots): once a log grows past a size threshold, Raft asks the application for a snapshot of its state as of a given log index and discards the log entries before that index, using InstallSnapshot to bring far-behind followers up to date.
• Linearizability: a history of client requests is correct if there is some total order of operations consistent with real-time non-overlap, where every read returns the value of the most recent preceding write in that order.

Walkthrough

Repairing a follower's log after a leader change (0:01)

Working through a concrete example, the lecture shows a new leader for term six sending AppendEntries with the index and term of the entry just before the new one. Followers whose logs disagree at that point reject the RPC, and the leader responds by decrementing its nextIndex for that follower and retrying with an earlier previous-entry reference, repeating until it finds a point of agreement. Once found, the follower deletes everything in its log after that point and accepts the leader's remaining entries. This is safe to do for uncommitted entries because, by construction, anything not held on a majority of servers could not have been reported to a client as committed, so the client will simply resend the request after timing out.

Why elections need a restriction, not "longest log wins" (9:16)

The lecture proposes and rejects a simpler alternative rule where voters favor whichever candidate has the longest log. A constructed scenario shows a server with an older, shorter-looking log at terms 5, 6, 7 while two other servers have already elected new leaders and committed entries at term 8; letting the term-7 server become leader would force it to erase already-committed term-8 entries from the other replicas, violating the requirement never to discard committed data. Raft's actual rule instead lets a voter approve a candidate only if the candidate's last log entry has a higher term than the voter's own last entry, or an equal term with a log at least as long, which in this example correctly blocks the outdated server from winning.

Faster log backup with XTerm and XIndex (20:45)

Backing up one entry per RPC is slow when a follower has missed a long run of entries, for example after being offline for a long time or trapped in a stale minority partition. The lecture describes an optimization (not fully specified in the paper) where a rejecting follower's reply includes the conflicting entry's term (XTerm), the index of the first entry with that term (XIndex), and the follower's log length if it has no entry at all at that position. Depending on whether the leader itself has any entries for XTerm, it can either jump nextIndex back to XIndex, back up to just past its own last entry of that term, or back up to the follower's reported log length, in each case skipping potentially many entries in one step rather than one at a time.

Persistent state and why it must survive a crash (32:02)

Raft's Figure 2 marks the log, currentTerm, and votedFor as persistent because they are needed to safely recover from a crash, including a simultaneous power failure of the whole cluster where servers cannot simply be replaced with blank ones. The log must persist because it is the only record from which application state can be rebuilt. votedFor must persist so a server does not vote twice in the same term after a crash and restart, which could elect two leaders. currentTerm must persist so surviving servers do not "forget" a term number and accidentally reuse an already-used one, causing confusion between two different leaders both claiming the same term. The lecture also discusses the real cost of persistence: a synchronous write to a spinning disk can take about 10 milliseconds, capping throughput at roughly 100 requests per second unless you use faster storage (SSD, battery-backed DRAM) or batch multiple client requests into a single persisted write.

Log compaction with snapshots (49:13)

Without compaction, a long-running Raft log grows without bound and a restarting server would have to replay potentially millions of entries to rebuild its state. Because application state is often much smaller than the log that produced it, Raft periodically asks the application to produce a snapshot of its state as of a specific log index, then discards the log entries up to that point, keeping only the snapshot plus the remaining log. This introduces a complication: if a follower has fallen so far behind that the leader no longer has the log entries it needs, AppendEntries cannot bring it up to date, so the leader instead sends the snapshot itself via an InstallSnapshot RPC before resuming normal log replication.

Defining correctness: linearizability (1:04:46)

The lecture closes by formalizing what "correct" means for a replicated service. A history of client requests, each with a start and end time, is linearizable if there exists a total order of the operations such that the order respects real-time (an operation that finished before another started must come earlier) and every read returns the value written by the most recent preceding write in that order. Two worked examples show the technique: draw the ordering constraints implied by non-overlapping timing and by which write each read observed, then check whether those constraints can be satisfied by some total order (linearizable) or form a cycle that no order can satisfy (not linearizable), which would indicate a bug in the system being analyzed.

Before you watch

• Watch the previous Raft lecture in this course first, since this one continues directly from its discussion of divergent logs and majority quorums without re-explaining them.
• Be familiar with Raft's Figure 2, particularly the AppendEntries RPC fields and the per-server persistent and volatile state, since the lecture works through it in detail.
• Know what a total order and a partial order are, which is assumed background for the linearizability definition at the end.

Check your understanding

1. Walk through how a leader's nextIndex for a follower changes across a sequence of rejected AppendEntries RPCs, and what happens once agreement is found.
2. Why would a "longest log wins" election rule be unsafe, and how does Raft's actual election restriction avoid the problem?
3. What information does the faster backup scheme add to an AppendEntries reply, and how does it let the leader skip back more than one entry at a time?
4. Why must votedFor and currentTerm be persisted to disk, but commitIndex and lastApplied do not need to be?
5. Given a short history of overlapping reads and writes on a single key, how would you determine whether it is linearizable?