Lecture 5: Go, Threads, and Raft

What this lecture covers

This is a teaching-assistant session focused on practical Go concurrency skills needed for the course's Raft labs, rather than new distributed systems theory. It works through goroutines, mutexes, condition variables, and channels with runnable examples, then walks through two real bugs students commonly introduce in a Raft implementation, and finishes with concrete debugging techniques.

After watching, you can write goroutines and locking code that avoids the most common Go concurrency mistakes, understand why locks must protect invariants rather than just individual variables, know when to reach for condition variables instead of busy-waiting, and use DPrintf, SIGQUIT stack dumps, and the -race flag to find deadlocks and data races in your own lab code.

Key ideas

• Closures over loop variables: a goroutine launched inside a for loop that reads the loop variable directly will often see a stale or changed value; pass it as an explicit function argument instead.
• Locks protect invariants, not just variables: a lock must be held for an entire operation that needs to appear atomic (such as a bank transfer), not just for the individual reads and writes inside it.
• Condition variables over busy-waiting: instead of polling a shared flag in a tight loop or with an arbitrary time.Sleep, a goroutine should Wait() on a condition variable and be woken by Broadcast() when the state it cares about changes.
• Never hold a lock across an RPC call: doing so risks deadlock when two peers each hold their own lock and wait on an RPC to the other, and it blocks unrelated work while a slow RPC is in flight.
• Re-check assumptions after an RPC returns: state may have changed while a lock was released for the call, so code must verify it is still in the same term and role before acting on the RPC's result.
• DPrintf for toggleable debug logging: a wrapper around log.Printf that can be turned on or off, used to narrow down where in the code execution gets stuck.
• -race is a detector, not a proof: it flags data races it observes during a run, but a clean run does not guarantee the code is race-free.

Walkthrough

Closures, goroutines, and the loop-variable trap (2:02)

The lecture opens by clarifying that concurrency in the labs is about expressing ideas cleanly, not chasing CPU performance, so students should favor simple, coarse-grained locking over fine-grained optimization. It introduces closures: an anonymous function passed to go can read and mutate variables from its enclosing scope, which is useful for spawning several RPCs in parallel, for example asking all Raft peers for a vote at once. The key gotcha is demonstrated directly: if a goroutine spawned inside a loop reads the loop variable itself rather than receiving it as an argument, the loop may have already advanced the variable by the time the goroutine runs, producing garbled output like several goroutines seeing the same final index instead of a spread of values.

Running work periodically and shutting it down cleanly (6:05)

A simple pattern for periodic background work is a goroutine looping forever with a time.Sleep between iterations. To stop it cleanly, for example when a Raft instance is killed, a shared boolean (guarded by a lock, or exposed through a Killed() method) is checked each iteration so the goroutine exits instead of running forever after shutdown. The lecture notes a subtlety of the Go memory model: without any synchronization primitive guarding a shared variable, the compiler is permitted to optimize a read out of a loop entirely, so a goroutine can fail to ever observe another thread's write. The practical rule given is to always hold a lock around any read or write of data shared across goroutines.

Mutexes, atomicity, and protecting invariants (14:27)

A counter incremented by many concurrent goroutines without a lock loses updates because increments are not atomic; wrapping the read-modify-write in a sync.Mutex (often with defer mu.Unlock()) fixes it. The lecture then extends this with a bank example transferring money between two accounts: even though every individual read and write is done under the lock, a concurrent auditor thread can still observe the invariant "Alice plus Bob equals a constant" being violated, because the decrement and increment are two separate locked sections rather than one atomic one. The lesson is that locks exist to protect invariants over a whole operation, not just to make individual variable accesses safe; the fix is to hold the lock across the entire transfer.

Condition variables replace busy-waiting (23:34)

Using the Raft vote-counting scenario, the lecture shows a naive way to wait for enough votes: a loop that repeatedly locks, checks the count, and unlocks, which burns a full CPU core. Adding a fixed sleep helps but introduces an arbitrary magic constant. The proper tool is a condition variable tied to the same lock: the waiting goroutine calls Wait(), which atomically releases the lock and parks the goroutine, while any goroutine that changes the shared state calls Broadcast() after making its change, waking waiters to recheck their condition. The recommended pattern for this course is to always check the condition in a for loop around Wait(), and to always use Broadcast() rather than Signal().

Channels are synchronous, not queues (34:37)

Unbuffered channels in Go have no internal storage: a send blocks until a receive is ready, and vice versa, exchanging data synchronously at that point. A demo shows a send taking a full second because the receiver was asleep that long, and another shows a goroutine deadlocking (or silently hanging) because it tries to send and receive on the same channel with no other goroutine to pair with. Buffered channels relax this only until their capacity fills. The lecture recommends using channels mainly for producer-consumer patterns or as a substitute for sync.WaitGroup, and otherwise preferring mutexes and condition variables, which the presenter finds easier to reason about.

A real Raft bug: holding a lock across an RPC call (46:54)

A worked example shows two Raft peers each starting an election, acquiring their own lock, and calling RequestVote on the other while still holding that lock. Each peer's RPC handler then tries to acquire the same lock to process the incoming request and blocks forever, producing a deadlock the Go race and deadlock detector catches. The fix is to never hold a lock while an RPC is outstanding: the needed term is captured into a local variable before the call, the lock is released before calling, and re-acquired only to process the reply. A second bug shows why this matters further: because votes are counted after an unlocked RPC round-trip, the peer's term or role may have changed in the meantime (for example, it may have already voted for someone else on a higher term), so code must re-check the term and candidate status before declaring victory, rather than trusting state captured before the RPC.

Debugging tools: DPrintf, stack dumps, and the race detector (59:14)

The final section demonstrates a debugging workflow on a stuck test: add DPrintf statements progressively deeper into the call chain to narrow down where execution stalls, discovering in one case that a goroutine tries to re-acquire a lock it already holds. Pressing Ctrl+\ sends SIGQUIT to a hung Go program, which prints every goroutine's stack trace, letting you see exactly where each goroutine is blocked. Running tests with go test -race surfaces genuine data races (though a clean run is not proof of correctness), and the lecture works through fixing an unprotected shared variable and, in a trickier case, another instance of a lock held across an RPC call that silently prevented heartbeats from being received. It closes by pointing to a script for running the test suite many times in parallel to catch flaky, timing-dependent bugs before submission.

Before you watch

• Have completed or read the Raft election and RPC code from lab 2A, since most examples build directly on attemptElection and RequestVote.
• Be familiar with basic Go syntax: goroutines (go), sync.Mutex, and sync.WaitGroup, which this lecture assumes rather than introduces from scratch.
• Review the assigned Go memory model reading mentioned at the start, since the lecture references it when explaining why unsynchronized reads can be optimized away.

Check your understanding

1. Why does spawning goroutines inside a loop that read the loop variable directly produce incorrect results, and how do you fix it?
2. What is the difference between "locks protect shared variables" and "locks protect invariants," using the bank transfer example?
3. Why is busy-waiting with a fixed sleep interval worse than using a condition variable, beyond wasting CPU?
4. Explain, using the two-peer election example, exactly how holding a lock across an RPC call causes a deadlock.
5. What can the -race flag tell you, and what can it not guarantee about your code?