Lecture 9: Zookeeper's API, Mini-Transactions, and Chain Replication

What this lecture covers

This lecture finishes the Zookeeper discussion by examining its actual API and two worked examples—an atomic counter and a distributed lock—then switches to a different replication architecture: chain replication and its optimization, CRAQ. It follows directly from the previous lecture's introduction to Zookeeper's consistency guarantees, and it sets up chain-style replication as an alternative to Raft that the course will keep coming back to.

After watching, you should be able to explain why Zookeeper needs a purpose-built API rather than a plain key-value store, walk through the "mini-transaction" pattern for atomic read-modify-write, and describe how chain replication places writes and reads on different servers to spread load, plus why it is vulnerable to split brain without an external configuration manager.

Key ideas

• Znodes and their types: Zookeeper's hierarchical namespace holds regular, ephemeral (deleted when the creating client's session dies), and sequential (guaranteed unique, ascending numeric suffix) znodes.
• Exclusive, versioned operations: create fails if the path exists; setData and delete can take a version number so the write only applies if nothing else has modified the file since it was read.
• Mini-transactions: reading a value and version, then conditionally writing a new value only if the version hasn't changed, gives atomic read-modify-write without full database transactions—useful for counters and test-and-set.
• The herd effect: a naive single-file lock, where every waiting client watches the same file, causes O(n²) total work as n clients contend; Zookeeper's paper describes a scalable lock using sequential files where each client only waits on its immediate predecessor.
• Zookeeper locks aren't threading locks: because a lock holder can crash mid-update, releasing the lock doesn't guarantee the protected data is in a consistent state, unlike a mutex in a single process.
• Chain replication: writes flow from a head server through each replica in turn to a tail, which sends the client's acknowledgment; reads go directly to the tail, which always has the latest committed state.
• Chain replication is linearizable and needs no leader broadcast: unlike Raft, the head only forwards each write once (down the chain, not to every replica), and consistency after failure is simple because a committed write must have reached every node.
• Split brain risk: chain replication itself has no partition tolerance; it requires an external, Raft/Paxos-backed configuration manager to be the sole authority on chain membership.

Walkthrough

Recap and API motivation (0:00)

The lecture recaps Zookeeper's guarantees: writes are linearizable, reads can be stale, and FIFO client order applies to each client's own operations. It then surveys real uses—implementing the test-and-set primitive VMware FT needed, publishing configuration like the current master's address, electing a master, and letting workers register or pull work items—to motivate why a general-purpose coordination API is valuable beyond a plain put/get store.

The znode API (5:03)

Zookeeper looks like a filesystem: a hierarchical namespace of znodes, which can be regular, ephemeral, or sequential. The create RPC is exclusive (it tells the caller whether they were first), exists can set a watch atomically with the existence check, and setData/delete accept an optional version for conditional writes.

Atomic counters and mini-transactions (12:13)

A naive get-then-put increment on a key-value store is not atomic and breaks under concurrent clients or stale reads. The fix is a retry loop: read the value and version, compute the new value, then call a conditional setData that only succeeds if the version is unchanged; on failure, loop and try again. This "mini-transaction" pattern achieves atomicity for one read-modify-write without general transactions, and the same pattern implements test-and-set. Under very high contention it degrades toward O(n²) total work across all waiting clients, which the lecture suggests fixing with exponential backoff.

A simple lock and the herd effect (27:38)

A first attempt at a lock uses one ephemeral file: try to create it, and if that fails, call exists with a watch and wait for deletion. Careful reasoning about interleavings shows this is correct, but it suffers from the herd effect: releasing the lock notifies every waiting client, all of whom retry, so total work across n clients scales as n².

The scalable lock (36:59)

Zookeeper's paper describes a better lock built from sequential files: each client creates a sequential file, lists the directory, and wins the lock if its file has the lowest number; otherwise it watches only the next-lower-numbered file and re-lists when that file is deleted (since that owner may have died rather than released). Because each release only wakes the one client waiting on it, the cost per acquire/release is constant rather than proportional to the number of waiters. The lecture also notes these locks don't give the same atomicity as threading mutexes, since a crashed holder can leave protected data partially updated.

Chain replication (57:46)

Chain replication arranges replicas in a line: writes go to the head, propagate node by node, and the tail acknowledges the client; reads go straight to the tail. Because a write is only exposed once it has passed through every node, the system is linearizable, and failure recovery is simpler than Raft's: a head failure just promotes the next node, a tail failure promotes its predecessor, and a middle failure means dropping that node and having its predecessor resend recent writes to the new successor. Chain replication also spreads load, since the head only sends each write once (down the chain) rather than to every replica in parallel, unlike a Raft leader.

Chain replication's tradeoffs and the need for a configuration manager (1:07:00)

Chain replication has no built-in defense against network partitions: if two adjacent nodes can't reach each other but both can reach a monitor, each might wrongly declare itself the surviving head or tail, causing split brain. In practice it's always paired with an external, Raft/Paxos-backed configuration manager that is the sole authority on which servers make up each chain, so there's never disagreement about membership. The lecture notes chain replication's weakness is that a single slow replica delays every write, since all writes have to pass through every node, whereas Raft only needs a majority—a tradeoff worth keeping in mind before the next lecture introduces CRAQ.

Before you watch

• Review Lecture 8's coverage of Zookeeper's consistency guarantees (linearizable writes, FIFO client order, stale reads).
• Recall Lab 3's key-value put/get interface, which the lecture uses as a baseline for why a richer API is needed.
• Be familiar with how VMware FT and GFS use a test-and-set or master-election primitive, since those motivate several examples.

Check your understanding

1. Why does a plain get-then-put increment fail under concurrent clients, and how does Zookeeper's conditional setData fix it?
2. Walk through why the single-lock-file scheme suffers from the herd effect, and explain how the sequential-file lock avoids it.
3. Why don't Zookeeper's locks provide the same atomicity guarantee as a threading mutex?
4. In chain replication, why is it safe for the next node to simply take over as head when the current head fails?
5. Why does chain replication need an external configuration manager instead of letting chain members decide liveness themselves?