Lecture 3: GFS

What this lecture covers

This lecture uses the Google File System (GFS) paper as the first detailed case study in building big, fault-tolerant storage. It starts by explaining, in the abstract, why big storage systems are hard: performance drives sharding, sharding drives failures, failures drive replication, and replication threatens consistency unless you pay for extra coordination. The lecture defines strong consistency using the intuition of a single server processing requests one at a time, then shows with a deliberately bad two-server replication design how naive replication breaks that intuition.

It then walks through GFS's actual architecture: a single master tracking file-to-chunk and chunk-to-server mappings, chunk servers storing 64-megabyte chunks as ordinary Linux files, and a primary/lease scheme for coordinating writes. The lecture pays particular attention to GFS's record-append operation, where its relaxed consistency guarantees show up directly, and closes with a retrospective on GFS's real-world successes and limitations. After watching, you should be able to explain the master's two main tables, trace the steps of a GFS read and a record append, and describe why GFS accepts weaker consistency than a single server would provide.

Key ideas

• Sharding: splitting data across many servers to get parallel read/write throughput, which is usually the starting motivation for big distributed storage systems.
• Split-brain: the dangerous situation where two servers both believe they are primary for the same data and accept conflicting writes, usually caused by a network partition rather than an actual crash.
• Chunk: GFS's unit of storage, a 64-megabyte piece of a file, stored as an ordinary file on a chunk server's local disk and typically replicated three times.
• Master's two tables: one mapping file names to arrays of chunk handles, another mapping each chunk handle to its replica locations, version number, primary, and lease expiration.
• Primary and lease: the master designates one replica as primary for a chunk for a limited time (60 seconds), so writes go through a single coordinator without requiring the master to be reachable on every write.
• Version numbers: used to distinguish up-to-date replicas from stale ones after a crash, and only incremented when the master assigns a new primary.
• Record append: GFS's core write operation, which guarantees that a successful append lands at the same offset in every replica but allows failed appends to leave replicas divergent, meaning different readers can see different data or duplicate records.

Walkthrough

Why big storage systems are hard (2:02)

The lecture sketches a recurring pattern: teams want aggregate performance, so they shard data across many servers; sharding at scale makes failures routine, so they add replication; replication risks replicas drifting out of sync, so achieving strong consistency requires extra communication that costs performance. It defines strong consistency using the mental model of a single-threaded server processing one request at a time, where reads always reflect the most recently completed write, and shows with a worked example why this still has subtleties when writes race each other.

A deliberately bad replication design (10:13)

To motivate GFS's design choices, the lecture walks through a naive two-server replication scheme where writes are sent to both servers independently, with no agreement on ordering. It shows how two servers can apply the same two writes in different orders, leaving them permanently inconsistent, and how routing all reads to a single "primary" server can produce even stranger behavior, such as a value appearing to change when no write occurred, simply because that server failed and reads shifted to the other replica.

GFS's goals and non-goals (15:24)

The lecture explains the context: Google in 2003 needed to store and process vast data sets (web crawls, logs, index files) far bigger than a single disk, shared across many internal applications, with automatic recovery from routine failures. GFS deliberately targeted a single data center, internal use only, and large sequential reads and writes rather than small random-access operations. Notably, the paper accepted weaker-than-usual consistency and a single, unreplicated master, a break from typical academic designs of the time, betting that most applications could tolerate occasional odd behavior.

Master structure and reads (23:40)

The lecture details the master's two tables (file-to-chunks and chunk-to-metadata) and which fields are kept on disk versus only in memory, explaining that the master uses an append-only log plus periodic checkpoints for efficient recovery rather than a full database structure. It then walks through a read: the client asks the master for the chunk handle and server list for a given file and offset, caches that mapping, and reads the requested bytes directly from a nearby chunk server, which simply serves the corresponding local file.

Writes, primaries, and leases (39:51)

Record-append writes are more involved: if no primary exists for a chunk, the master must find replicas holding the current version number, pick a primary among them, increment the version, and grant a lease. The lease mechanism is essential for correctness: because pings can fail due to network partitions rather than real crashes, the master cannot safely appoint a new primary until the old primary's lease has definitely expired, which prevents the split-brain scenario of two primaries processing conflicting writes at once.

How record append actually works (52:13)

The client sends data to the primary and all secondaries, who stage it; once staged everywhere, the client asks the primary to commit the append. The primary picks an offset, tells all replicas to write there, and only reports success if every secondary confirms. Crucially, if any secondary fails to write, the primary reports failure to the client but does not undo the writes that did succeed elsewhere, so replicas can end up permanently different: some containing a record, others not, and successful retries appending the same data again at a new offset, producing duplicates.

Toward strong consistency, and what GFS actually achieved (1:16:04)

Responding to questions, the lecture lists what would be needed to give GFS strong, single-server-like consistency: duplicate-request detection at the primary, a way to force secondaries to actually perform operations rather than fail silently, two-phase commit-style coordination, and resynchronization after a primary crash. It notes that MIT's later labs build exactly this kind of strongly consistent replicated system. The lecture closes with a brief retrospective: GFS was highly successful inside Google and underpinned systems like BigTable and MapReduce, but its single master became a memory and request-rate bottleneck as usage grew, and master failover required slow manual intervention.

Before you watch

• Lecture 1's discussion of scalability, fault tolerance, and consistency introduces the vocabulary this lecture builds on directly.
• Lecture 2's coverage of threads and RPC is useful background, since GFS's client-master-chunkserver interactions rely on those mechanisms.
• Reading the GFS paper before the lecture is recommended, as the lecture assumes familiarity with its structure (figure 1 in particular).

Check your understanding

1. Why does sharding for performance tend to lead, step by step, to a tension between fault tolerance and consistency?
2. What information does the GFS master store only in memory, and what does it store on disk, and why does that split matter for recovery after a crash?
3. Why is a lease necessary for the primary, rather than just having the master directly designate and revoke a primary whenever it likes?
4. In the record-append walkthrough, how can two different clients reading the same file end up seeing the same records in different orders, or with duplicates?
5. What specific changes would be needed to turn GFS's append mechanism into a strongly consistent one, and why does GFS avoid making those changes?