Lecture 9: Distributed Data-Parallel Computing Using Spark

What this lecture covers

This lecture moves the course from parallelism within a single chip to parallelism across a cluster of separate machines, each running its own operating system. It asks how you program hundreds of thousands of cores spread across many nodes, when nodes and networks can fail and disk I/O bandwidth becomes the bottleneck instead of compute. It builds directly on the data-parallel primitives (map, reduce) from the previous lecture, applying them to a new setting where fault tolerance and data locality matter as much as raw parallelism.

The lecture works through warehouse-scale computer organization, message passing between separate address spaces, distributed file systems for durable storage, the MapReduce programming model with a worked example, and then motivates Spark's resilient distributed dataset (RDD) abstraction as a faster, memory-centric alternative. The lecture runs out of time before finishing Spark's optimization story, so the discussion of RDD fusion is left incomplete, to be continued in a later class. After watching, you should be able to explain why clusters are needed for very large data, how MapReduce achieves fault tolerance, and what problem RDDs solve that MapReduce does not.

Key ideas

• Why use a cluster: processing hundreds of terabytes of data is bounded by storage I/O bandwidth; a single node might take 23 days, while a thousand nodes with proportional disk bandwidth can do it in about 33 minutes.
• Warehouse-scale computers: large data centers (Google, Facebook, Amazon-style) are organized into racks of 20-40 servers connected by a top-of-rack switch, with rack-to-rack bandwidth traditionally much lower than a node's own local disk bandwidth, though modern networks have closed much of that gap.
• Message passing: since nodes run separate operating systems and do not share an address space, communication happens via explicit send/receive calls, which act as their own synchronization but can deadlock if a message never arrives.
• Distributed file system: systems like the Google File System (GFS) and its open-source counterpart HDFS split large files into 64-256 MB blocks, replicate each block across multiple racks for durability, and use a master (name) node to track where replicas live.
• MapReduce: a programming model built from a mapper function (called once per input record, emitting key-value pairs) and a reducer function (called once per unique key, combining all its values); the middle step, grouping and sorting all values for the same key to the same reducer, is often called shuffle.
• Fault tolerance in MapReduce: since map and reduce are side-effect-free functional operations, failed tasks can simply be re-run from their inputs; a job scheduler tracks node heartbeats, restarts failed mapper or reducer tasks, and mitigates slow ("straggler") machines by racing duplicate copies of a task.
• Limits of MapReduce: its linear map-then-reduce structure makes iterative algorithms (like PageRank) and ad-hoc interactive queries inefficient, since every iteration requires a full distributed file system read and write.
• Spark's RDD abstraction: a resilient distributed dataset is a read-only, ordered, immutable collection created either from persistent storage or by applying a transformation (map, filter, and similar) to another RDD; the recorded sequence of transformations that built an RDD is its lineage, which lets Spark recompute lost data instead of storing costly replicated logs.

Walkthrough

Why distributed computing, and the warehouse-scale computer (1:07)

The lecture opens by framing distributed computing as the answer to a different bottleneck than earlier lectures: I/O bandwidth to storage rather than compute or memory bandwidth. Processing hundreds of terabytes of log data from a large website is shown to take 23 days on one node but about 33 minutes across a thousand nodes, because the aggregate disk bandwidth scales with the number of machines. This motivates the idea of a warehouse-scale computer, a data center's racks of commodity servers networked together and treated as a single large computer, an idea credited to Luiz Barroso, that must be jointly optimized for compute, networking, power, and cooling.

Node and rack hardware, and the bandwidth hierarchy (8:09)

Each rack holds 20-40 servers connected through a top-of-rack switch, with the number of servers per rack limited by available power. A typical node has two CPU sockets (16-32 cores each), 128 GB to a few terabytes of DRAM reachable at roughly 100-200 GB/s, and solid-state storage of 10-30 TB. The lecture highlights that network bandwidth between racks was historically far lower than local disk bandwidth (a tenth of a gigabyte per second in early clusters versus local disk), though modern data-center networks (1-2 GB/s within a rack) have narrowed that gap, changing what it means to fetch data from a remote node versus locally. Since each node runs its own operating system with a separate address space, cross-node communication uses message passing (send/receive) rather than shared memory.

Storing data durably: the distributed file system (18:16)

Because components in a system with hundreds of thousands of parts fail regularly, data must be stored durably before it can be safely processed. The lecture describes distributed file systems such as Google File System (GFS) and HDFS, designed for large files that are mostly appended to and read, rarely updated in place, matching a log-processing workload. Files are split into large blocks, replicated across multiple racks (so losing one rack's switch does not lose data), with a master node holding the metadata that maps blocks to replica locations; clients consult the master to find replicas and then read or write directly to the data nodes holding them.

The MapReduce programming model (27:26)

Rather than programming distributed computers directly with message passing (via something like MPI, which is powerful but painful and does not handle fault tolerance for in-memory computation), the lecture reintroduces map and reduce as the basis for a higher-level model. A mapper function runs once per input record, here per log line, and emits key-value pairs, for example marking whether a page view came from a mobile client. A reducer function then runs once per unique key across all its associated values, for example summing view counts per client type. Because map does not mutate its input, results can be recomputed safely, which becomes the basis for MapReduce's fault tolerance.

Word count and the shuffle step (34:33)

A word-count example shows one map task per input file block, each producing key-value pairs (word, 1) for its portion of text, and multiple reduce tasks that sum counts per word. The lecture stresses that correctly parallelizing across keys requires all pairs for a given key to reach the same reducer, which requires a large sort or shuffle step between the map and reduce phases, so MapReduce is more precisely "map, group by key, reduce." A scheduler decides where each mapper and reducer task runs, and the lecture notes early systems ran mapper tasks on the node already holding the relevant data block, since network bandwidth was the scarce resource, or used a hash of the key to route reducer inputs.

Fault tolerance and stragglers (44:42)

With many nodes of varying age and speed, some tasks fail and others run slowly. A job scheduler monitors node heartbeats; when a mapper node is declared dead, its tasks are rerun on another node using replicated file system blocks, which is safe because map does not mutate its input. Failed reducer tasks that have not completed are restarted and must refetch their key-value data. Slow ("straggler") machines are handled by launching a duplicate copy of the task elsewhere and using whichever copy finishes first, discarding the other.

Why Spark: memory locality and the RDD abstraction (54:48)

MapReduce's strength, its functional, fault-tolerant model, comes with a cost: every stage reads from and writes back to the distributed file system, which is inefficient given that memory bandwidth vastly exceeds network and disk bandwidth. Citing measurements showing that 97-99.5% of large companies' working sets fit in a modest amount of memory, the lecture motivates keeping intermediate data in memory instead. The risk, losing that data on a power failure or crash, leads to Spark's goal of in-memory, fault-tolerant distributed computing. Spark's core abstraction, the resilient distributed dataset (RDD), is a read-only, ordered, immutable collection created either from persistent storage or by transforming another RDD (such as filter or map); the chain of transformations that produced an RDD, its lineage, lets Spark recompute lost partitions instead of paying for replicated logs or full checkpoints, and the lecture ends mid-explanation of using lineage's dependency structure (narrow versus wide dependencies) to fuse transformations for efficiency, a topic to be completed in a later lecture.

Before you watch

• Review the map, reduce, and other data-parallel primitives from the previous lecture (Lecture 8), since this lecture directly reuses and extends them for a distributed setting.
• Recall the earlier course discussion of message passing versus shared memory, since distributed nodes communicate only through explicit sends and receives.
• No new CUDA or GPU knowledge is required, but understanding why parallel operations need to be side-effect-free (from Lecture 8) helps explain MapReduce's fault-tolerance story.

Check your understanding

1. Why does processing very large datasets favor using many machines over a single fast machine, in terms of the specific resource being scaled?
2. Walk through what happens, step by step, when a mapper node fails partway through a MapReduce job, and explain why this recovery is safe.
3. Why is the shuffle (group-by-key) step necessary between the map and reduce phases, and what would go wrong without it?
4. What specific inefficiency in MapReduce does Spark's in-memory model address, and what data does the lecture cite to justify that design?
5. What is an RDD's lineage, and how does it let Spark avoid the cost of replicating or logging every intermediate result?