Lecture 15: Big Data: Spark

What this lecture covers

Spark is presented as an evolutionary successor to MapReduce that generalizes map and reduce into a full multi-step data flow graph, making it much better suited to iterative algorithms like PageRank than chaining together separate MapReduce jobs. The lecture asks how Spark represents a computation before running it, how it executes that representation efficiently across a cluster, and how it recovers from a worker failure without restarting from scratch.

Using a live walkthrough of a small PageRank program run in the Spark shell, it shows that Spark builds a lineage graph of transformations lazily and only executes when an action like collect is called, distinguishes cheap per-record "narrow" transformations from expensive "wide" transformations that require shuffling data across the network, and explains why Spark's immutable, deterministic RDDs make failure recovery by recomputation practical. After watching, you should be able to explain lazy evaluation of a lineage graph, tell narrow from wide transformations, and describe how checkpointing avoids expensive full recomputation after a worker fails.

Key ideas

• Lineage graph: the sequence of transformations a Spark program builds is recorded as a graph of dependencies (an RDD's "recipe"), not executed until an action is called.
• Lazy execution and actions: transformations like map, distinct, and join just extend the lineage graph; an action such as collect triggers Spark to compile and actually run the graph.
• RDD (resilient distributed dataset): an immutable, partitioned dataset produced by a transformation; because it's immutable and transformations are deterministic, any lost partition can be recomputed exactly.
• Narrow transformations: operations like map that process each record independently, requiring no communication between workers, and can be chained together as local function calls on each record.
• Wide transformations: operations like distinct, groupByKey, join, and reduceByKey that need to bring together all records sharing a key, requiring a network shuffle and forming a computation barrier.
• Persist/cache: explicitly telling Spark to keep an RDD in memory (or on HDFS) across iterations, avoiding recomputing it from scratch on every use, as the lecture does with the links RDD reused across PageRank iterations.
• Failure recovery by recomputation: if a worker fails, Spark recomputes only the lost partitions by re-running the lineage graph, rather than treating fault tolerance as a database-style no-data-loss guarantee.
• Checkpointing: because wide transformations discard their inputs once consumed, recovering a lost partition after a wide transformation can require recomputing the whole graph on every worker; periodically saving intermediate output to HDFS avoids this.

Walkthrough

Why PageRank motivates Spark (2:14)

The lecture introduces PageRank as a classic example that MapReduce handles poorly, since it requires iteration and MapReduce has no native support for loops. Running PageRank as a sequence of MapReduce jobs works but forces every iteration to read from and write back to GFS, adding heavy file I/O. The PageRank algorithm itself is explained briefly: each page's rank estimates the probability a random clicking user ends up there, computed by repeatedly pushing a page's rank fraction to the pages it links to.

Building the lineage graph line by line (9:26)

Working in the Spark shell against a tiny three-page example, the lecture shows that reading a file, mapping each line into a from/to URL pair, and calling distinct and groupByKey do not process any data immediately; each call only extends a lineage graph. Calling collect at an intermediate point forces execution just to inspect an early result, revealing that nothing actually runs until an action is invoked. This distinction between building the recipe and running it is treated as the core of Spark's programming model.

Persisting data for iteration (22:35)

Because the per-page link list is reused every iteration of the PageRank loop, the lecture shows the need to explicitly cache (persist) it in memory; otherwise Spark would recompute the read, map, and distinct steps from scratch on every use. Each loop iteration then joins the current ranks with the links, computes each page's contribution to the pages it links to, and sums contributions per page with reduceByKey, appending new nodes to the lineage graph rather than mutating any variable.

Executing narrow versus wide transformations (38:58)

Once an action triggers execution, Spark assigns each worker a partition of the HDFS input and streams records through as many narrow transformations as possible without any network communication, since each depends only on the current record. Wide transformations like distinct, groupByKey, and join instead need to shuffle data by key across workers, forming a barrier where all preceding work must finish first. The lecture notes that Spark can sometimes skip an expected shuffle when it recognizes data is already partitioned correctly from an earlier wide transformation, an optimization only possible because the whole lineage graph is visible before execution starts.

Failure recovery and the problem with wide dependencies (50:13)

Spark's basic recovery strategy is to recompute whatever a failed worker was responsible for, since HDFS input is already replicated and safe. This works cleanly for narrow transformations, but wide transformations discard their input once consumed by later stages, so recovering a lost partition after a wide transformation can require recomputing that stage on every worker back to the start. The lecture demonstrates this problem with a graph containing a wide dependency partway through, showing how a single failed worker could otherwise force a full recomputation across the entire cluster.

Checkpointing as the fix (58:21)

To avoid that cost, Spark supports checkpointing: explicitly saving the output of a chosen transformation to HDFS so that recovery can read it back instead of recomputing from the beginning. The lecture suggests checkpointing the ranks periodically, for example every tenth PageRank iteration, trading the cost of extra disk writes against the cost of recomputation after a failure. It's left as an open question whether cache alone (in-memory only) offers the same reliability guarantee as an HDFS checkpoint, since cached data can be evicted if a worker runs low on memory or is lost.

Where Spark fits and its limits (1:03:29)

The lecture closes by placing Spark in context: it is built for batch processing of large, already-available datasets, not for transactional workloads like bank transfers or live shopping carts, and the original design has nothing to say about streaming input, which the separate Spark Streaming project addresses. The emphasis on deterministic, immutable RDDs is tied directly to failure recovery: it's what makes recomputing a lost partition safe and correct, in contrast to older distributed shared-memory systems that allowed mutable, non-deterministic state and struggled to recover cleanly at scale.

Before you watch

• Be familiar with MapReduce's map/reduce/shuffle model, since the lecture frames Spark as fixing MapReduce's lack of native iteration support.
• Recall how GFS/HDFS shards and replicates files, since Spark's partitioning and fault tolerance both build directly on that.
• No prior exposure to PageRank is required; the lecture explains the algorithm before using it as the running example.

Check your understanding

1. What does it mean that Spark's transformations are "lazy," and what actually triggers execution of a lineage graph?
2. Why is distinct (or groupByKey, or join) a wide transformation while map is a narrow transformation?
3. Why does caching the links RDD matter for the performance of the PageRank loop?
4. Why can a single worker failure force recomputation across many workers when the lineage graph includes a wide transformation, and how does checkpointing fix that?
5. Why does Spark's reliance on immutable, deterministic RDDs matter specifically for failure recovery at large cluster scale?