Lecture 8: Data-Parallel Thinking

What this lecture covers

This lecture answers a different question from the systems-focused lectures before it: instead of asking how to implement parallelism efficiently on hardware, it asks how to think about and express algorithms so they contain enough parallelism to use a machine with hundreds of thousands of execution contexts. It follows directly from the CUDA and GPU architecture lecture, using the fact that a modern GPU has roughly 163,000 execution contexts as motivation: programs need to expose parallelism at that scale, not just parallelize a single loop.

The lecture introduces a small set of data-parallel primitives, map, fold, scan, gather, and scatter, and works through progressively harder examples: computing a scan efficiently on different kinds of hardware, sparse matrix multiplication, and building a spatial grid data structure for a particle simulation. After watching, you should be able to explain what each primitive does, recognize when a problem can be reformulated in terms of them, and understand why efficient implementations of primitives like scan differ depending on whether you are targeting a few cores, many independent cores, or SIMD lanes.

Key ideas

• Sequences: an ordered collection (arrays in ISPC/CUDA, Seq in Scala, tensors in PyTorch) whose elements can only be touched through specific operations, which prevents programs from creating unintended dependencies between elements.
• Map: applies a function independently to every element of a sequence to produce an output sequence; because each call only sees one input element, map is trivially parallel.
• Fold: reduces a sequence to a single value using a binary combining function; fold can only be parallelized correctly if the function is associative, since a parallel implementation must partition the sequence, fold each part, then combine partial results.
• Scan: produces, for every position, the fold of all elements up to (inclusive scan) or before (exclusive scan) that position; a naive parallel scan does O(n log n) work, while the Blelloch work-efficient algorithm does O(n) work in log n steps using an up-sweep and down-sweep over a combining tree.
• Hardware-dependent implementation choice: the best way to parallelize scan differs by machine, a simple two-processor split-and-merge is best for a couple of cores, the work-efficient O(n) algorithm is best when parallelism vastly exceeds processor count, and a straightforward O(n log n) SIMD scan can be optimal on a single warp because it keeps all SIMD lanes busy.
• Segmented scan: applies scan independently within each subsequence of a sequence-of-sequences (such as per-vertex edge lists or per-document word lists), enabling irregular, nested parallel problems to be flattened into one regular data-parallel computation.
• Gather and scatter: gather reads elements from arbitrary source locations using an index array (densifying scattered data); scatter writes elements to arbitrary destination locations, sometimes combined with an update operation like addition, and is used to implement operations such as histograms.
• Data-parallel libraries in practice: this style of programming underlies real systems, Nvidia's Thrust library exposes map, sort, scan, and segmented scan for CUDA, and Apache Spark's RDDs restrict programs to the same class of operators to get cluster-wide parallelism and fault tolerance.

Walkthrough

Why algorithms need to expose massive parallelism (1:05)

The lecture opens by connecting back to the previous lecture's GPU architecture numbers: with around 163,000 execution contexts on one chip, programs need hundreds of thousands of independent units of work to use the hardware well. Rather than reasoning about dependencies at a fine grain as in earlier assignments, the lecture proposes building programs out of calls to a small set of operations that are already known to have efficient, highly parallel implementations, so that any program built from them inherits that parallelism.

Sequences and the map operation (5:10)

A sequence is introduced as an ordered collection, similar to a NumPy array or C++ vector, but with the key restriction that elements can only be accessed through defined operations rather than arbitrary indexing, which removes a common source of hidden dependencies. Map is shown as the operation most code in the course has already been using without naming it: it takes a function and a sequence and applies the function independently to every element. Because the function only ever sees one element, map has an obvious parallel implementation: partition the sequence across threads, apply the function locally, and concatenate the results.

Fold and the requirement of associativity (13:16)

Fold reduces a sequence to a single value by repeatedly applying a binary function, such as summing an array. The lecture uses class discussion to establish that fold can only be safely parallelized, by splitting the sequence, folding the parts, and combining partial results, when the combining function is associative; a non-associative function like exclusive-or on ordered data can give different answers depending on execution order. Given an associative function, a parallel fold applies it within partitions and then combines the partial results with the same function.

Scan: naive versus work-efficient parallel implementations (19:26)

Scan produces the running fold at every position instead of a single final value. A straightforward divide-and-conquer parallel scan does O(n log n) total work across log n steps, which wastes work compared to the O(n) sequential algorithm. The lecture then derives the Blelloch work-efficient scan, an up-sweep phase that builds partial sums in a combining tree followed by a down-sweep phase that pushes those partials back out to the correct positions, achieving O(n) total work while keeping the log n step count. The lecture notes this algorithm is not fully processor-efficient in practice (not all processors stay busy at every step) but is a standard building block, and it becomes a warm-up CUDA assignment.

Matching the scan algorithm to the machine (32:46)

The lecture argues that the "best" scan implementation depends on the target hardware. With only two processors, splitting the array in half, scanning each half sequentially, and applying the first half's base to the second half is simplest and fast. On a single 32-wide SIMD warp, a direct O(n log n) scan written as five sequential SIMD steps actually beats the work-efficient algorithm, because the work-efficient version leaves SIMD lanes idle and needs more total cycles even though it does less asymptotic work. For larger inputs, CUDA code combines both ideas: scanning within 32-wide warps, then scanning the per-warp partial sums, then distributing the bases back out, layering this recursively for larger sizes.

Segmented scan and sparse matrix multiplication (45:58)

Many real problems are sequences of sequences with varying lengths, such as edges per graph vertex or words per document, where parallelizing only over the outer sequence may not expose enough parallelism. Segmented scan extends scan to operate independently within each subsequence, represented compactly as a flat array plus a bit-flag array marking subsequence starts. The lecture applies this to sparse matrix-vector multiplication stored in compressed sparse row format: a gather pulls the needed vector elements, a map multiplies them against the matrix's nonzero values, and a segmented scan with addition sums each row, giving parallelism proportional to the number of nonzeros rather than the number of rows.

Gather, scatter, and building a spatial grid in parallel (1:03:13)

Gather and scatter are introduced as the data-movement primitives, reading from or writing to arbitrary, data-dependent addresses, which can be costly since each parallel lane may touch a different cache line. The lecture then works through building a uniform grid data structure for a particle simulation (assigning particles to grid cells) as a case study. After discarding approaches that all bottleneck on a shared lock, per-cell locks, or duplicated cell lists per thread, the lecture arrives at a fully data-parallel solution: map each particle to its cell, sort particles by cell, then map again to detect where each cell's range starts and ends in the sorted array, giving parallelism proportional to the number of particles rather than the number of cells.

Before you watch

• Be familiar with the CUDA and GPU architecture material from the previous lecture, especially thread blocks, warps, and why the class benchmarks parallelism against a specific execution-context count.
• Recall how to reason about dependencies in a program, since this lecture reframes that skill as choosing operations known to be dependency-free.
• Basic familiarity with functional-style operations (map, reduce/fold) from any language is helpful, though the lecture defines each from scratch.

Check your understanding

1. Why can fold not be safely parallelized for an arbitrary binary function, and what property must the function have?
2. Walk through the two phases of the work-efficient scan algorithm and explain why it achieves O(n) total work instead of O(n log n).
3. Why does a simple O(n log n) scan implementation outperform the work-efficient O(n) algorithm when running on a single 32-wide SIMD warp?
4. How does compressed sparse row format combined with gather, map, and segmented scan implement sparse matrix-vector multiplication?
5. Why do lock-based and per-thread-duplication approaches to building the particle grid fail to scale to hundreds of thousands of threads, and how does the sort-based data-parallel approach avoid that problem?