Lecture 5: GPUs

What this lecture covers

The lecture demystifies GPU hardware: why matrix multiplies get mysteriously slow at certain sizes, and what tricks (precision, fusion, recomputation, memory coalescing, tiling) make GPU code fast. It builds up from the GPU's physical layout and execution model to a full explanation of why Flash Attention works.

This is the systems-foundations lecture that assignment two builds on, since students implement parts of Flash Attention 2 in Triton. By the end you should be able to read a GPU throughput-vs-matrix-size plot and explain its wavy pattern, and describe how tiling and online softmax let Flash Attention avoid materializing the full attention matrix.

Key ideas

• Compute has scaled far faster than memory: GPU matrix-multiply throughput has grown up to 100,000x while memory bandwidth has grown only about 100x, making memory movement, not flops, the usual bottleneck.
• GPUs optimize for throughput, CPUs for latency: a GPU has many small compute units (SMs, each with streaming processors) and little control logic, running the same instruction across many threads (SIMT) rather than finishing any one task fast.
• Memory hierarchy: registers and shared memory inside an SM are fastest, L2 cache is slower, and global memory (DRAM, off-chip) is slowest, roughly 10x slower per hop.
• Lower precision doubles effective memory bandwidth: moving from FP32 to FP16 halves the bytes moved per flop, though careful mixed-precision design (e.g. FP32 accumulation) is needed for stability.
• Kernel fusion: chaining operations within the compute unit instead of writing intermediate results back to global memory avoids redundant slow memory round-trips.
• Recomputation: recomputing intermediate activations during the backward pass, instead of storing and reloading them, trades cheap compute for scarce memory bandwidth.
• Memory coalescing and burst mode: DRAM returns a whole aligned chunk ("burst section") per access, so threads accessing contiguous, aligned memory get much higher effective throughput than threads accessing scattered addresses.
• Tiling: splitting matrices into blocks small enough to fit in fast shared memory, then doing many computations per load, cuts the number of slow global memory reads by roughly the tile size.

Walkthrough

Why hardware matters for scaling (0:05)

The lecture opens with the goal of making CUDA and GPUs less mysterious, previewing a matrix-multiply throughput chart with unpredictable wavy patterns that will be explained by the end. It frames hardware understanding as essential because compute scaling, not just better deep learning ideas, has driven most of the field's progress, referencing the end of single-thread (Dennard) scaling and the shift to parallel scaling.

GPU anatomy and execution model (6:08)

The lecture contrasts CPU design (large control units, few threads, latency-optimized) with GPU design (many streaming multiprocessors, or SMs, each containing streaming processors that execute threads in parallel, throughput-optimized). It introduces the three granularities of GPU execution: blocks (assigned to an SM), warps (groups of 32 threads executing together), and threads, plus the memory hierarchy from registers up through shared memory, L2 cache, and global memory. TPUs are briefly compared, noting their similar structure (tensor cores, scalar/vector units, a dedicated matrix-multiply unit) but simpler, matmul-only design.

Why matrix multiplies dominate, and the memory-compute gap (20:18)

The lecture shows that specialized tensor cores have made matrix multiplies orders of magnitude faster than other GPU operations, meaning any efficient neural architecture must be dominated by matmuls. It then establishes the core theme: GPU compute has scaled far faster than memory bandwidth, so memory movement is increasingly the bottleneck as hardware gets newer.

The roofline model and non-memory bottlenecks (25:24)

Using the earlier throughput-vs-matrix-size chart as a running puzzle, the lecture introduces the roofline model: performance is memory-bound at small problem sizes and compute-bound at large ones. It first covers a non-memory pitfall, warp divergence from conditional branches, where threads in a warp taking different code paths must execute serially instead of in parallel.

Core memory optimization tricks (29:24)

The lecture works through four techniques in sequence: lower precision (fewer bytes moved per flop), operator fusion (keeping a chain of operations in the compute unit instead of round-tripping to global memory), recomputation (recomputing activations in the backward pass instead of storing and reloading them), and memory coalescing (exploiting DRAM's burst-mode reads by having threads access contiguous, aligned memory).

Tiling and why matrix-multiply performance looks so erratic (46:32)

Tiling is introduced as the way to minimize global memory reads in matrix multiplication: load a submatrix ("tile") into shared memory once, then reuse it for many computations. The lecture then explains the earlier mystery chart: tile-alignment issues (matrix sizes not divisible by clean powers of two) hurt coalescing, and "wave quantization" (tile counts that slightly exceed a multiple of the GPU's SM count) leave some SMs idle on a final, mostly-wasted pass, producing the sharp performance cliffs observed.

Putting it together: Flash Attention (1:04:42)

The lecture closes by showing how tiling and recomputation combine to produce Flash Attention. The QKV matrix multiplies are handled with standard tiling, but softmax is a global, row-wise operation that would normally require materializing the full attention matrix. The online softmax algorithm solves this by incrementally tracking a running maximum and normalizer as tiles are processed, so the exact softmax can be computed tile-by-tile without ever storing the full quadratic-sized matrix. The backward pass similarly avoids storing activations by recomputing them tile-by-tile.

Before you watch

• Review basic attention mechanics (Q, K, V matrix multiplies and softmax), since the lecture assumes you already know how to implement standard attention.
• Recall backpropagation and the idea of storing activations during the forward pass for use in the backward pass.
• Some familiarity with mixed-precision training will help with the precision-tradeoff discussion.

Check your understanding

1. Why is memory bandwidth, rather than compute, typically the bottleneck on modern GPUs?
2. What is memory coalescing, and why does traversal order (row-major vs. column-major) affect it?
3. How does tiling reduce the number of global memory reads needed for a matrix multiply?
4. Why can't a naive softmax be computed tile-by-tile without an online algorithm?
5. What causes the sharp performance drops ("wave quantization") seen when matrix dimensions slightly exceed a multiple of the tile size times the SM count?