Lecture 10: Efficiently Evaluating DNNs on GPUs

What this lecture covers

This lecture asks how modern GPUs and CPUs run deep neural networks efficiently. It starts from a single neuron - a dot product followed by a nonlinearity - and builds up to full convolutional layers and Transformer attention blocks, showing at each step how the computation maps onto hardware that is good at dense matrix multiplication.

The lecture sits after the course's core parallel programming material and feeds directly into assignment four. After watching, you should be able to explain why convolution is implemented as matrix multiplication, why blocking a matrix multiply for cache reuse matters, why fusing operations like bias-add and max-pool saves memory traffic, and why GPUs (with features like tensor cores) are well suited to this workload.

Key ideas

• Neuron as dot product: a neuron computes a dot product between weights and inputs, then applies a nonlinear function such as max(x, 0).
• Convolution as matrix multiply (im2col): copying overlapping input patches into a matrix turns a convolution into a matrix-matrix product, so any fast GEMM library can execute it.
• Arithmetic intensity: naive matrix multiply reads far more data than the math it does per byte; blocking submatrices into cache raises the ratio of computation to memory traffic.
• Cache/shared-memory blocking: choosing a block size that fits the cache (or, on a GPU, shared memory as an explicit scratchpad) is the key trick for turning a bandwidth-bound loop into a compute-bound one.
• Implicit GEMM: rather than materializing the huge duplicated im2col matrix, compute the matrix indices on the fly and fetch values from the original tensor, avoiding the memory blow-up.
• Layer fusion: chaining a convolution's output straight into bias-add and max-pool while it is still on-chip avoids repeated round-trips to memory, which otherwise dominate runtime.
• FlashAttention: computing attention's softmax in chunks (using a running max and sum) lets an entire attention block be fused, cutting memory footprint from quadratic to block-sized and enabling much longer sequences.
• Tensor cores: a GPU instruction that performs a small fixed-size matrix multiply amortizes instruction overhead across many math operations, delivering an order of magnitude more throughput than scalar multiply-add.

Walkthrough

The deep-network workload (6:09)

The lecture opens by explaining why it appears here: it is useful preparation for assignment four. It frames a neuron as a dot product between a weight vector and an input vector, passed through a nonlinearity (here, a max with zero). Stacking neurons into layers produces either fully connected layers, where every output of one layer feeds every neuron of the next, or convolutional layers, where each output depends on a small sliding window of inputs.

Convolution as image processing and as matrix multiplication (12:16)

A concrete convolution example shows how different fixed weights average pixels (blurring) or compute finite differences (edge detection). The lecture then generalizes to filters with many input and output channels, arranged as tensors, and shows how a naive implementation needs seven nested loops. It walks through the classic trick of copying overlapping input patches into a wide matrix (im2col) so the convolution becomes a plain matrix-matrix product against a matrix of filter weights - letting any fast matrix multiply library do the work.

Why algorithm design matters first (18:23)

Before optimizing execution, the lecture lists three ways to make deep networks faster: better network architectures (fewer FLOPs for the same accuracy, as seen across years of ImageNet models), mapping a fixed architecture well onto hardware, and specialized accelerator hardware. It shows how architectural changes alone produced roughly a 25x reduction in weights and computation over a few years - much faster than hardware improved in the same period.

Blocking matrix multiply for cache reuse (34:39)

Using plain three-loop matrix multiplication as an example, the lecture shows that naive execution is bandwidth-bound: for n-cubed work you only touch n-squared data, but a straightforward implementation still reads two values per multiply-add. The fix is to block the computation into submatrices sized to fit in cache (or, on a GPU, explicitly copied into shared memory as a scratchpad), raising arithmetic intensity from roughly constant to proportional to the block size. Multiple levels of blocking (L1, L2, L3, even registers) compound this benefit.

Implicit GEMM and layer fusion (48:59)

Turning a convolution into an explicit im2col matrix wastes enormous memory, since each input pixel gets duplicated many times. The lecture introduces implicit GEMM, where the same loop structure is used but the matrix indices are computed on the fly and values are fetched directly from the original tensor, avoiding that duplication. It also covers operator fusion: rather than writing a convolution's output to memory and reading it back for bias-add and max-pool, doing all three while the data is still in cache or shared memory removes several bandwidth-bound round trips - explaining why libraries like cuDNN expose fused kernels and why compilers such as Jax and Triton try to automate this fusion.

FlashAttention: fusing through softmax (1:04:13)

Attention layers compute an outer product of query and key vectors into an n-by-n matrix, apply softmax row by row, then multiply by a value matrix. Softmax appears to require the whole row before it can scale anything, which normally forces the huge intermediate matrix through memory multiple times. The lecture shows how softmax can be computed incrementally in blocks using a running maximum and sum, enabling the whole attention computation to be fused block by block. This shrinks memory use from quadratic in sequence length to block-sized, which is credited with enabling much longer context windows in large language models.

Why GPUs suit this workload (1:14:18)

The lecture closes by connecting the workload's properties - abundant parallelism, high potential arithmetic intensity, and heavy use of SIMD-style operations - to why GPUs are a good fit. It also explains tensor cores as instructions that perform small fixed-size matrix multiplies, amortizing per-instruction overhead across many math operations and delivering roughly an order of magnitude more throughput than doing the same work with scalar multiply-adds.

Before you watch

• Be comfortable with dependency graphs, SIMD, and arithmetic intensity/bandwidth-bound reasoning from earlier lectures in this course.
• Basic familiarity with what a convolution and a matrix multiplication are will help; the lecture defines both from scratch but moves quickly.
• Recall the blocked, bandwidth-bound examples from earlier assignments, since this lecture reuses that reasoning for matrix multiply.

Check your understanding

1. Why does expressing a convolution as an im2col matrix multiplication create a much larger memory footprint than the original input, and how does implicit GEMM avoid that?
2. Explain why the naive three-loop matrix multiplication is bandwidth-bound even though it does n-cubed work on n-squared data.
3. How does blocking the matrix multiplication change its arithmetic intensity, and what determines the largest block size you can choose?
4. Why does fusing bias-add and max-pool into the same pass as a convolution improve performance?
5. What property of softmax made it seem impossible to fuse through an attention block, and how does the chunked computation get around it?