Lecture 12: GPU Acceleration

What this lecture covers

Following the previous lecture's CPU acceleration techniques, this lecture asks how to use a massively parallel accelerator like a GPU. It explains why GPUs are built differently from CPUs: instead of a few flexible cores with strong control logic, a GPU has a huge number of simple compute units driven by relatively few control units, suited to applying the same operation across many data elements at once (the single-instruction-multiple-thread model).

The lecture uses CUDA terminology throughout, since concepts map closely onto other GPU programming models such as OpenCL and Metal. It works through a basic vector-add kernel, the thread/block/grid hierarchy, the importance of keeping data resident on the GPU rather than copying it back and forth, and then a case study applying register tiling and shared-memory tiling (building on the previous lecture's CPU tiling ideas) to matrix multiplication. After watching, you should be able to write a simple CUDA kernel, explain why host-device memory copies are often the real bottleneck, and describe how shared memory reduces redundant loads across threads in a block.

Key ideas

• CPUs vs. GPUs: CPUs have few cores with strong control units suited to flexible, independent tasks; GPUs have many simple compute units driven by relatively few control units, suited to applying the same instruction across massive amounts of data.
• Single instruction, multiple threads (SIMT): every GPU thread runs the same code, but each thread has its own context (thread index, block index) that makes it operate on different data.
• Thread hierarchy: threads are grouped into thread blocks, and thread blocks are grouped into a launch grid; threads in the same block can share fast on-chip shared memory.
• Host and device memory are separate: GPU memory must be explicitly allocated and data explicitly copied between host (CPU) and device (GPU) memory over the PCIe bus; naive code that copies data back and forth for every operation can be bottlenecked by these transfers rather than compute.
• Keep data resident on the GPU: efficient GPU pipelines minimize host-device transfers by keeping intermediate results on the device across a sequence of operations, rather than copying back to the CPU after each step.
• Data-parallel programs are easiest to parallelize: when each output element's computation is independent of the others (as in vector addition), the work maps cleanly onto independent threads; computations with sequential dependencies (like a running maximum) are harder to parallelize directly.
• Shared memory reduces redundant loads: when neighboring threads need overlapping input data (such as a sliding-window sum), cooperatively loading that data into shared memory once and having threads reuse it cuts the total number of memory loads compared to each thread loading its own window independently.
• Register and shared-memory tiling for matrix multiplication: the same tiling ideas from CPU acceleration apply on the GPU — each thread computes a small tile using register reuse, and each thread block cooperatively loads a larger tile into shared memory to reuse data across threads in the block.
• Trade-offs in choosing tile sizes: larger register tiles increase reuse per thread but reduce how many threads can run concurrently (since total registers per streaming multiprocessor are fixed); larger shared-memory tiles reduce global memory traffic but limit how many thread blocks can be resident at once. These trade-offs are often explored empirically through auto-tuning.

Walkthrough

CPU vs. GPU architecture (2:04)

The lecture contrasts a small number of flexible, independently controlled CPU cores with a GPU's design: a large number of simple compute units ("soldiers") controlled by relatively few control units ("commanders"), well suited to tasks like adding the same brightness value to every pixel in an image. This architecture motivates GPUs' strength at data-parallel workloads and explains why GPU adoption produced large speedups (the lecture describes an early convolution workload going from about a week on CPU to a few hours on GPU).

The CUDA programming model: threads, blocks, and grids (7:08)

CUDA's single-instruction-multiple-thread model has every thread execute the same code, distinguished only by its thread index and block index. Threads are grouped into thread blocks (which can share fast shared memory), and thread blocks are grouped into a launch grid. A vector-add kernel is shown computing each output element's global thread index from its block index and thread index, with each thread performing one element's addition independently.

Host-side code and the cost of memory copies (15:13)

Running a CUDA kernel requires host-side code to allocate GPU memory, copy input data from host to device, launch the kernel with a chosen number of blocks and threads per block, then copy results back. The lecture recounts an early mistake: naively copying data to and from the GPU around every operation produced only a 1.3x speedup over an FFT-based CPU implementation of convolution, because the PCIe transfer became the bottleneck. The fix is to keep data resident on the GPU across a sequence of operations, copying back to the host only when necessary — the same principle used in libraries like PyTorch and needle when working with GPU arrays.

GPU memory hierarchy and shared memory (22:23)

Physically, GPUs map thread blocks onto streaming multiprocessors, each containing multiple compute cores and a shared memory region accessible to all threads in a resident block, in addition to each thread's private registers. A sliding-window-sum example (a simplified 1D convolution) shows that naively, each thread would reload its own window of input elements, but neighboring threads' windows overlap substantially. Cooperatively loading the needed region into shared memory once, synchronizing with a barrier, and then having each thread read from shared memory cuts the number of loads considerably.

Matrix multiplication case study: register and shared-memory tiling (29:30)

Applying the register tiling technique from the previous CPU lecture, each GPU thread computes a small v-by-v submatrix, loading strips of the input matrices into registers and reusing each loaded value v times, reducing per-thread memory loads to roughly n^3/v. A second level of tiling uses shared memory: each thread block cooperatively loads a larger L-by-L tile of the input matrices into shared memory once, so all threads in the block can reuse that data via register tiling internally, reducing global memory traffic further. Choosing the tile sizes L and v involves a trade-off between reuse and occupancy — using more registers or shared memory per thread limits how many threads or blocks can run concurrently on a streaming multiprocessor — and is often explored via auto-tuning rather than solved analytically.

Further GPU optimization techniques (41:44)

The lecture closes by naming additional techniques beyond the scope of the lecture itself: ensuring threads in a block access contiguous memory regions, avoiding shared-memory bank conflicts, software pipelining to overlap data loading with computation, warp-level optimizations, and specialized tensor cores for matrix-matrix operations on modern GPUs. These are pointed to as further reading via the CUDA programming guide rather than covered in depth here.

Before you watch

• Review the previous lecture's CPU acceleration case study on register tiling and cache-aware tiling for matrix multiplication, since this lecture builds directly on those ideas for the GPU.
• Basic familiarity with matrix multiplication loop structure (C[i,j] = sum_k A[i,k] * B[j,k]) is assumed.
• No prior CUDA experience is required; the lecture introduces the programming model from scratch.

Check your understanding

1. Why is a GPU architecture well suited to tasks like adding a constant brightness value to every pixel of an image, but less suited to running many different independent tasks simultaneously?
2. In the vector-add CUDA example, how is each thread's global index computed from its thread index and block index?
3. Why did an early naive GPU implementation of a convolution kernel only achieve a 1.3x speedup, and what change fixed the bottleneck?
4. How does loading data into shared memory reduce the total number of memory loads in the sliding-window-sum example, compared to each thread loading its own window independently?
5. What trade-off arises when choosing larger register tile sizes or larger shared-memory tile sizes for the GPU matrix multiplication case study?