Lecture 20: GPU Architecture II and Memory Overview

What this lecture covers

This lecture closes out the GPU architecture material by asking how a GPU can recover performance lost to branch divergence, then pivots to a new unit: memory systems. It sits between the SIMD/GPU block and roughly six lectures on memory organization, caches, and DRAM that follow.

The first half answers: when threads in a warp take different branches, how can hardware avoid wasting SIMD lanes? The second half motivates the entire memory unit with data on instruction throughput, energy costs, and reliability, then starts building physical memory from the bit cell up. After watching, you should be able to reason about SIMD utilization under branch divergence and explain why processors spend most of their time waiting on memory.

Key ideas

• Branch divergence: threads in a warp that take different paths after a branch force the GPU to execute each path separately, using an active mask to ignore inactive threads and lowering SIMD utilization.
• Dynamic warp formation: hardware can merge threads from different warps that reach the same program counter into a new warp, as long as they don't collide on the same SIMD lane, recovering utilization lost to divergence.
• Lane conflicts: two threads mapped to the same lane (and register file partition) can never be merged into one warp, which limits how much dynamic warp formation can help.
• Tensor cores: specialized SIMD units for matrix multiply-accumulate, later extended with sparsity support (skipping zero-valued entries) and lower-precision data types (down to FP8) to save energy.
• The memory wall: data from real processors (Alpha-based "it's the memory, stupid," and a Google 2015 study) shows flagship CPUs retire instructions only 10-20% of the time; the rest is spent waiting on memory.
• Energy asymmetry: a DRAM access can cost roughly 100 to 800 times the energy of a basic arithmetic operation, and going all the way to SSD storage multiplies that further, motivating processing-in-memory research.
• Memory technologies: flip-flops (fast, expensive), SRAM (crossed-coupled inverters, used for caches and register files), DRAM (one transistor plus one capacitor per bit, used for main memory), and flash/magnetic storage (cheapest, nonvolatile, slowest).
• Hierarchical arrays: every memory, from a tiny register file to a DRAM chip, is built as a two-dimensional array of rows and columns, and large memories are built by banking smaller arrays together.

Walkthrough

Dynamic warp formation and SIMD utilization (3:42)

The lecture reviews how a GPU executes SPMD programs on SIMD hardware: when a warp reaches a branch and threads diverge, the GPU creates two warps with complementary active masks, one per path, and merges them back once the paths reconverge. This wastes SIMD lanes proportional to how unevenly threads split across the branch.

Merging warps across the register file (16:52)

The lecture then shows dynamic warp formation: if enough warps are waiting and some of their active threads land on different SIMD lanes, hardware can combine them into a fuller warp, reducing wasted cycles. A worked example walks a control-flow graph with two warps (X and Y) through blocks A through G, comparing baseline execution against merged execution, and shows the technique's limit: threads that map to the same lane in two different warps can never be merged.

GPU latency tolerance and real GPU examples (30:00)

The lecture covers a scheduling technique (from the lecturer's own research) that staggers warp groups so their memory requests don't all arrive at once, cutting stall time by 20-30% in some cases. It then walks through the evolution of real Nvidia GPUs from a 2009 GTX 285 (240 stream processors) to newer generations with tensor cores, showing how core count, memory bandwidth, sparsity support, and lower-precision data types have grown.

Why memory matters: performance, energy, and reliability (1:04:05)

After a break, the lecture opens the memory unit with motivating data: the Alpha 21264 team's "it's the memory, stupid" finding that their processor ran at about 5% of peak throughput on database workloads, Google's 2015 data center analysis showing processors retiring instructions only 10-20% of the time, and energy figures showing a DRAM access costs roughly 800 times a 64-bit arithmetic operation. It also notes memory's role in system reliability, citing RowHammer as an example of memory-density-driven errors.

Building memory from bit cells to banks (1:27:17)

The lecture introduces virtual memory as an abstraction layer, then sets it aside to focus on physical memory. It compares flip-flops, SRAM, and DRAM by cost and speed, explains how an SRAM cross-coupled inverter cell and a DRAM capacitor cell are read and written, and introduces banking as the general technique for building large, fast memories out of many smaller arrays.

The DRAM hierarchy from channel to bank (1:44:28)

The lecture closes with a top-down tour of a real DRAM system: memory controller, channel, DIMM (rank), chip, and bank, explaining how a 64-bit access is assembled from 8-bit contributions of multiple chips and how a bank's row buffer works. This sets up the following lecture's deeper dive into memory organization.

Before you watch

• Review SIMD and vector processing, and the SPMD programming model covered in the previous lecture on GPU architecture I.
• Be comfortable with warps, active masks, and fine-grained multithreading, which this lecture builds on directly.
• Familiarity with out-of-order execution and pipelining helps when the lecture connects memory latency to earlier pipeline stalls.

Check your understanding

1. Why can't hardware always merge two divergent warps into one, even when both still have inactive lanes?
2. How does staggering warp groups reduce the total time spent waiting on long-latency memory operations?
3. What did the Alpha 21264 team's "it's the memory, stupid" finding say about peak versus actual instruction throughput, and why?
4. Why is a DRAM access so much more energy-costly than a basic arithmetic operation, according to the lecture's numbers?
5. Why are large memories built as hierarchies of banks and arrays rather than one large monolithic array?