Seyed Masoud Hosseini · Overview · Study log · Weekly summaries · Ideas · Search · Transcript · RSS feed
Digital Design & Computer Architecture · Lecture 25 of 37 · 1:51:12
Lecture 20: GPU Architecture II and Memory Overview
Study guide
What this lecture covers
This lecture closes out the GPU architecture material by asking how a GPU can recover performance lost to branch divergence, then pivots to a new unit: memory systems. It sits between the SIMD/GPU block and roughly six lectures on memory organization, caches, and DRAM that follow.
The first half answers: when threads in a warp take different branches, how can hardware avoid wasting SIMD lanes? The second half motivates the entire memory unit with data on instruction throughput, energy costs, and reliability, then starts building physical memory from the bit cell up. After watching, you should be able to reason about SIMD utilization under branch divergence and explain why processors spend most of their time waiting on memory.
Key ideas
- Branch divergence: threads in a warp that take different paths after a branch force the GPU to execute each path separately, using an active mask to ignore inactive threads and lowering SIMD utilization.
- Dynamic warp formation: hardware can merge threads from different warps that reach the same program counter into a new warp, as long as they don't collide on the same SIMD lane, recovering utilization lost to divergence.
- Lane conflicts: two threads mapped to the same lane (and register file partition) can never be merged into one warp, which limits how much dynamic warp formation can help.
- Tensor cores: specialized SIMD units for matrix multiply-accumulate, later extended with sparsity support (skipping zero-valued entries) and lower-precision data types (down to FP8) to save energy.
- The memory wall: data from real processors (Alpha-based "it's the memory, stupid," and a Google 2015 study) shows flagship CPUs retire instructions only 10-20% of the time; the rest is spent waiting on memory.
- Energy asymmetry: a DRAM access can cost roughly 100 to 800 times the energy of a basic arithmetic operation, and going all the way to SSD storage multiplies that further, motivating processing-in-memory research.
- Memory technologies: flip-flops (fast, expensive), SRAM (crossed-coupled inverters, used for caches and register files), DRAM (one transistor plus one capacitor per bit, used for main memory), and flash/magnetic storage (cheapest, nonvolatile, slowest).
- Hierarchical arrays: every memory, from a tiny register file to a DRAM chip, is built as a two-dimensional array of rows and columns, and large memories are built by banking smaller arrays together.
Walkthrough
Dynamic warp formation and SIMD utilization (3:42)
The lecture reviews how a GPU executes SPMD programs on SIMD hardware: when a warp reaches a branch and threads diverge, the GPU creates two warps with complementary active masks, one per path, and merges them back once the paths reconverge. This wastes SIMD lanes proportional to how unevenly threads split across the branch.
Merging warps across the register file (16:52)
The lecture then shows dynamic warp formation: if enough warps are waiting and some of their active threads land on different SIMD lanes, hardware can combine them into a fuller warp, reducing wasted cycles. A worked example walks a control-flow graph with two warps (X and Y) through blocks A through G, comparing baseline execution against merged execution, and shows the technique's limit: threads that map to the same lane in two different warps can never be merged.
GPU latency tolerance and real GPU examples (30:00)
The lecture covers a scheduling technique (from the lecturer's own research) that staggers warp groups so their memory requests don't all arrive at once, cutting stall time by 20-30% in some cases. It then walks through the evolution of real Nvidia GPUs from a 2009 GTX 285 (240 stream processors) to newer generations with tensor cores, showing how core count, memory bandwidth, sparsity support, and lower-precision data types have grown.
Why memory matters: performance, energy, and reliability (1:04:05)
After a break, the lecture opens the memory unit with motivating data: the Alpha 21264 team's "it's the memory, stupid" finding that their processor ran at about 5% of peak throughput on database workloads, Google's 2015 data center analysis showing processors retiring instructions only 10-20% of the time, and energy figures showing a DRAM access costs roughly 800 times a 64-bit arithmetic operation. It also notes memory's role in system reliability, citing RowHammer as an example of memory-density-driven errors.
Building memory from bit cells to banks (1:27:17)
The lecture introduces virtual memory as an abstraction layer, then sets it aside to focus on physical memory. It compares flip-flops, SRAM, and DRAM by cost and speed, explains how an SRAM cross-coupled inverter cell and a DRAM capacitor cell are read and written, and introduces banking as the general technique for building large, fast memories out of many smaller arrays.
The DRAM hierarchy from channel to bank (1:44:28)
The lecture closes with a top-down tour of a real DRAM system: memory controller, channel, DIMM (rank), chip, and bank, explaining how a 64-bit access is assembled from 8-bit contributions of multiple chips and how a bank's row buffer works. This sets up the following lecture's deeper dive into memory organization.
Before you watch
- Review SIMD and vector processing, and the SPMD programming model covered in the previous lecture on GPU architecture I.
- Be comfortable with warps, active masks, and fine-grained multithreading, which this lecture builds on directly.
- Familiarity with out-of-order execution and pipelining helps when the lecture connects memory latency to earlier pipeline stalls.
Check your understanding
- Why can't hardware always merge two divergent warps into one, even when both still have inactive lanes?
- How does staggering warp groups reduce the total time spent waiting on long-latency memory operations?
- What did the Alpha 21264 team's "it's the memory, stupid" finding say about peak versus actual instruction throughput, and why?
- Why is a DRAM access so much more energy-costly than a basic arithmetic operation, according to the lecture's numbers?
- Why are large memories built as hierarchies of banks and arrays rather than one large monolithic array?
Vocabulary
- branch divergence (noun)
- A situation where threads in the same group split into different paths after a branch.
Branch divergence forces the GPU to run each path separately. - active mask (noun)
- A record showing which threads in a warp are currently doing real work.
The active mask marks which threads are participating in this path. - dynamic warp formation (noun)
- Merging threads from different warps that reach the same instruction into a new, fuller warp.
Dynamic warp formation recovers utilization lost to divergence. - SIMD utilization (noun)
- How much of a SIMD unit's capacity is actually doing useful work.
Branch divergence lowers SIMD utilization. - lane conflict (noun)
- A case where two threads are mapped to the same hardware lane and cannot be combined.
A lane conflict prevents two warps from merging. - tensor core (noun)
- A specialized hardware unit built for fast matrix multiply-accumulate operations.
Tensor cores speed up the matrix math used in neural networks. - sparsity (noun)
- The presence of many zero values in a dataset that can be skipped to save work.
Sparsity support lets the hardware skip zero-valued entries. - memory wall (noun)
- The performance limit caused by memory being far slower than the processor.
The memory wall explains why processors spend so much time waiting. - retire (verb)
- To finish an instruction by making its result final and visible.
Processors only retire instructions a small fraction of the time due to memory delays. - flip-flop (noun)
- A small, fast memory element that stores one bit of data.
Flip-flops are the fastest but most expensive form of memory. - SRAM (noun)
- A fast type of memory built from cross-coupled inverters, commonly used for caches.
SRAM is faster than DRAM but takes up more space per bit. - DRAM (noun)
- A memory type storing each bit with a single transistor and capacitor, used for main memory.
DRAM offers much higher density than SRAM but is slower. - capacitor (noun)
- A small component that stores an electric charge, used to represent a bit in DRAM.
A DRAM cell stores its bit as charge in a capacitor. - nonvolatile (adjective)
- Describes storage that keeps its data even when power is turned off.
Flash storage is nonvolatile, unlike DRAM. - memory controller (noun)
- The hardware that manages requests to and from main memory.
The memory controller coordinates access across multiple channels. - row buffer (noun)
- A temporary storage area in a DRAM bank that holds the most recently accessed row.
Reading from the row buffer is much faster than a fresh row access. - RowHammer (noun)
- A hardware vulnerability where repeatedly accessing one DRAM row can flip bits in nearby rows.
RowHammer shows how memory density can cause reliability problems. - processing-in-memory (noun)
- An approach that performs computation directly inside or near memory to reduce data movement.
Processing-in-memory research aims to cut the energy cost of moving data. - reconverge (verb)
- To come back together after having split apart.
The two divergent warps reconverge once both paths finish. - complementary (adjective)
- Combining together to make a complete whole.
The two warps use complementary active masks covering all threads. - stagger (verb)
- To arrange events to start at slightly different times instead of all at once.
Staggering warp groups spreads out memory requests over time. - peak throughput (phrase)
- The maximum possible rate of work a system could theoretically achieve.
The processor ran at only about 5% of peak throughput. - reliability (noun)
- How consistently a system works correctly without failures.
Memory reliability is a growing concern as density increases. - hierarchy (noun)
- A system organized into ranked levels, from broad to specific.
DRAM is organized as a hierarchy from channel down to bank. - cross-coupled inverter (noun)
- A pair of inverters wired to feed each other, forming a stable memory cell.
An SRAM cell is built from a cross-coupled inverter pair. - assemble (verb)
- To put separate pieces together to form a whole.
A 64-bit access is assembled from smaller contributions of multiple chips. - monolithic (adjective)
- Built as one single large unit rather than several smaller connected parts.
Large memories avoid being built as one monolithic array.
From the YouTube description
Digital Design and Computer Architecture, ETH Zürich, Spring 2025 (https://safari.ethz.ch/ddca/spring2025/)
Lecture 20: GPU Architectures II & Memory Overview, Organization and Technology
Lecturer: Prof. Onur Mutlu
Date: 9 May 2025
Lecture 20a: GPU Architectures II
Lecture 20a Slides (pptx): https://safari.ethz.ch/ddca/spring2025/lib/exe/fetch.php?media=onur-ddca-2025-lecture20a-gpu-ii-beforelecture.pptx
Lecture 20a Slides (pdf): https://safari.ethz.ch/ddca/spring2025/lib/exe/fetch.php?media=onur-ddca-2025-lecture20a-gpu-ii-beforelecture.pdf
Lecture 20b: Memory Overview, Organization and Technology
Lecture 20b Slides (pptx): https://safari.ethz.ch/ddca/spring2025/lib/exe/fetch.php?media=onur-ddca-2025-lecture20b-memory-organization-beforelecture.pptx
Lecture 20b Slides (pdf): https://safari.ethz.ch/ddca/spring2025/lib/exe/fetch.php?media=onur-ddca-2025-lecture20b-memory-organization-beforelecture.pdf
Recommended Reading:
====================
Intelligent Architectures for Intelligent Computing Systems
https://people.inf.ethz.ch/omutlu/pub/intelligent-architectures-for-intelligent-computingsystems-invited_paper_DATE21.pdf
A Modern Primer on Processing in Memory
https://people.inf.ethz.ch/omutlu/pub/ModernPrimerOnPIM_springer-emerging-computing-bookchapter21.pdf
RowHammer: A Retrospective
https://people.inf.ethz.ch/omutlu/pub/RowHammer-Retrospective_ieee_tcad19.pdf
RECOMMENDED LECTURE VIDEOS & PLAYLISTS:
========================================
Computer Architecture Fall 2021 Lectures Playlist:
https://www.youtube.com/watch?v=4yfkM_5EFgo&list=PL5Q2soXY2Zi-Mnk1PxjEIG32HAGILkTOF
Computer Architecture Fall 2022 Lectures Playlist:
https://www.youtube.com/watch?v=BIpPTqHK-Lc&list=PL5Q2soXY2Zi-cAls3cyauNzM7-74Eq31O
Digital Design and Computer Architecture Spring 2022 Livestream Lectures Playlist:
https://www.youtube.com/watch?v=cpXdE3HwvK0&list=PL5Q2soXY2Zi97Ya5DEUpMpO2bbAoaG7c6
Digital Design and Computer Architecture Spring 2021 Livestream Lectures Playlist:
https://www.youtube.com/watch?v=LbC0EZY8yw4&list=PL5Q2soXY2Zi_uej3aY39YB5pfW4SJ7LlN
Featured Lectures:
https://www.youtube.com/watch?v=jVYCchBGNVc&list=PL5Q2soXY2Zi8VrmOTz44l2WupethSdh-M&index=1
Interview with Professor Onur Mutlu:
https://www.youtube.com/watch?v=8ffSEKZhmvo&list=PL5Q2soXY2Zi8VrmOTz44l2WupethSdh-M&index=9
The Story of RowHammer Lecture:
https://www.youtube.com/watch?v=sgd7PHQQ1AI&list=PL5Q2soXY2Zi8D_5MGV6EnXEJHnV2YFBJl&index=39
Accelerating Genome Analysis Lecture:
https://www.youtube.com/watch?v=r7sn41lH-4A&list=PL5Q2soXY2Zi8D_5MGV6EnXEJHnV2YFBJl&index=41
Memory-Centric Computing Systems Tutorial at IEDM 2021:
https://www.youtube.com/watch?v=H3sEaINPBOE&list=PL5Q2soXY2Zi8D_5MGV6EnXEJHnV2YFBJl&index=35
Intelligent Architectures for Intelligent Machines Lecture:
https://www.youtube.com/watch?v=GTieZPY4Wmc&list=PL5Q2soXY2Zi8D_5MGV6EnXEJHnV2YFBJl&index=38
Computer Architecture Fall 2020 Lectures Playlist:
https://www.youtube.com/watch?v=c3mPdZA-Fmc&list=PL5Q2soXY2Zi9xidyIgBxUz7xRPS-wisBN
Digital Design and Computer Architecture Spring 2020 Lectures Playlist:
https://www.youtube.com/watch?v=AJBmIaUneB0&list=PL5Q2soXY2Zi_FRrloMa2fUYWPGiZUBQo2
Public Lectures by Onur Mutlu, Playlist:
https://www.youtube.com/watch?v=kgiZlSOcGFM&list=PL5Q2soXY2Zi8D_5MGV6EnXEJHnV2YFBJl
Computer Architecture at Carnegie Mellon Spring 2015 Lectures Playlist:
https://www.youtube.com/watch?v=zLP_X4wyHbY&list=PL5PHm2jkkXmi5CxxI7b3JCL1TWybTDtKq
Rethinking Memory System Design Lecture @stanfordonline :
https://www.youtube.com/watch?v=F7xZLNMIY1E&list=PL5Q2soXY2Zi8D_5MGV6EnXEJHnV2YFBJl&index=4
← Lecture 19: GPU Architectures · Lecture 21: Memory Organization, Technology, and Caches →