Lecture 21: Memory Organization, Technology, and Caches

What this lecture covers

This lecture goes deeper into memory than the previous one, answering a concrete question: how does a DRAM chip physically store and retrieve a bit, and why does that make some accesses fast and others slow? It sits early in the memory unit, right after the overview lecture that motivated why memory dominates performance and energy, and right before the course moves on to building actual cache designs.

The lecture walks from the DRAM hierarchy (channel, rank, chip, bank) down to the row buffer, sense amplifiers, and the capacitor cell itself, then compares DRAM against SRAM and emerging technologies like phase-change memory. It closes by introducing the memory hierarchy and the locality principle that justifies it. After watching, you should be able to explain why a row-buffer hit is fast and a row-buffer conflict is slow, describe how a DRAM cell is sensed and restored, and explain why caches exploit temporal and spatial locality.

Key ideas

• Row buffer as a cache: accessing a DRAM row brings its entire contents into a row of sense amplifiers (the row buffer); a later access to the same row (a row-buffer hit) is fast, while accessing a different row (a row-buffer conflict) requires writing back the open row, activating the new one, then reading a column, which is much slower.
• Sense amplifier operation: a DRAM cell's tiny capacitor shares charge with the bit line; a differential sense amplifier (built from a modified cross-coupled inverter) detects whether the resulting voltage is above or below a reference and amplifies it to a clean logical one or zero, then restores that charge back into the capacitor.
• DRAM refresh: because a DRAM capacitor leaks charge over time, every row must be read and rewritten periodically (on the order of tens of milliseconds) or the stored data corrupts.
• SRAM vs. DRAM trade-off: SRAM (a cross-coupled inverter, six-plus transistors per bit) is faster and needs no refresh but is far less dense and more expensive per bit; DRAM (one transistor plus one capacitor) is denser and cheaper but slower and requires refresh and a different, incompatible fabrication process from logic.
• Accessing a cache block: a processor's 64-byte cache-line request is served by activating a row across all chips in a rank simultaneously, then reading 8 bytes at a time (one byte per chip) across eight consecutive column accesses.
• Emerging memory technologies: phase-change memory and resistive memories (like MRAM and memristors) store data via resistance rather than charge, trading access speed for potential nonvolatility, density, or lower cost; Intel's discontinued 3D XPoint was a real product built on this idea.
• Locality principle: temporal locality (recently accessed data is likely accessed again soon) and spatial locality (nearby data is likely accessed soon) are the empirical justification for caching, going back to Maurice Wilkes's 1965 slave-memory proposal.
• Hierarchical latency equation: a cache level's perceived access time depends on its own intrinsic latency plus its miss rate times the perceived access time of the next level down, which is why even a 90% hit rate can leave a surprisingly high average latency, and why multiple cache levels are used to approximate a fast, large ideal memory.

Walkthrough

DRAM hierarchy recap and row-buffer hits vs. conflicts (7:21)

The lecture reviews the DRAM hierarchy from channel to bank, then works through a worked example of accessing the same open row repeatedly (row-buffer hits, fast) versus switching to a different row (a row-buffer conflict, requiring precharge, activate, and column read). It gives approximate DRAM numbers: about 25 nanoseconds for a hit versus about 75 nanoseconds for a conflict, translated into hundreds of processor cycles at modern clock speeds.

Inside the bank: subarrays, sense amplifiers, and the DRAM cell (21:29)

The lecture explains why a bank is itself partitioned into smaller subarrays to keep interconnects short, then builds up the sense-amplifier circuit from the cross-coupled inverter seen earlier in the course. It walks through charge sharing, amplification, and charge restoration for a DRAM capacitor cell step by step, and shows how these cells and sense amplifiers are wired into a two-dimensional subarray with word lines and bit lines.

Accessing a 64-byte cache block across chips (35:38)

The lecture ties the physical structure back to a concrete access: a 64-byte cache block maps to one rank, and each of the rank's eight chips supplies 8 bits per cycle, so eight sequential column accesses (all row-buffer hits after the first) assemble the full 64 bytes.

DRAM refresh and SRAM comparison (39:41)

The lecture explains why DRAM needs periodic refresh due to capacitor leakage, then contrasts DRAM with SRAM: SRAM's feedback loop holds its state without refresh but costs more transistors and area per bit, and only SRAM is compatible with standard logic fabrication, which is why on-chip caches are built from SRAM while main memory is DRAM.

Emerging memory technologies (46:44)

The lecture surveys resistance-based memories such as phase-change memory, MRAM, and memristors, comparing them to DRAM on speed, density, endurance, volatility, and manufacturing maturity, and mentions Intel's 3D XPoint as a real (now discontinued) product built on phase-change memory.

Why memory hierarchies exist: locality (1:11:12)

The lecture states the fundamental trade-off (bigger memory is slower and cheaper per bit; smaller memory is faster and costlier per bit) and introduces the memory hierarchy as the way to approximate both speed and capacity. It grounds this in temporal and spatial locality, using everyday analogies (attention in class, sitting with friends, a bookshelf) and program examples like loop variables and array traversal.

Hierarchical latency analysis with a real processor (1:41:31)

The lecture derives the recursive formula for a cache level's perceived access time and applies it to Pentium 4's L1 and L2 cache parameters under different hit-rate scenarios, showing that different combinations of hit rates at each level can produce similar overall perceived latency, which explains why cache design involves several interacting choices rather than one dominant parameter.

Before you watch

• Review the previous lecture's DRAM hierarchy overview (channel, rank, chip, bank) and its energy and performance motivation for studying memory.
• Recall the cross-coupled inverter and CMOS transistor basics covered in the earlier digital logic lectures, since the sense-amplifier circuit builds directly on them.
• Be comfortable with the concept of banking from the earlier SIMD and vector processing lectures, since DRAM banks reuse the same idea.

Check your understanding

1. Why is a row-buffer hit much faster than a row-buffer conflict, and what three steps does a conflict require?
2. How does a DRAM sense amplifier turn a tiny, ambiguous charge difference into a clean logical one or zero, and why must the charge be restored afterward?
3. Why can SRAM be built directly alongside logic on the same chip while DRAM generally cannot?
4. Using the hierarchical latency equation, why can two very different sets of per-level hit rates produce a similar perceived access latency at the processor?
5. What is the difference between temporal and spatial locality, and how does the 64-byte cache block exploit spatial locality specifically?