Lecture 2: A Modern Multi-Core Processor

What this lecture covers

This lecture continues Stanford's CS149 by first deepening the previous session's discussion of caches, then introducing the three major ways modern chips add parallel execution capability: multicore, SIMD (vector) execution, and hardware multithreading. It sits early in the course, right after the introductory lecture on programs, instructions, and memory, and it builds the vocabulary the rest of the course will reuse when discussing CPUs and GPUs.

The lecture works through cache lines, hits, and misses using a worked example, explains why chip designers stopped trying to find parallelism automatically and instead pushed the responsibility onto the programmer, and shows how a straight-line C program computing an approximation of sine per array element can be transformed into a multicore, then vectorized, then multithreaded version. It ends by explaining why branches inside vectorized code can waste most of a chip's execution capacity. After watching, you should be able to trace how a cache line gets filled and evicted, and explain the difference between running instructions on more cores, wider vectors, or more threads per core.

Key ideas

• Cache line: the fixed-size chunk of bytes (four bytes in the lecture's simplified example) that memory always transfers to cache as one unit, even if a program asked for only one byte.
• Cold miss: a cache miss that happens because the data has never been accessed before, regardless of cache size.
• Capacity miss: a miss that happens only because the cache is too small to hold everything that has been recently touched.
• Temporal and spatial locality: caches help because programs tend to reaccess the same data soon (temporal) and access nearby addresses soon after (spatial).
• Multicore: instead of building one complex core that automatically finds parallelism, replicate several simpler cores, each running its own independent instruction stream.
• SIMD (vector) execution: one instruction operates on multiple data elements at once (eight in the lecture's examples), amortizing the cost of fetching and decoding instructions across many arithmetic units.
• Divergence: when different loop iterations inside a SIMD group need to take different branches of an if/else, the hardware must run both branches and mask off the inactive lanes, which can drop utilization as low as one over the vector width.
• Hardware multithreading: a core holds the register state for several threads and switches to a runnable one instantly whenever the current thread stalls on a long-latency operation like a memory access.

Walkthrough

Recap: instructions, memory, and caches (0:05)

The lecture opens by restating the prior session's ideas: a program is a list of instructions, executing an instruction changes state held in registers or memory, and a simplified processor diagram has orange control logic, a yellow execution unit, and blue register/memory state. It also recaps superscalar execution, where a processor automatically finds independent instructions in one stream and runs more than one per clock without the program changing.

How a cache actually works: lines, misses, and locality (7:12)

Using a small worked example with an eight-byte, two-line cache, the lecture walks through what happens as a program reads addresses in sequence: the first access to any line is a cold miss that pulls in the whole line, subsequent accesses to bytes already in that line are hits, and once the cache is full, bringing in a new line evicts the least recently used one. The lecture distinguishes cold misses (never touched before) from capacity misses (would have hit with a bigger cache) and briefly names conflict misses without going into their implementation. It stresses that removing the cache entirely would not change a program's result, only its speed, which is why a cache is purely an implementation detail sitting underneath the abstract meaning of memory.

Multicore: replicating simple cores instead of finding parallelism automatically (33:32)

Using a loop that computes an approximation of sine for every element of an array, the lecture shows that the inner instruction sequence has little instruction-level parallelism for a superscalar processor to find, even though the outer loop is completely independent across iterations. Since hardware cannot reliably prove that far-apart loop iterations are independent, architects instead strip a complex core down to something simpler and replicate it, creating multiple cores that each run their own instruction stream. Getting a speedup then requires the programmer to explicitly split the work, for example by spawning threads that each handle half the array.

SIMD: doing eight things per instruction (44:41)

Because many loop iterations execute the exact same instructions on different data, the lecture introduces vector instructions: a single instruction operates on a wide vector, such as eight 32-bit values packed into one register. Rewriting the sine-approximation loop with vector data types turns the inner loop into one that processes eight array elements per iteration, and combining this with multiple cores multiplies peak throughput, for example sixteen cores times eight-wide vectors giving 128 pieces of data processed at once.

Divergence and utilization: when if-statements hurt SIMD (53:46)

When a loop body contains an if/else and different vector lanes need different branches, the hardware executes the if branch across all lanes while masking off lanes that did not need it, then does the same for the else branch, so no lane runs both but many lanes sit idle during each branch. In a worst case, code can end up running at as little as one over the vector width of its peak utilization. This behavior is called divergence, and code where all lanes always follow the same path is called coherent; GPUs and CPUs both rely on programmers or compilers minimizing divergence to keep SIMD units busy.

Hardware multithreading: hiding memory latency (1:11:04)

The lecture closes by introducing a third technique: giving one core the ability to hold register state for several threads at once, so that when one thread stalls waiting on a slow memory access, the core immediately switches to running an already-ready thread instead of sitting idle. This does not add any new execution capability, it only improves how fully the existing capability is used, and it comes at the cost of extra on-chip storage and a longer completion time for any individual thread since it shares the core with others.

Before you watch

• Watch Lecture 1 first for the definitions of program, instruction, processor state, and the basic cache abstraction this lecture builds on.
• Being comfortable with C-style array loops and basic pointer/array indexing will make the sine-approximation example easier to follow.

Check your understanding

1. What is the difference between a cold miss and a capacity miss, and why does the distinction matter for how you might change a program?
2. Why can't a processor automatically vectorize or parallelize the outer loop of the sine-approximation example the way it can find instruction-level parallelism within a few nearby instructions?
3. In the divergence example, why does running an if/else across a SIMD group end up executing both branches instead of skipping the untaken one?
4. What specifically does hardware multithreading add to a processor, and what does it not add?