Lecture 3: Multi-core Arch Part II and ISPC Programming Abstractions

What this lecture covers

This is the third lecture of Stanford's CS149, continuing directly from the previous session's introduction of multicore, SIMD, and hardware multithreading. It opens with a deep review of hardware multithreading, working through examples of how many threads a core needs to fully hide a given memory stall, then shows how multicore, SIMD, and multithreading combine inside a realistic chip model resembling both a laptop CPU and an Nvidia GPU core.

The second half introduces a distinction the course had not yet made explicit: latency versus bandwidth, illustrated with a highway and a laundry pipeline before being applied to memory systems, where a simple array-addition program is shown to run at roughly one percent efficiency because arithmetic throughput vastly outstrips memory bandwidth. The lecture closes by introducing ispc, a small C-like language used in the course's first assignment, covering its program count and program index abstractions and the difference between interleaved and blocked work assignment. After watching, you should be able to compute how many threads hide a given latency and explain why bandwidth, not just cache misses, limits many real programs.

Key ideas

• Switch-on-stall multithreading: a core holding several threads' register state switches to another ready thread the instant the current one stalls, keeping execution units busy at the cost of any single thread finishing slower.
• Threads needed to hide latency: if a thread does a fixed amount of work before a stall of a given length, the threads required for full utilization equal roughly the stall length divided by the work per thread, plus one.
• Latency versus throughput (bandwidth): latency is how long one operation takes end to end; bandwidth is how many operations complete per unit time. Adding parallel lanes raises bandwidth without changing individual latency.
• Pipelining: overlapping stages of repeated work, like washing while a previous load dries, so throughput is set by the slowest stage even though each item's own latency is the sum of all stages.
• Memory bandwidth wall: a chip needing roughly 100 terabytes per second of data to feed its arithmetic units can be fed by only about 1 terabyte per second of real memory bandwidth, so low-arithmetic-intensity code can run at roughly 1% of peak efficiency regardless of latency hiding.
• Arithmetic intensity: the ratio of math operations to bytes of memory traffic; graphics and machine learning workloads have much higher ratios than simple elementwise array operations, which is why they use chips like this efficiently.
• ispc program instance: calling an ispc function spawns a gang of program instances (not called threads), each with its own local variables, distinguished by the built-in program index, with program count instances running total.
• Interleaved versus blocked assignment and foreach: a programmer can manually assign loop iterations to program instances round-robin (interleaved) or in contiguous chunks (blocked), or use foreach to let ispc choose.

Walkthrough

Reviewing hardware multithreading with concrete thread counts (8:14)

The lecture re-derives switch-on-stall multithreading using an office-hours analogy: helping many students at once by interleaving them is more efficient overall but slower for any individual student. It then works a numeric example where a thread does three cycles of math before a twelve-cycle memory stall, showing the core runs at only 20% utilization with one thread, and that five threads are needed for full utilization because each extra thread supplies three more cycles of useful work to cover the stall. Changing the program to six cycles of math before the same stall lowers the requirement to three threads, showing the ratio of computation to memory latency sets how much multithreading is needed.

Combining multicore, SIMD, and multithreading into one chip (17:22)

The lecture builds up a series of fake chip diagrams: a sixteen-core, four-way-multithreaded chip needing 512 independent pieces of work to run at peak with full latency hiding, then a closer approximation of an actual laptop-class core that is two-way multithreaded and superscalar with vector arithmetic units. This illustrates how Intel's hyper-threading was motivated by not finding enough instruction-level parallelism within a single thread to fill all execution units. A brief comparison sketches an Nvidia GPU core as roughly 32-wide vectors drawing from many more resident threads per core, at much larger scale than a CPU core.

Latency versus bandwidth: highways and laundry (38:45)

Before returning to hardware, the lecture sets up vocabulary with everyday analogies: driving from San Francisco to Stanford has a fixed latency, but throughput (cars completed per hour) can rise independently by driving faster, adding lanes, or spacing cars closer together. A laundry example with a washer and dryer as two pipeline stages shows a single load's latency is the sum of both stages, but running many loads back to back gives a steady throughput set by the slower stage, with unfinished loads piling up as buffered work in between.

Why arithmetic throughput outruns memory bandwidth (53:55)

Applying the pipelining model to a load-heavy instruction stream, the lecture shows a processor issuing loads that queue up because memory bandwidth, not latency, becomes the bottleneck once enough loads are in flight. Scaling this up to a chip capable of roughly eight trillion math operations per second, each needing about twelve bytes of memory traffic, implies a need for around 100 terabytes per second of bandwidth, while real memory systems provide roughly one terabyte per second. A simple elementwise array-addition program therefore runs at about 1% efficiency on such a chip regardless of latency hiding, because the bottleneck is bandwidth, not outstanding requests or prefetch accuracy.

Introducing ispc: program instances, gangs, and foreach (1:03:04)

The lecture rewrites the course's running sine-approximation example in ispc. Calling an ispc function spawns a gang of program instances, each running the function body with its own local variables but distinguished by the built-in program index, out of program count total instances. Two versions of the computation are shown: one where instances take interleaved array elements (each handles every eighth element) and one where each handles a contiguous block, both producing identical results but assigning work differently. The lecture ends with foreach, which lets the ispc runtime choose how to divide loop iterations among instances, stressing that a programmer should first state what a program computes before deciding which instance does what.

Before you watch

• Watch Lecture 2 first, since this lecture opens with a direct review of its multicore, SIMD, and hardware multithreading diagrams and reuses the sine-approximation example introduced there.
• Being comfortable with basic loop and array code in C will help with the ispc section, which reuses that same syntax with a few new keywords.

Check your understanding

1. If a thread performs six cycles of computation before a twelve-cycle memory stall, how many threads does a core need to reach full utilization, and why?
2. Why does adding more outstanding memory requests or a better prefetcher fail to fix the array-addition example's low efficiency?
3. In the laundry pipeline analogy, why is the throughput of finished loads limited by the dryer even though the washer finishes faster?
4. In ispc, what is the difference between assigning array elements to program instances in an interleaved pattern versus a blocked pattern, and what does foreach let you avoid deciding?