Lecture 23b: Multi-Core Issues in Caching

What this lecture covers

Building on single-core cache design, this lecture asks what changes when many cores share a cache hierarchy. It covers the decision to make a cache private to one core or shared across cores, the benefits and costs of sharing hardware resources among threads, and the coherence problem that arises when the same memory location can be cached in multiple private caches at once.

After watching, you can explain why L1 caches are almost always private while L3 caches are almost always shared, describe how unfair sharing can destroy one thread's performance when it runs alongside another, and outline how snooping (broadcast-based) and directory-based cache coherence protocols keep multiple cached copies of a block consistent.

Key ideas

• Private vs. shared cache: a private cache belongs to one core (a shared block may be replicated across caches); a shared cache is used by multiple cores, so a block exists in only one place, avoiding replication.
• Resource sharing trade-off: sharing improves utilization (idle capacity in one thread can be used by another, no static-partition waste) but creates contention, since one thread's use of a shared resource can degrade another's performance.
• Performance isolation: without it, an application's cache performance depends on which other threads or applications happen to run alongside it, making performance unpredictable even when a programmer optimizes carefully for the hardware.
• Unfair sharing: a memory-intensive thread can evict most of a shared cache's contents, starving a less-demanding co-running thread even though that thread's needs were modest.
• Cache coherence: the guarantee that all processors see a consistent view of a cache block's value, needed because a shared-memory program relies on one thread's write being visible to another thread's read.
• Snooping (broadcast) coherence: every write to a cached block is broadcast on a shared bus so other caches can invalidate or update their copy; simple and low latency, but not scalable beyond a modest number of cores.
• Directory-based coherence: a directory tracks which caches hold each block and arbitrates read/write permission, avoiding a shared bus bottleneck at the cost of extra indirection latency; this scales to systems with many more processors.
• MESI protocol: a four-state (Modified, Exclusive, Shared, Invalid) coherence scheme used in real processors, extending the two-state valid/invalid example shown in the lecture.

Walkthrough

Private vs. shared caches in multicore systems (0:02)

The lecture opens by noting that L1 caches are essentially always private because they are tightly coupled to a core's pipeline, while L2 caches can be either private or shared depending on the design, and L3 caches are almost always shared. It gives the example of a separate chip bonded on top of a processor die purely to act as a large shared cache for many cores, illustrating how far designers now go to add shared cache capacity. It frames the central design questions for multicore caches: should caches be shared or private, how do you maximize system-wide throughput rather than single-thread performance, and how do you provide quality of service and predictable performance to individual threads.

Benefits and costs of resource sharing (1:02)

Sharing a resource such as a cache, bus, or memory bandwidth improves utilization: idle capacity left by one thread can be used by another, and shared data does not need to be replicated across private caches, which also reduces communication latency and fits naturally with the shared-memory programming model. However, sharing also introduces contention, since one thread's occupation of a resource blocks another thread from using it, and it can eliminate performance isolation, meaning a program's performance now depends on which other threads run alongside it rather than being predictable on its own.

Private vs. shared cache trade-offs, and unfair sharing (12:11)

Shared caches offer higher effective capacity (no static partitioning, no fragmentation) and simpler coherence, since a block lives in only one place, but they cannot be customized or tightly coupled to a single core and require a network to reach the cache, adding latency. They also let cores interfere with each other: one thread's access pattern can evict another's useful data. The lecture illustrates this with two co-running threads where a memory-intensive thread occupies nearly the entire shared cache, leaving a much less demanding thread with far less space than it would get running alone, significantly reducing its performance. The lecture frames the underlying tension as resource sharing (better utilization) versus partitioning (predictable, isolated performance), noting that real systems (for example, Intel's cache and memory-bandwidth partitioning features) try to combine both by making shared resources controllable and partitionable.

The cache coherence problem (18:19)

In the shared-memory programming model, one thread writes to a memory address and another thread reads from it, so every read should observe the most recently written value (assuming proper synchronization). The lecture shows how this breaks down when the same address is cached privately by two processors: one processor updates its cached copy in a write-back cache, but main memory and the other processor's cache remain stale, so a naive read by the second processor would return an outdated value. Cache coherence protocols exist to prevent this inconsistency.

Snooping (broadcast-based) coherence (21:24)

The first solution the lecture presents is a broadcast-based protocol: when a processor writes to a block, it broadcasts that write (or an invalidation request) over a shared bus, and every other cache holding that block invalidates (or updates) its copy. The lecture walks through a simple two-state (valid/invalid) coherence state machine for a write-through cache, where a bus-write signal to a block a cache holds causes that cache to invalidate it. This approach has low latency since there is no intermediary, but it does not scale well because it depends on all caches sharing a single bus where broadcasts are visible to everyone.

Directory-based coherence (24:25)

The second approach introduces a directory as a "middleman" that tracks which caches hold each block, using something like a bit vector per block. A cache must request permission from the directory before writing; the directory asks other caches holding the block to invalidate it, waits for their acknowledgments, then grants write permission. This avoids the scalability limits of a shared bus, at the cost of added indirection latency, and is the approach used to scale coherence to systems with tens of thousands of processors. The lecture notes that an exclusive bit lets a cache that holds the only copy of a block update it silently without contacting the directory again, and closes by naming the MESI protocol (Modified, Exclusive, Shared, Invalid) as the four-state coherence scheme used in real processors like the Pentium Pro, extending the simple valid/invalid example shown earlier.

Before you watch

• Review the previous two lectures on cache design (placement, replacement, write policies) and the multi-level cache hierarchy, since this lecture assumes that background.
• Be familiar with write-back vs. write-through caches and the dirty bit, since the coherence examples build directly on them.
• Some prior exposure to shared-memory parallel programming (threads reading and writing shared variables) is helpful but not required; the lecture briefly reviews it.

Check your understanding

1. Why are L1 caches almost always private while L3 caches are almost always shared?
2. Give an example of how sharing a cache between two threads can help one workload but hurt another running alongside it.
3. Why does a shared-memory program require cache coherence, and what specifically goes wrong without it?
4. Compare the latency and scalability trade-offs between snooping (broadcast-based) and directory-based coherence protocols.
5. What does the exclusive bit in a directory-based protocol let a cache do, and why is that an optimization?