Lecture 6: Locality, Communication, and Contention

What this lecture covers

Following a lecture on workload scheduling, this session shifts to the cost of moving data between processors. It answers a practical question: once work is balanced across threads, why can a program still run slowly, and what can a programmer do about it? The lecture works through message passing as an alternative to shared memory, the danger of naive blocking sends causing deadlock, the distinction between communication a program's algorithm inherently requires and communication caused by how hardware actually moves data, and closes with the roofline model for diagnosing whether a program is compute-bound or bandwidth-bound.

After watching, you should be able to explain why shared-memory communication costs vary by hardware topology, implement a grid computation using explicit message passing with ghost cells, recognize and avoid deadlock from naive blocking sends, distinguish inherent from artifactual communication, apply cache blocking and loop fusion to raise arithmetic intensity, and read a roofline plot to judge whether an optimization is worth pursuing.

Key ideas

• Shared memory is not uniform: even in a single chip, the cost of a load or store depends on network topology (rings, crossbars) and which cache bank holds the data, so "shared address space" hides real distance-dependent costs.
• Message passing: each thread has its own private address space, and the only way to move data between threads is explicit send/receive calls tagged with an identifier, unlike shared memory's implicit load/store.
• Ghost cells: in a distributed grid computation, each node over-allocates extra rows to hold copies of neighboring data it needs but doesn't own, refreshed each iteration via messages.
• Blocking vs. asynchronous communication: blocking send/receive only returns once the transfer completes, which is simple but can deadlock if everyone waits in the same order; asynchronous calls return a handle immediately and require checking completion before touching the data again.
• Inherent vs. artifactual communication: inherent communication is required by the algorithm itself (data that truly must move); artifactual communication is extra data moved because of how hardware works, such as whole cache lines or oversized network packets.
• Arithmetic intensity: the ratio of math operations to bytes moved; raising it (via better work partitioning, cache blocking, or loop fusion) is the main lever for speeding up bandwidth-bound code.
• Tiled vs. row-based partitioning: dividing a grid into square tiles rather than row strips reduces the perimeter-to-area ratio, cutting inherent communication roughly from n/P to n/sqrt(P).
• Roofline model: plots achievable performance against arithmetic intensity; programs to the left of the "knee" are memory-bound (capped by a bandwidth-determined slope), programs to the right are compute-bound (capped by peak flops).

Walkthrough

Why shared-memory communication cost is not uniform (2:05)

The lecture opens by pointing out that the abstraction of a single shared address space hides substantial hardware complexity: multicore chips connect cores to memory and to each other through networks such as rings or crossbars, and the L3 cache is often sharded so that the cost of accessing a given address depends on which core issues the request and where that address's data physically lives. On multi-socket boards, a load from a core on one socket to an address near another socket can be noticeably slower even setting aside caching effects. The takeaway is that even though programmers can treat memory as one big shared pool, real placement and topology affect performance.

The message-passing model and ghost cells (7:09)

The lecture introduces message passing as a model where each thread operates in its own private address space, and data only moves between threads through explicit send and receive calls tagged with message identifiers, analogous to mailing a package rather than posting to a shared bulletin board. Reworking the red-black grid solver for a message-passing cluster, each node holds only its slice of the grid plus extra "ghost" rows that store copies of the boundary data owned by neighboring nodes. Before each update phase, nodes exchange these boundary rows via send/receive; the lecture walks through the full per-iteration code, noting that convergence checking (previously done with a shared variable and barriers) is instead achieved by every thread sending its partial difference to one designated thread, which computes and broadcasts a boolean "done" decision.

Blocking sends and the deadlock trap (27:30)

Having established that blocking send/receive only return once the transfer is confirmed complete, the lecture reveals a bug in a naive implementation: if every thread tries to send in the same direction before posting a matching receive, all sends block waiting for a partner that is itself still blocked trying to send, producing deadlock. The fix is to structure sends and receives by parity (alternate who sends first and who receives first) so the pattern always has a receiver ready. This motivates asynchronous send/receive, where a call returns immediately with a handle that can later be checked for completion; the lecture cautions that modifying a variable after an asynchronous send but before confirming completion is a bug, since the library's copy timing is not otherwise guaranteed.

Inherent vs. artifactual communication and tiling (45:42)

The lecture separates communication that is inherent to a computation (data that genuinely must move to get the right answer) from artifactual communication caused by hardware realities like fixed cache-line sizes or minimum network packet sizes. For the grid solver partitioned into row strips, arithmetic intensity works out to roughly n/P operations per byte communicated; message-passing with interleaved rows is far worse, moving two rows of data per row processed. Reshaping the same row-strip partition into square tiles improves the ratio to roughly n/sqrt(P), because a square has a better area-to-perimeter ratio than a thin strip — a substantial improvement on machines with many cores, since it can mean maintaining full utilization with a fraction of the bandwidth.

Cache blocking and loop fusion to raise arithmetic intensity (52:49)

Within a single thread, the lecture shows how iteration order affects cache behavior: sweeping across a grid row by row causes data to fall out of a small cache before it's reused on the next row, so most accesses reload the same cache lines repeatedly. Reordering the traversal (cache blocking) so nearby elements are processed while still cache-resident raises the ratio of useful output per cache line loaded. A second example shows this same idea applied to library-style vector operations: computing a full elementwise expression in a single pass over the arrays, rather than as separate add then multiply passes, cuts the number of array reads and writes and raises arithmetic intensity — an optimization the lecture notes modern deep learning compilers (like a TensorFlow or PyTorch JIT) perform automatically by fusing operations.

Contention as a separate cost from bandwidth (1:03:00)

Using an office-hours analogy (students arriving at the same time all wait behind each other even though each individual visit takes the same time), the lecture distinguishes contention for a shared resource from raw bandwidth limits: average-case bandwidth math can understate real slowdown if many requests arrive at memory simultaneously. It notes that randomizing request timing, similar to staggering when cars enter a highway, is a common technique to reduce contention in practice.

The roofline model (1:09:09)

The lecture closes with the roofline model, a plot of achievable performance (in operations per second) against arithmetic intensity (operations per byte). For low arithmetic intensity, all programs on a given machine top out along the same rising line, since they're capped by memory bandwidth; past a "knee" point, performance flattens at the machine's peak compute rate, and programs there are compute-bound. Placing a program's measured arithmetic intensity and achieved performance on this chart shows immediately whether it's memory-bound (in which case cache blocking, tiling, or fusion are the productive next steps) or already near peak compute (in which case those transformations won't help further). The lecture also notes that a machine with more compute capability needs a correspondingly higher arithmetic intensity to stay compute-bound, and that an algorithmic change reducing total work can be worthwhile even if it lowers arithmetic intensity, as long as overall wall-clock time improves.

Before you watch

• Review the grid solver and shared-memory synchronization (locks, barriers) from the previous lecture, since this session rewrites the same program using message passing.
• Be comfortable with basic cache concepts (cache lines, cache capacity, cache hits and misses).
• Understand static and dynamic work assignment from the prior lecture, since this lecture builds on top of the assignment step by adding communication cost.

Check your understanding

1. Why can two loads to the same shared address take different amounts of time depending on which core issues them?
2. What is a ghost row in the message-passing grid solver, and why does each node need one?
3. Walk through why the naive blocking-send version of the grid solver deadlocks, and explain one way to fix it without switching to asynchronous communication.
4. Why does partitioning a grid into square tiles rather than row strips reduce inherent communication, and roughly how much does it reduce it by on a machine with many cores?
5. Given a roofline plot, how would you decide whether cache blocking is likely to help a specific program go faster?