Lecture 4: Parallel Programming Basics

What this lecture covers

This lecture answers a question students kept raising in office hours after the first assignment: what exactly does an ISPC program mean, separate from how it runs on hardware? The instructor revisits the gang-of-program-instances model, the for each construct, and ISPC tasks, then shows a live demo of why thread-per-task scheduling is disastrously slow compared to a fixed thread pool. It sits early in the course, right after the SIMD and ISPC material, and builds the vocabulary (decomposition, assignment, orchestration, mapping) that the rest of the course reuses.

After watching, you should be able to distinguish what an ISPC program specifies from how the compiler implements it, explain why writing to shared program-instance variables without a reduction function is a race condition, and walk through applying decomposition, assignment, and orchestration to a real iterative solver, including why locks and barriers are needed and how to reduce synchronization overhead.

Key ideas

• Program instance: one of the gang-size logical copies of an ISPC function, distinguished only by the value of programIndex; the model says nothing about how those copies actually execute.
• Gang size vs. SIMD width: setting gang size equal to the machine's vector width lets the compiler implement all instances as SIMD lane operations within a single thread.
• for each: a construct that hands the compiler a set of independent loop iterations without specifying which program instance runs which iteration, so the compiler is free to interleave, block, or otherwise schedule them.
• Undefined behavior under for each: code that reads a neighboring, not-yet-known loop index (e.g. x[i-1]) has no defined output because the iteration order is left to the compiler.
• Cross-instance reduction: ISPC provides library functions (like reduce_add) to safely combine per-instance values into a single uniform value, avoiding a race on a shared accumulator.
• Tasks: the task abstraction lets a programmer decompose work into many independent units and leave assignment of tasks to worker threads up to the ISPC runtime.
• Decomposition, assignment, orchestration, mapping: the four-step vocabulary for parallelizing any program — split work into independent pieces, assign pieces to workers, synchronize/communicate between workers, and map workers onto actual hardware.
• Amdahl's law in practice: even a small serial fraction sharply caps achievable speedup as core counts grow, which is why picking a parallel-friendly algorithm matters as much as parallelizing the one you started with.

Walkthrough

Reviewing the ISPC program model (3:07)

The lecture reopens the sinx example from the previous class to re-establish that calling an ISPC function does not transfer control once, the way a normal C call does. It spawns gang-size copies of the function body, each seeing a different programIndex, and the return happens only once every copy finishes. The instructor stresses that nothing said about this model implies an implementation: it would be technically valid, if silly, to implement it as a for-loop that calls the scalar function gang-size times in sequence. In practice, when gang size equals the machine's SIMD width, the compiler instead emits a single stream of vector instructions, so a traced execution just shows scalar instructions, then a burst of vector instructions, then scalar again.

Interleaved vs. blocked assignment and the for each contract (9:13)

Two hand-written versions of the sine program divide loop iterations differently across program instances: one interleaves indices across instances, the other blocks them. A table of which instance touches which index each iteration makes clear why interleaved access is preferred: it produces contiguous memory accesses that map cleanly onto cache lines, while a blocked/strided access pattern can force each SIMD lane to touch a different cache line or even a different page. The lecture then introduces for each, which drops the requirement to specify this mapping at all — the programmer just states there are n independent iterations, and the compiler decides how to spread them across program instances (usually favoring good memory locality).

Race conditions and safe reduction inside for each (20:19)

Because for each iterations can run in any order, a program that reads a value written by a neighboring iteration (such as x[i-1]) has undefined output, since the answer depends on the unspecified schedule the compiler picks. A related example shows why a naive parallel sum is broken twice over: a per-instance sum cannot be returned because ISPC only allows a single uniform return value, and even after making sum uniform, unsynchronized += from multiple instances is a race. The fix keeps a private partial accumulator per instance during the loop and calls a cross-instance reduction function once at the end, which the compiler lowers to a vectorized accumulate followed by a short sequential sum over the lanes.

Tasks and why thread-per-task scheduling is slow (27:25)

ISPC tasks extend the same idea one level up: instead of assigning loop iterations to program instances, a program can create many independent tasks and leave their assignment to worker threads up to the runtime. A live demo compares three strategies for running many cheap "do nothing" tasks: sequential execution, spawning a new OS thread per task, and a fixed pool of eight worker threads pulling tasks from a queue. The one-thread-per-task version was roughly 300x slower than the thread-pool version, and even the thread pool lost to plain sequential execution when the task itself does almost no work — illustrating that thread creation and OS-level scheduling overhead can dominate unless task granularity is large enough.

Decomposition, assignment, orchestration, and Amdahl's law (40:38)

The lecture names the general recipe for parallelizing any program: decomposition (finding independent work), assignment (mapping that work to workers), orchestration (the synchronization and communication needed to coordinate workers), and mapping (placing workers onto actual hardware). It stresses that decomposition is almost always the programmer's job; automatic parallelizing compilers for general code largely do not exist. A brightness-and-average image example demonstrates Amdahl's law concretely: if the sequential portion of a two-phase program can't be shrunk, total speedup is bounded regardless of processor count, and the fraction of serial code has an outsized effect on achievable speedup as core counts grow into the tens or hundreds.

Case study: a grid solver and the checkerboard algorithm (53:54)

A convergence-style grid solver (each cell's new value depends on its previous neighbors) looks unparallelizable as written, because later iterations of the inner loop depend on values just computed earlier in the same iteration. Rather than trying to extract limited parallelism from that exact algorithm, the lecture swaps in red-black checkerboarding: update all "red" cells in parallel using only "black" neighbor values, then update all "black" cells using the new "red" values. This changes the algorithm (it converges to the same answer but may need more iterations) specifically to make it friendly to parallel execution, illustrating that choosing a different, more parallelizable algorithm is often more productive than fighting to parallelize the original one.

Implementing the solver with threads, locks, and barriers (1:05:03)

The solver is written twice: first as abstract parallel work (a for all red cells style loop with no explicit threads), then as explicit shared-memory threaded code where each thread computes its own row range from its thread ID. The threaded version needs a lock around updates to a shared diff accumulator to make the read-modify-write sequence atomic, and a per-thread local accumulator is used to avoid taking that lock inside the innermost loop. Three barriers are required in the loop body: one so no thread checks convergence before all threads finish updating diff, and others to prevent a thread from resetting diff for the next iteration before others have read it, or from writing into diff before others have finished checking it. The lecture closes with a challenge: figure out how to reduce the three barriers to just one.

Before you watch

• Review the ISPC gang/program-instance model and the sinx example from the previous lecture, since this session assumes familiarity with it.
• Be comfortable with basic multithreading concepts (thread creation, locks, mutual exclusion) from a systems or OS course.
• Know what a race condition is and why unsynchronized read-modify-write sequences on shared memory can lose updates.

Check your understanding

1. Why is it invalid, under the for each contract, to write a loop body that reads a neighboring iteration's just-computed value?
2. What two separate problems make the naive "increment a per-instance sum and return it" program invalid in ISPC, and how does the corrected version fix each one?
3. Why did the thread-per-task strategy in the demo perform dramatically worse than the fixed-size thread pool, even though both eventually run the same amount of work?
4. Explain why red-black checkerboarding makes the grid solver parallelizable when the original update order does not.
5. In the threaded solver implementation, what specifically goes wrong if the barrier before the convergence check is removed?