Lecture 5: Work Distribution and Scheduling

What this lecture covers

This lecture opens by resolving a puzzle left from the previous session (how to implement the grid solver's barrier synchronization with just one barrier instead of three), then moves into the week's real topic: how to assign parallel work to threads so that every worker stays busy. It sits right after the introduction to decomposition and assignment, and focuses specifically on the "assignment" step, leaving communication and memory efficiency for the next lecture.

After watching, you should be able to choose between static, semi-static, and dynamic work assignment for a given workload, reason about the tradeoff between task granularity and scheduling overhead, and explain how Cilk's spawn/sync model and its work-stealing scheduler let a programmer expose recursive parallelism without hand-managing threads.

Key ideas

• Reducing false dependencies: replacing a single reused accumulator variable (like diff) with per-iteration copies removes an unnecessary synchronization dependency and lets the grid solver's three barriers collapse to one.
• Static assignment: dividing work into fixed chunks up front (e.g. equal-sized image regions) works well when the cost per unit of work is predictable, either uniformly or on average.
• Semi-static assignment: recomputing a static assignment periodically as workload characteristics drift, useful in long-running simulations or training jobs.
• Dynamic assignment: workers pull from a shared work queue (often just an atomically incremented counter) whenever they go idle, which handles unpredictable per-task cost at the price of synchronization overhead.
• Task granularity tradeoff: smaller tasks improve load balance but increase synchronization overhead; the fix is usually to batch a few units of work per queue pop rather than redesigning the scheme.
• Distributed work queues: giving each thread its own queue, with idle threads stealing from others, reduces contention compared to one shared queue.
• Cilk's spawn/sync: spawn marks a function call as asynchronous work the caller need not wait for; sync is a barrier for everything spawned in the current block, and there's an implicit sync at function return.
• Work-stealing policy: each thread runs its own work locally (LIFO) and steals from the front (largest, oldest) of another thread's queue when idle, which minimizes synchronization and tends toward good load balance.

Walkthrough

Closing the barrier puzzle from last time (0:05)

The lecture revisits the grid solver's three barriers and explains that they existed because a single diff variable was being reused across loop iterations, creating a false dependency between iterations that never actually needed to interact. The fix is to give each iteration (or really, just the previous, current, and next iteration) its own copy of diff, the same trick used earlier to give each thread a private partial accumulator. With separate copies, threads never need to wait for others to finish reading or resetting a shared value, and the three barriers collapse into one.

Static assignment and when it works well (8:13)

The lecture states a strong recommendation before introducing any scheduling technique: always implement the simplest possible parallel scheme first, measure it, and only add sophistication if the numbers justify it. It then works through static assignment using the mandelbrot-style image example from assignment one: dividing pixels into contiguous, equal-sized chunks works when the cost per pixel is either predictable or, more commonly, unpredictable per-pixel but uniform on average across a large enough interleaved chunk. Static assignment's main benefit is that workers never need to communicate once the assignment is fixed, since each knows its share up front. Semi-static assignment extends this to long-running simulations (such as a turbulent-flow mesh around an airplane wing) where the workload is recomputed periodically as conditions change, rather than fixed for the whole run.

Dynamic assignment with a shared work queue (16:18)

For workloads where per-task cost can't be predicted (the example given is testing primality of numbers in an array), the lecture builds a dynamic scheme: an atomically incremented shared counter acts as a work queue, and each thread grabs the next index whenever it's free. This guarantees all elements get processed exactly once and terminates naturally once the counter passes the array bound. The lecture frames this as a general pattern: whatever the units of work are, throw them all into a shared queue and let idle workers pull from it, trading some synchronization cost for good load balance.

Measuring overhead and tuning task granularity (25:24)

The lecture walks through how to diagnose whether a dynamic scheme is worth its cost: time the whole program, then time only the useful work (e.g. calls to the primality test), and see how much of the total is overhead from queue synchronization. If overhead is a small fraction of total time, there's little room for improvement; if it's a large fraction, the fix is often simply to increase task granularity, for example by incrementing the shared counter by more than one at a time so each queue pop hands out several units of work. This reduces the number of synchronization events without sacrificing much load balance, since modern hardware makes the queue operation itself cheap relative to a reasonably sized batch of work. An example with deliberately uneven task sizes also shows that even dynamic scheduling can get unlucky if a very large task happens to be picked up last, motivating strategies like scheduling larger tasks first when task cost can be estimated in advance.

Distributed work queues and revealing dependencies (33:28)

To reduce contention on a single shared counter or queue, especially at large thread counts, the lecture notes that systems commonly give each thread its own queue rather than sharing one, applying the same "make private copies, then reconcile" idea used for the diff variable earlier. It also flags that real scheduling problems often involve dependencies between tasks (task bar can only run after task foo completes), which is more complex than the fully independent tasks used in the examples so far and is the subject of the course's second assignment.

Cilk's spawn/sync programming model (41:37)

Using quicksort as a running example, the lecture introduces Cilk, a small extension available in many C/C++ compilers that adds two constructs: spawn, which marks a function call as work the caller may continue past asynchronously, and sync, which acts as a barrier waiting for everything spawned in the current block to finish (with an implicit sync at the end of every function). The semantics are defined independent of implementation: stripping out all spawn/sync keywords yields a valid sequential program, and turning every spawn into an actual thread creation with sync as a join is also valid, just slow. In the quicksort implementation, each recursive call is spawned so both halves of the partition can run concurrently, and the recursion itself progressively reveals more and more independent parallel work as it descends toward the base case.

Work stealing: running the child vs. the continuation first (57:53)

A practical Cilk implementation uses a thread pool where each worker keeps its own double-ended queue of pending work. When a thread spawns a function, it can choose to start executing the spawned call immediately and queue the "continuation" (the rest of the current function) for possible stealing, or vice versa. The lecture shows that running the child first and queuing the continuation produces exponentially shrinking tasks in recursive algorithms like quicksort, which then favors a specific stealing rule: idle threads should steal from the top of another thread's queue, where the oldest and largest pending tasks sit, while the owning thread pushes and pops from the bottom, matching normal sequential call order. This keeps threads out of each other's way, minimizes synchronization, and is theoretically near-optimal even when a stealing thread picks a victim at random.

Implementing sync with distributed queues (1:11:03)

The lecture closes by sketching how sync is implemented once work has been distributed across per-thread queues via stealing: each spawning block keeps a reference count of how many of its spawned calls are still outstanding, incremented whenever a piece of that block's work gets stolen and decremented as each piece completes. Whichever thread happens to finish the last outstanding piece of work for a block continues on with that block's continuation, a scheme called greedy join scheduling. The lecture notes this is more advanced than what the second assignment requires, but explains the underlying theory of why work stealing achieves good load balance with low synchronization cost.

Before you watch

• Review the previous lecture's grid-solver example (locks, barriers, and the diff accumulator), since this lecture opens by finishing that discussion.
• Be familiar with basic quicksort (pivot selection, partitioning, and recursion), which is used throughout as the running example for divide-and-conquer parallelism.
• Understand what an atomic increment does and why it's used to implement a lock-free counter-based work queue.

Check your understanding

1. Why did reusing a single diff variable across loop iterations force the grid solver to use three barriers instead of one?
2. Under what conditions does a static work assignment perform about as well as a dynamic one, and when does it fail?
3. If profiling shows that over half your program's runtime is spent acquiring a work-queue lock, what is the simplest fix to try first, and why does it help?
4. In Cilk's model, what exactly does spawn guarantee about when a function runs, and what does sync guarantee?
5. Why does a work-stealing scheduler have idle threads steal from the top of a victim's queue rather than the bottom, while the owning thread works from the bottom?