Lecture 1: Why Parallelism? Why Efficiency?

What this lecture covers

This is the first lecture of Stanford's CS149 parallel computing course. It opens by asking why parallelism matters at all: the instructors run a live exercise where students add up sets of numbers alone, then in pairs, then in groups of four, and finally the whole room tries to count how many people are present. Each round exposes a different obstacle to speedup, mainly the cost of communicating and synchronizing between workers, which sets up the course's central theme that parallelism and efficiency are not the same thing.

After course logistics, the lecture moves into technical groundwork: what a computer program and an instruction actually are, what a processor does when it executes one, and how automatic instruction-level parallelism let chip designers speed up programs without programmers changing anything. It closes by introducing memory as an abstract array of addressable values and previewing caches as a faster, smaller storage layer sitting between the processor and DRAM. By the end you should be able to describe a processor in terms of control, arithmetic, and state, and explain why communication overhead limits real-world speedup.

Key ideas

• Parallelism versus efficiency: getting a speedup from more workers is not automatic; communication and coordination costs can erase most of the theoretical gain, as shown when two people took barely any less time than one to add 16 numbers.
• Sequential semantics, parallel execution: a program written as a strict sequence of instructions can still be run out of order or in parallel by hardware, as long as the final result matches what the sequential order would have produced.
• Instruction: a single command a processor executes, typically an arithmetic operation, a memory access, or a control-flow jump.
• State: the values held in processor registers and in memory; executing an instruction changes this state.
• Superscalar execution: a processor automatically finds independent instructions in a single instruction stream and executes more than one of them per clock, without the programmer's knowledge.
• The end of automatic scaling: transistor counts kept growing, but processors could only find about three or four independent operations per program automatically, and clock frequency stopped increasing around 15 years before this lecture because power grows roughly with the square of frequency.
• Memory as an abstraction: memory is logically just an array of addresses, each holding a value; a load reads a value at an address and a store writes one.
• Cache: a small, fast storage layer close to the processor that holds a copy of recently used memory so the processor does not have to wait hundreds of cycles for DRAM every time.

Walkthrough

The class experiment: adding numbers as parallel computing (5:10)

The instructor times a single volunteer adding 16 numbers, then runs the same task with two volunteers who cannot talk to each other, only pass a note. The two-person team is expected to be about twice as fast but is barely faster than one person, because getting a single number from one side of the room to the other consumes most of the saved time. The exercise then scales to four people with unequal workloads (exposing load imbalance), four people with equal workloads and a self-organized scheme (pooling and grabbing tasks), and finally the whole class estimating its own headcount, which drags out to several minutes instead of the expected few seconds because of the cost of moving people and communicating partial counts. The takeaway is that the hard part of parallel computing is usually not finding the parallel work but minimizing the cost of communication and synchronization between workers.

Course logistics (33:04)

The lecture covers how the course website and slide comments work, that there is no textbook, and that grading is dominated by four programming assignments (a thread pool, a CUDA renderer, and a Transformer inference kernel among them), with written assignments, participation comments, and eight total late days available across the quarter.

Why more transistors stopped meaning more speed (41:12)

Historically, programs got faster for free as processors got faster year over year. Smaller transistors allowed higher clock frequencies and more transistors per chip, and architects used the extra transistors to automatically extract parallelism from single-threaded programs. But clock frequency stopped climbing roughly 15 years before this lecture because power consumption scales with the square of frequency, making further increases too costly in heat and energy.

From instructions to automatic parallelism (50:23)

Using a short sequence of instructions computing something like a dot product, the lecture shows that even code written as a strict sequence has dependency structure: some operations only depend on earlier ones that are unrelated to each other, so they can run in parallel while others must wait. Processors exploit this by reordering independent instructions across multiple execution units, a technique called superscalar execution, without the programmer writing anything different. Studies showed this approach runs out of headroom at around three or four simultaneous operations, since ordinary programs simply do not contain more automatically discoverable parallelism than that.

Memory, latency, and the first look at caches (1:03:31)

Memory is introduced as an abstraction: give it an address and it returns the value stored there, regardless of whether the underlying implementation is DRAM or an on-chip cache. DRAM can take hundreds of processor cycles to respond, so processors keep a small amount of much faster storage, a cache, holding copies of recently accessed data. The lecture ends by showing that data moves between memory and cache in fixed-size blocks called cache lines, so accessing one address effectively preloads nearby addresses, setting up the deeper cache discussion in the next lecture.

Before you watch

• No prior lecture in this course exists yet; this is the first session, though familiarity with basic assembly instructions and threads (as covered in an introductory systems course) is assumed.
• It helps to already know roughly what a CPU register and a memory address are.

Check your understanding

1. Why did adding numbers with two people barely beat adding them with one person, even though the workload should have split evenly?
2. What two hardware trends historically let sequential programs get faster automatically, and why did each of them stop?
3. What does it mean for a processor to execute instructions out of order while still respecting a program's sequential semantics?
4. What is the difference between what memory means abstractly and how it is implemented in DRAM versus a cache?