Lecture 11: Hardware Acceleration

What this lecture covers

Up to this point in the course, the needle deep learning library has relied on NumPy for its actual array arithmetic. This lecture starts the shift toward replacing that backend with a custom tensor library, motivating the effort by pointing out that modern deep learning depends on hardware acceleration to keep up with model and dataset size, and that understanding these techniques explains why identical-looking code can run at very different speeds.

The lecture covers general CPU acceleration techniques: vectorization, memory layout and strides, and parallelization, then works through a detailed case study of accelerating matrix multiplication, first the naive version, then register-tiled, then cache-and-register-tiled. After watching, you should be able to explain why a naive matrix multiplication is bottlenecked by memory loads rather than arithmetic, and how tiling reduces the number of loads by reusing data already sitting in fast memory.

Key ideas

• Vectorization: modern CPUs offer instructions that load, operate on, and store multiple contiguous values at once (for example four floats per instruction) instead of one at a time, but this requires the data's memory address to be suitably aligned.
• Memory alignment: loading a vector register efficiently generally requires the array's starting address to be a multiple of the vector width in bytes, which is why array libraries often allocate memory with stricter alignment than a default malloc.
• Data layout: row-major vs. column-major: a 2D array can be stored in memory as row-major (C convention, index i*n+j) or column-major (Fortran and classic BLAS convention, index j*shape0+i); both are special cases of a more general strided layout.
• Strides: representing an array by strides per dimension generalizes row-major and column-major layouts, and lets operations like transpose, slicing and broadcasting be done by changing strides instead of copying data — though non-contiguous strided arrays are harder to vectorize and sometimes must be made "compact" (contiguous) before certain operations.
• Parallelization: loop iterations can be distributed across CPU cores (for example, with OpenMP-style annotations), splitting the work so each core handles a chunk of the array.
• Memory hierarchy dominates cost: fetching data from DRAM costs roughly 200 nanoseconds versus about 0.5 nanoseconds from L1 cache, a roughly 200x difference, so an algorithm's practical speed depends heavily on how much data it can keep in fast memory rather than just its arithmetic operation count.
• Register tiling: computing a small submatrix at a time and reusing loaded values across multiple inner-loop iterations reduces the total number of DRAM loads from being proportional to n^3 down to n^3 divided by the tile size.
• Cache-aware tiling: adding a second level of blocking that stages data through L1 cache before it reaches registers further reduces DRAM traffic, at the cost of needing the tile sizes to divide evenly into the register tile sizes and to fit within cache capacity.
• Reuse from loop structure: looking at which array index does not depend on a given loop variable (for example, A[i,k] doesn't depend on j) shows which loops can be tiled to reuse already-loaded data, a general principle behind matrix multiplication and convolution acceleration.

Walkthrough

Why learn hardware acceleration (1:01)

The lecture frames hardware acceleration as necessary because training modern deep networks requires GPUs and other accelerators to keep pace with growing model and dataset sizes, and because understanding these techniques clarifies practical questions like why certain code runs fast or slow, or how to implement a new operator efficiently on available hardware. Machine learning frameworks are described as sitting on two layers: a computational graph layer (what the course built already) on top of a tensor linear algebra layer that performs the actual array arithmetic, which is the layer this lecture focuses on optimizing.

Vectorization (4:01)

Instead of adding two 256-element vectors one element at a time, modern hardware can load several contiguous values into vector registers, add them in a single instruction, and store the result back, cutting a 256-iteration scalar loop down to 64 vectorized iterations. This requires the underlying memory addresses to be aligned to the vector width, which is why numerical libraries often allocate memory with alignment guarantees beyond the platform default.

Data layout and strides (9:06)

A multi-dimensional array must be mapped onto a flat, linearly addressed memory space. The lecture contrasts row-major order (A[i,j] at i*shape1+j, the C and NumPy default) with column-major order (A[i,j] at j*shape0+i, used by Fortran and classic BLAS libraries), then generalizes both as special cases of a strided representation, where each dimension has its own stride used to compute an offset. Strides make operations like transposition, slicing, and broadcasting possible without copying data, simply by changing the stride and shape metadata, but non-contiguous strided arrays are harder to vectorize efficiently and sometimes need to be made contiguous ("compacted") before certain computations.

Parallelization (17:09)

A loop can be annotated for parallel execution (the lecture shows an OpenMP-style example) so that different iterations are assigned to different CPU cores, dividing the work — for instance across four cores — to speed up the overall computation.

Naive matrix multiplication and the memory hierarchy (18:12)

A straightforward triple-loop matrix multiplication computing C = A @ B^T performs n^3 multiplications, an O(n^3) algorithm that most production BLAS libraries still use rather than asymptotically faster algorithms. The key insight is that the real bottleneck is not arithmetic but data movement: fetching from DRAM costs around 200 nanoseconds versus roughly 0.5 nanoseconds from L1 cache, so an algorithm's practical performance depends on how effectively it reuses data already loaded into fast memory.

Register tiling (24:15)

Instead of computing one output element at a time, register tiling computes a small v1-by-v2 submatrix per outer iteration, loading a v1-by-v3 strip of A and a v3-by-v2 strip of B into registers and reusing each loaded value across multiple inner-product computations. Counting loads shows the total DRAM traffic for A and B drops to n^3/v2 and n^3/v1 respectively, so larger tile sizes reduce loading cost, limited only by the number of registers available (v1*v3 + v2*v3 + v1*v2 registers used).

Adding cache-level tiling (32:25)

A further blocking level stages data through L1 cache before it reaches registers: outer tiles of size b1-by-b2 (multiples of the register tile sizes v1, v2) are first loaded from DRAM into cache, and register tiling is then applied within that cached block. Counting loads again shows DRAM-to-cache traffic becomes n^2 for A and n^3/b1 for B, subject to the cache-block sizes dividing evenly into the register tile sizes and fitting within the L1 cache capacity. Combining both levels of tiling can produce order-of-magnitude or larger speedups over the naive implementation, even though the asymptotic complexity remains O(n^3).

The general principle: reuse from loop structure (41:33)

The lecture closes by generalizing the tiling case study: in C[i,j] = sum_k A[i,k] * B[j,k], the access to A does not depend on j and the access to B does not depend on i, so tiling along those "missing" loop indices creates opportunities to reuse a loaded value across multiple iterations before discarding it. This same principle of identifying reuse patterns from indexing applies beyond matrix multiplication, including to convolutions.

Before you watch

• Be comfortable with how the needle library's computational graph and tensor operations work from earlier lectures, since this lecture assumes that layer already exists.
• Basic familiarity with nested loops for matrix multiplication and with the concept of Big-O complexity will help with the case study.
• No GPU background is needed yet; this lecture is CPU-focused and precedes the GPU acceleration lecture.

Check your understanding

1. Why does the naive matrix multiplication's practical speed depend heavily on memory access patterns even though its arithmetic complexity is O(n^3)?
2. How do strides generalize row-major and column-major array layouts, and why do they make transposition and slicing cheap?
3. In register tiling, why does increasing the tile sizes v1 and v2 reduce the total number of DRAM loads, and what limits how large they can be?
4. What problem does adding an L1-cache tiling level on top of register tiling solve, and what constraint must the cache tile sizes satisfy relative to the register tile sizes?
5. How can you identify, from the indexing pattern of a loop nest like matrix multiplication, which loop dimensions are good candidates for tiling to increase data reuse?