Lecture 13: Hardware Acceleration Implementation

What this lecture covers

This is a live-coding lecture that puts the previous two lectures on CPU and GPU acceleration into practice by walking through needle's new NDArray backend, the library built for the course's third homework to replace NumPy as the array backend. The instructor works in a Google Colab notebook with GPU support enabled, builds the CUDA extension with pybind11, and traces through the code path executed when creating an array and running simple arithmetic on it.

The lecture explains the NDArray class's fields (shape, strides, offset, device, handle), how the same one-dimensional memory buffer can represent reshaped, sliced, transposed, or broadcast views by changing only shape and strides, and why a compact operation is needed before certain arithmetic operations. It ends by adding a new elementwise/scalar division operator to the library as a worked example. After watching, you should be able to explain how NDArray maps multi-dimensional array operations onto a flat memory buffer, and why the same view-based logic underlies reshape, slicing, transpose, and broadcasting.

Key ideas

• NDArray fields: an NDArray is defined by shape, strides, offset, device, and a handle pointing to a flat one-dimensional memory buffer allocated on the CPU or GPU.
• Strided indexing maps multi-dimensional access to flat memory: an element [i,j] is located at offset + i*strides[0] + j*strides[1], generalizing across CPU and GPU backends.
• Views share memory without copying: reshape, slicing, and transpose can all be implemented by constructing a new NDArray with different shape/strides/offset over the same underlying handle, rather than copying data.
• Broadcasting via zero strides: broadcasting a new dimension is implemented by giving that dimension a stride of 0, so indexing along it always reads the same underlying value.
• Compact vs. non-compact arrays: an array is "compact" when its strides follow normal row-major order with zero offset; non-compact arrays (after slicing, transposing, or broadcasting) generally cannot be operated on by directly iterating the flat buffer, so a compact operation copies them into normal layout first when needed.
• Backend dispatch: array creation and arithmetic operations dispatch through a BackendDevice object to either a NumPy, CPU (pybind11-wrapped C++), or CUDA implementation, keeping needle's array API consistent across devices.
• Adding an operator requires kernel plus host-side function: implementing elementwise or scalar division follows the same pattern as existing operators (add, multiply): define a CUDA kernel, a host-side launcher function, and register it, mirroring the existing code for other operators.
• Compositional design: needle's tensor and autodiff layer is built on top of the NDArray array API, so the array backend can be developed and tested independently before being used to power full automatic differentiation.

Walkthrough

Setting up the environment and repository (0:00)

The instructor sets up a Colab notebook with a GPU runtime (an NVIDIA T4 in this session), clones the lecture's version of needle, and installs pybind11 to build the new CUDA extension. The new code base adds a src folder with CUDA and C++ source files and a backend_ndarray folder that implements the new array backend intended to replace NumPy.

Creating an NDArray and tracing array creation (4:05)

After importing backend_ndarray, the instructor creates an array on the CUDA device and traces the code path: NDArray.make calls into the device's array function to allocate a flat buffer of the requested size (through the CPU, CUDA, or NumPy backend module), then from_numpy copies data from a source NumPy array into that allocated buffer. On CPU this uses an aligned allocation and a memory copy; on GPU it uses cudaMalloc and cudaMemcpy from host to device.

Tracing arithmetic: y = x + 1 (16:19)

Adding a scalar to an NDArray dispatches through an overloaded operator to a scalar_add function, which allocates a new output array via make and then calls the backend's scalar-add kernel, passing the input handle, the scalar value, and the output array view. On GPU, this compiles down to a CUDA kernel launched with a computed grid and block size; on CPU, the equivalent function is a straightforward loop over the array's elements.

Shape, strides, and offset: reshape, slicing, transpose, broadcasting (25:44)

The instructor demonstrates that reshaping a 1D array into a 2D array, slicing out a submatrix, transposing an array, and broadcasting a new dimension can all be implemented by constructing a new NDArray with different shape, strides, and offset values over the same underlying handle — no data is copied. Transposition swaps the corresponding strides; slicing adjusts the offset and shape; broadcasting sets the new dimension's stride to 0 so every index along it reads the same value.

Why compaction is needed (33:52)

Because arithmetic kernels generally iterate directly over the flat underlying buffer, an array whose strides and offset don't correspond to plain row-major order (for example, after transposing or slicing) cannot be operated on directly without producing wrong results. The compact function checks whether an array's layout is already in standard order and, if not, allocates a new buffer and copies the data into standard layout. The lecture recommends always calling compact before arithmetic to keep the homework implementation simple, while noting that production libraries sometimes implement operations that work directly on non-compact, strided arrays for efficiency.

Live-coding a new operator: scalar and elementwise division (40:58)

As a worked example, the instructor adds elementwise and scalar division to the CUDA backend by copying the structure of the existing multiply kernel and its host-side launcher function, changing the operation and function names accordingly, then rebuilding the extension. After hitting an import-caching issue in the notebook (fixed by running a separate Python script rather than reusing the notebook's already-imported module) and one bug in the first attempt, the division kernel is shown working correctly.

Connecting the array backend to needle's tensor layer (47:07)

The lecture closes by showing that needle's tensor and operator implementation files now import backend_ndarray as the array API instead of NumPy, so the same array logic developed and tested independently in this lecture now powers needle's automatic differentiation and neural network layers, completing the transition away from NumPy as the backend.

Before you watch

• Watch the two previous lectures on CPU acceleration and GPU acceleration in this course, since this lecture directly implements the tiling and CUDA kernel concepts they introduced.
• Basic familiarity with C++ and CUDA syntax will help with following the kernel code shown on screen.
• This lecture assumes access to the course's needle repository and a CUDA-enabled environment (such as Google Colab), which the instructor sets up at the start.

Check your understanding

1. What fields make up an NDArray, and how does an element's flat memory location get computed from its shape, strides, and offset?
2. Why can reshaping, slicing, transposing, and broadcasting all be implemented without copying any data?
3. What makes an array "non-compact," and why does that matter for arithmetic kernels that iterate over the flat buffer?
4. Walk through what happens, function by function, when you write y = x + 1 on a CUDA-backed NDArray.
5. Why does needle's tensor and automatic differentiation layer import the array API from backend_ndarray instead of NumPy after this lecture's changes?