Lecture 8: Deep Learning Software

What this lecture covers

After several lectures on training tricks, this one steps back to the hardware and software that make training practical. It answers a concrete question: why does deep learning run on GPUs rather than CPUs, and what do the major frameworks actually do for you when you use them.

You'll come away understanding why GPUs suit matrix multiplication and convolution, why libraries like cuDNN matter, and how TensorFlow and PyTorch differ in building computational graphs. The lecture builds a single running example, a two-layer fully connected network trained on random data, and reimplements it in Numpy, TensorFlow and PyTorch so you can compare them directly.

Key ideas

• CPU vs GPU: CPUs have a handful of powerful, independent cores; GPUs have thousands of simpler cores suited to running the same operation on many data points at once, which is why they excel at matrix multiplication and convolution.
• cuDNN and cuBLAS: NVIDIA's optimized libraries for common deep learning operations; writing raw CUDA yourself is rarely worth it, and skipping cuDNN can cost a large fraction of the available speedup.
• Data-loading bottleneck: if the GPU computes faster than data can be read from disk, training stalls; prefetching with background CPU threads (often handled by the framework) avoids this.
• Computational graph: frameworks let you build a graph of operations, compute gradients automatically, and run everything on GPU without managing memory transfers yourself.
• Static graphs (TensorFlow): you build the graph once, then run it repeatedly, feeding data in through placeholders; variables can persist inside the graph across runs.
• Dynamic graphs (PyTorch): a new graph is built on every forward pass, so ordinary Python control flow (loops, conditionals) works directly, at the cost of needing the original code to reuse the model later.
• Higher-level wrappers: raw computational graph code is verbose, so libraries such as Keras, tf.layers, and PyTorch's nn module handle weight initialization and layer composition for you.
• Framework choice: TensorFlow suits general-purpose and production use; PyTorch suits fast research iteration; Caffe and Caffe2 suit production and mobile deployment.

Walkthrough

CPUs and GPUs (10:05)

The lecture opens by comparing consumer CPUs and GPUs side by side. CPUs have a small number of fast, independent cores; a top-end GPU like the NVIDIA Titan XP has thousands of simpler cores that must work together on the same kind of operation. This makes GPUs well suited to matrix multiplication, where every output element is an independent dot product, and to convolution, which has the same structure. NVIDIA dominates deep learning hardware, and its cuBLAS and cuDNN libraries provide heavily optimized implementations of matrix operations and convolutions so you rarely need to write CUDA yourself. Benchmarks shown in the lecture found roughly a 65-75x speedup running the same network on a top GPU versus a CPU, and close to a 3x speedup from using cuDNN instead of naive hand-written CUDA.

CPU/GPU communication and data loading (16:51)

Because the model's weights live in GPU memory but the dataset usually sits on disk, reading data sequentially can bottleneck training even though the GPU itself is fast. Solutions include loading a small dataset entirely into RAM, using an SSD instead of a hard drive, and running background CPU threads that prefetch and buffer minibatches so the GPU is never left waiting.

Deep learning frameworks and computational graphs (22:05)

The lecture surveys the framework landscape: Caffe, Torch and Theano came from academia, while newer frameworks like TensorFlow (Google) and PyTorch and Caffe2 (Facebook) came from industry. Frameworks exist for three reasons: managing complex computational graphs, computing gradients automatically, and running efficiently on GPU. A simple example (C computed from inputs X, Y, Z) shows that Numpy can express the forward pass easily but requires you to hand-write the backward pass and cannot run on GPU.

Building and running graphs in TensorFlow (27:44)

Using the two-layer ReLU network as a running example, the lecture shows TensorFlow's two-stage pattern: first define placeholders and operations to build the graph, then open a session and call session.run repeatedly with concrete Numpy data. A first version feeds weights in as placeholders on every call, which is wasteful because it copies data between CPU and GPU each time. The fix is to declare weights as variables that persist inside the graph, initialized once, and updated with assign operations. A subtlety follows: TensorFlow only executes the parts of the graph needed for the requested output, so update operations must be explicitly grouped (or handled by an optimizer's minimize call) or they silently never run. The lecture also shows convenience layers such as tf.losses.mean_squared_error and tf.layers.dense, which set up weights and biases for you, and notes the many competing higher-level wrappers around TensorFlow (Keras, TF-Slim, tf.contrib.learn, Sonnet).

PyTorch tensors, variables and modules (50:50)

PyTorch is presented as three layers of abstraction: tensors (like Numpy arrays, but GPU-capable), variables (graph nodes that support automatic differentiation), and modules (composable neural network layers). The same two-layer network is rebuilt first with raw tensors and manual backprop, then with variables and loss.backward() for automatic gradients, then with the nn package for layers and optimizers, and finally as a custom nn.Module subclass. PyTorch's DataLoader handles minibatching and multithreaded data loading, and pretrained models are available through torchvision.models.

Static vs dynamic graphs (1:01:56)

This is framed as the key distinction between the two frameworks. Static graphs (TensorFlow) are built once and can potentially be optimized and serialized for deployment without the original code. Dynamic graphs (PyTorch) are rebuilt on every forward pass, which lets you use normal Python control flow for conditionals and loops instead of special graph operators like tf.cond or foldl. Dynamic graphs are shown to be a natural fit for recurrent networks with variable-length sequences, recursive networks over parse trees, and neural module networks for visual question answering.

Caffe and Caffe2 (1:12:03)

Caffe lets you train networks by editing prototxt configuration files and calling a binary, without writing Python, though these files can become huge for large models such as the 152-layer ResNet. Caffe2, from Facebook, uses static graphs similar to TensorFlow and lets you define the graph in Python before serializing it, aiming at production and mobile deployment. The lecture closes by summarizing the practical recommendation: TensorFlow as a general-purpose default, PyTorch for research iteration, and Caffe or Caffe2 for production and mobile deployment.

Before you watch

• Be comfortable with the computational graph and backpropagation ideas from the earlier optimization and backprop lectures in this course.
• Know what a matrix multiplication and a convolution compute, since the hardware discussion builds directly on those operations.
• Having written a two-layer network's forward and backward pass in Numpy (as in the course's first assignment) makes the framework comparisons much easier to follow.

Check your understanding

1. Why are GPUs especially well suited to matrix multiplication and convolution, while CPUs are not?
2. In the buggy TensorFlow example, why did the loss stay flat even after the assign operations were added to the graph?
3. What is the practical difference between a static computational graph and a dynamic one, and how does each affect handling control flow like loops or conditionals?
4. Why might a static graph be preferable when deploying a trained model to production?
5. Based on the lecture's closing advice, which framework would you choose for a fast-moving research project, and which for a mobile deployment?