Lecture 26: Structure of Neural Nets for Deep Learning

What this lecture covers

This lecture builds up the structure of a deep neural network as a learning function f(x): a composition of affine (linear plus offset) layers and a nonlinear activation, trained so that f gets most training examples classified correctly. It follows the previous guest lecture on stochastic gradient descent and sets up backpropagation, which is covered next.

After watching, you can describe how a neural network's layers combine matrix multiplication with a pointwise nonlinearity, explain why the resulting function is continuous and piecewise linear, and reason about how the number of "flat pieces" that function can produce grows with the number of folds and the input dimension.

Key ideas

• Learning function: the network builds a function f(x) from training data so that f(x) is negative for one class and positive for the other, without necessarily fitting every training point exactly (to avoid overfitting).
• ReLU activation: the most commonly used nonlinearity is ReLU(x) = max(0, x), applied component-wise after each linear layer; without a nonlinearity the whole network would collapse into one linear classifier.
• Affine layer: each layer computes y = Ax + b, a linear map plus an offset vector, using a weight matrix A and bias b.
• Composition of layers: the full network is f(x) = f3(f2(f1(x))), chaining several affine-plus-ReLU layers; the chain rule (used later for backpropagation) differentiates this kind of composition.
• Continuous piecewise linear function: because affine maps and ReLU are each continuous and piecewise linear, their composition f is also continuous and piecewise linear.
• Epoch and mini-batch: an epoch is one full pass through the training data; stochastic gradient descent processes it in randomized mini-batches rather than one point at a time.
• Fold-counting argument: viewing ReLU layers as folds of a flat surface, the number of flat regions the function can produce follows a recursion involving binomial coefficients, giving intuition for how expressive a network with many layers and neurons can be.
• Universality: with enough folds (layers and neurons), this class of functions can approximate any continuous function, which is why deep nets are so flexible.

Walkthrough

Setting up the classification problem (0:22)

The lecture frames the task as binary classification: given feature vectors and known labels, construct a function f(x) that is negative for one class and positive for the other on almost all training data, while avoiding overfitting to unusual examples.

Training data and the TensorFlow Playground demo (3:03)

Using the TensorFlow Playground tool as an example, the lecture shows two-dimensional data (blue and orange points) that must be separated by a learned decision boundary. It explains why a nonlinear activation such as ReLU is required, since a purely linear classifier cannot separate certain shapes (for example, a ring or a spiral of one class around another). It also reviews mini-batches, replacement versus non-replacement sampling, and what an epoch is.

Building the layers of the network (7:15)

Starting from a feature vector, the lecture constructs a first layer as a matrix multiplication plus a bias vector, producing an intermediate output, then applies ReLU component-wise to get the next layer's input. It counts the weights involved (for example, a 6-by-5 weight matrix plus 6 biases) and generalizes to a network with several such layers, noting that in practice the number of weights can reach the tens of thousands.

The learning function as a composition (10:35)

The lecture writes the full network as a composition of functions, f(x) = f3(f2(f1(x))), where each f_i applies an affine map followed by ReLU. It identifies the resulting function as continuous piecewise linear, and previews that computing its gradient (needed for training) will require the chain rule, which is the subject of backpropagation.

Counting flat pieces by folding (19:05)

Using a two-dimensional analogy, the lecture visualizes each ReLU layer as folding a flat surface along a line, and asks how many flat pieces result after several folds. Working through examples with two, three, and four folds, it builds up a recursion for the number of pieces.

The recursion formula and binomial counts (28:25)

Referencing a paper by Kleinberg and others, the lecture presents a formula for the number of flat pieces R(n, m) in terms of binomial coefficients, for n folds in m dimensions, and verifies it against the earlier hand-counted examples. It also introduces a recursion, R(n, m) = R(n-1, m) + R(n-1, m-1), connecting successive fold counts.

Setting up the next step: the gradient (43:30)

The lecture closes by framing the remaining problem: choosing the weight matrices and bias vectors to minimize a loss over the training data requires a gradient descent method, and computing that gradient via the chain rule (backpropagation) is left for the next lecture.

Before you watch

• Review stochastic gradient descent and the finite-sum optimization setup from the previous guest lecture in this course.
• Be comfortable with matrix-vector multiplication and basic function composition.
• A working familiarity with binomial coefficients helps follow the fold-counting section.

Check your understanding

1. Why does a neural network need a nonlinear activation function like ReLU, rather than only affine layers?
2. Why is the function computed by a ReLU network continuous and piecewise linear?
3. How does viewing each ReLU layer as a "fold" help explain the recursion for counting flat pieces?
4. What does the universality idea say about how well these piecewise linear functions can approximate arbitrary continuous functions?
5. Why does training the network require computing a gradient, and what method does the lecture say will be used to compute it?