Lecture 3 (Part I): Manual Neural Networks

What this lecture covers

This lecture explains why softmax regression's linear hypothesis class is limited, and how neural networks generalize it by learning a non-linear feature extractor jointly with a final linear classifier. It's the first of a two-part lecture, still done "the old fashioned way" by hand, before the course moves to automatic differentiation.

You'll see why manually engineered features (like random Fourier features) work but don't scale as a general approach, how a two-layer neural network emerges naturally from making the feature-extraction weights learnable, and a constructive proof that a one-hidden-layer ReLU network can approximate any 1D function arbitrarily well. The lecture ends by introducing general L-layer networks (multi-layer perceptrons) and surveying, critically, the common arguments for why deep networks are used in practice. Part II covers backpropagation, the gradient computation these networks need.

Key ideas

• Linear hypothesis classes are limited: a linear classifier can only carve input space into linear regions, so data with non-linear class boundaries (like concentric rings) cannot be separated by theta^T x alone.
• Feature mapping: classical machine learning applies a hand-designed function phi(x) to inputs before a linear classifier, e.g. theta^T phi(x); this is powerful but requires manual feature engineering.
• Learnable features are what makes a neural network: replacing a fixed feature map with sigma(W^T x), where W is also trained, turns feature extraction into something optimized jointly with the classifier.
• Nonlinearity is essential: composing two linear functions is still linear (no representational gain); the nonlinear function sigma (ReLU, sigmoid, tanh, etc.) is what prevents this collapse.
• Two-layer network: h_theta(x) = W_2^T sigma(W_1^T x), where theta now denotes the full set of parameters {W_1, W_2}.
• Universal function approximation: a two-layer (one hidden layer) network can approximate any continuous function on a closed region arbitrarily well, though this requires the hidden layer to grow with the number of sample points and is not a practical training method by itself.
• Multi-layer perceptron (MLP): an L-layer network defined recursively as Z_1 = X, Z_(i+1) = sigma(Z_i W_i), with the final output being Z_(L+1).
• Why use deep (multi-layer) networks: the lecture reviews and critiques common justifications (brain analogy, circuit-theory efficiency arguments like the parity function) and lands on the empirical argument: for a fixed parameter budget, deeper structured networks tend to perform better in practice.

Walkthrough

Why linear hypothesis classes fail (2:32)

The lecture recalls the linear hypothesis class from softmax regression and shows visually why it can only separate data using straight-line decision boundaries. It presents a dataset with concentric class regions that no linear classifier can correctly separate, motivating the need for non-linear hypothesis classes.

Feature mappings, old and new (9:31)

The lecture introduces manual feature engineering: applying a hand-chosen function phi(x) before a linear classifier, illustrated with a feature like x_1^2 + x_2^2 that makes concentric classes linearly separable. It notes this was, and still is, a major part of applied machine learning, then pivots to the question of learning such features automatically instead.

From random features to learnable weights (12:37)

The lecture shows that composing two linear functions is pointless (it collapses to another linear function), but inserting a nonlinearity sigma between two linear transforms avoids this collapse and yields much richer hypothesis classes. It mentions random Fourier features (random W, cosine nonlinearity) as one example, then argues that making W itself trainable, rather than fixed, is the natural next step and is exactly what neural networks do.

Two-layer neural networks and universal approximation (22:45)

The lecture formally defines a two-layer network and clarifies that "neural network" and "deep learning" are just names for hypothesis classes built from composed differentiable functions, without much remaining connection to biological brains. It then works through a constructive proof, in 1D, that a one-hidden-layer ReLU network can represent a piecewise-linear approximation to any continuous function by adding one ReLU term per sample point, demonstrating universal function approximation while noting this isn't a practical training method.

General L-layer networks and why depth is used (42:26)

The lecture generalizes to L-layer networks (multi-layer perceptrons), defining each layer recursively as Z_(i+1) = sigma(Z_i W_i). It closes by examining, and largely rejecting, common justifications for depth (brain inspiration, circuit-complexity arguments like the parity function) in favor of the empirical observation that deeper, structured networks tend to work better at a given parameter budget.

Before you watch

• Watch Lectures 1 and 2 first; this lecture builds directly on the softmax regression hypothesis class, loss function, and matrix batch notation from Lecture 2.
• Be comfortable with matrix multiplication and function composition.
• This is Part I of a two-part lecture; Part II covers backpropagation and gradient computation for these networks.

Check your understanding

1. Why does composing two linear functions fail to create a more expressive hypothesis class, and what fixes this?
2. What is the difference between a manually engineered feature map and the feature extraction performed by a neural network?
3. In the universal approximation proof, why does the size of the hidden layer need to grow with the number of sample points?
4. What is the parity function example meant to illustrate about depth, and why does the lecture consider it a weak argument in practice?
5. According to the lecture, what is the most realistic justification for using deep (multi-layer) networks?