Lecture 10: Introduction to Neural Networks

What this lecture covers

This lecture, taught by Kian Katanforoosh, introduces neural networks by first reframing logistic regression as a single "neuron" and then showing how stacking and connecting more neurons produces the multi-class, multi-layer networks used in deep learning. It works through a running cat-classification example, building up notation for layers, weights, biases and activations, and ends by previewing why training these networks (backpropagation) means differentiating from the output layer backward using the chain rule.

The lecture sits at the start of a two-part unit on neural networks (continued the following Monday) and follows the course's earlier linear-model material. After watching, you should be able to describe a neuron as a linear operation plus an activation, count the parameters in a small network, distinguish sigmoid, softmax and ReLU outputs for different prediction tasks, and explain in outline why backpropagation proceeds from the last layer toward the first.

Key ideas

• Neuron = linear + activation: every neuron computes a linear function Wx + b and passes the result through an activation function such as sigmoid or ReLU.
• Model = architecture + parameters: an architecture (the graph of neurons and layers) combined with trained weights and biases fully specifies a model.
• Logistic regression is a one-neuron network: flattening an image into a vector and applying sigmoid(Wx + b) is exactly logistic regression, just relabeled with neural-network vocabulary.
• Independent output neurons vs. softmax: separate sigmoid neurons can detect co-occurring classes (a cat and a lion in the same image) because they don't communicate, while a softmax layer forces outputs to sum to 1 and is used when classes are mutually exclusive.
• Hidden layers build complexity: layers between input and output are called hidden because they see only the previous layer's output, not the raw input or the true label; in image models earlier layers tend to detect simple features like edges, and later layers combine them into more complex concepts.
• Fully-connected layers let the network choose features: rather than hand-engineering which inputs matter, connecting every neuron in one layer to every neuron in the next lets training discover useful intermediate representations, at the cost of interpretability ("black box" models, end-to-end learning).
• Batching requires broadcasting: vectorizing computation across a batch of m examples means the bias vector must be broadcast (repeated) across columns without actually increasing the number of parameters.
• Backpropagation moves backward through the chain rule: because the loss depends on later layers most directly, derivatives are computed starting at the output layer and reused via the chain rule when computing derivatives for earlier layers.

Walkthrough

Deep learning intro and logistic regression as one neuron (1:02)

After a short overview of why deep learning became practical (faster compute including GPUs, larger datasets, and better algorithms), the lecture sets a concrete goal: detect whether a 64x64 color image contains a cat. Flattening the image into a 12,288-dimensional vector and computing y_hat = sigmoid(Wx + b) reproduces logistic regression exactly, just with the parameters split into a weight vector W and bias b instead of a single Theta. The lecture counts parameters (one weight per input pixel plus one bias) and introduces the vocabulary "neuron" (linear part plus activation) and "model = architecture + parameters" that carries through the rest of the lecture.

From one neuron to independent multi-label neurons (13:42)

Extending the goal to detecting cats, lions and iguanas independently, the lecture adds two more neurons, each connected to all the same inputs but with its own weights and bias, using layer notation in square brackets and neuron-index subscripts. Because the three neurons don't communicate, the network can correctly handle images containing more than one animal, as long as the training labels reflect co-occurrence (for example, [1, 1, 0] for an image with both a cat and a lion). This independence is why the setup is really three separate logistic regressions trained together rather than a true interacting neural network.

Softmax regression for mutually exclusive classes (29:40)

When at most one animal can appear in an image, the lecture replaces the three independent sigmoid outputs with a softmax layer, where each output is exp(Z_k) / sum(exp(Z_j)) so the outputs necessarily sum to 1. This requires a different loss (summing the logistic-style loss across all classes, related to what is later called softmax cross-entropy) and only works correctly when labels are one-hot vectors that also sum to 1. The lecture contrasts this with the independent-neuron case and notes that softmax should be used whenever labeling and problem structure guarantee mutual exclusivity, such as predicting a single discrete age category.

Regression outputs and the ReLU activation (39:32)

To predict a continuous quantity like a cat's age directly rather than a class, the sigmoid output must be replaced, since sigmoid restricts outputs to (0, 1). Using a purely linear output would collapse the network into linear regression, so the lecture introduces the rectified linear unit (ReLU), which behaves like a linear function for positive inputs and outputs 0 for negative inputs, matching the fact that predicted ages can't be negative. The loss function must also change to a regression loss such as squared error, since classification losses are shaped differently and harder to optimize for continuous targets.

Building deeper networks with hidden layers (45:30)

The lecture assembles a full three-layer network (input, hidden, output) and works through parameter counts and matrix shapes for each layer, emphasizing that shape bookkeeping matters both for understanding and for implementation. It defines input, hidden and output layers, explains why hidden layers are "hidden" (they see only the prior layer's output, not the raw input or the true label), and illustrates with a house-price example how fully-connected layers let the network discover useful intermediate features (like implicit "school quality" or "family size" signals) instead of requiring them to be hand-engineered. Vectorizing across a batch of m examples is shown to require broadcasting the bias term, a detail handled automatically by libraries such as NumPy.

Toward backward propagation (1:12:42)

Having defined the cost function as an average of per-example losses, the lecture turns to optimization: computing the derivative of the cost with respect to every weight and bias so gradient descent can update them. It explains why this process is called backward propagation: the relationship between the loss and the last layer's weights is the most direct, so the derivative there is computed first, and the chain rule lets that result be reused (multiplied by a few additional easy terms) when computing derivatives for earlier layers, rather than recomputing everything from scratch. The full derivation is left for the following lecture.

Before you watch

• Review logistic regression, the sigmoid function, and the logistic loss (negative log-likelihood), since the lecture builds neural networks directly on top of this notation.
• Be comfortable with basic matrix and vector shapes, since much of the lecture is spent tracking dimensions through layers.
• Recall the difference between stochastic and batch gradient descent, referenced when discussing vectorized, batched computation.

Check your understanding

1. How does writing logistic regression as sigmoid(Wx + b) connect it to the definition of a "neuron" used later in the lecture?
2. Why can three independent sigmoid output neurons correctly handle images containing multiple animals, while a softmax layer cannot?
3. When should you use a softmax output instead of independent sigmoid outputs, and why does the choice depend on how you label your data?
4. Why is ReLU used instead of sigmoid or a purely linear function when predicting a continuous value like age?
5. Why does backpropagation compute derivatives starting from the output layer and move backward, reusing earlier results via the chain rule?