Lecture 10: Recurrent Neural Networks

What this lecture covers

After a recap of CNN architectures, this lecture answers a different question: what happens when your input or output is not a fixed-size vector but a variable-length sequence? It introduces recurrent neural networks (RNNs) as a general way to handle sequences, then builds up to language modeling, image captioning with attention, and finally the vanishing and exploding gradient problems that motivate the LSTM architecture.

You'll come away able to describe an RNN's recurrence relation, explain how backpropagation through time works (including its truncated approximation for long sequences), and understand why plain RNNs struggle with long sequences while LSTMs handle them better. The lecture also shows several concrete applications: character-level text generation, image captioning, visual question answering, and models that combine CNNs with RNNs.

Key ideas

• Sequence flexibility: RNNs support one-to-many (image to caption), many-to-one (sequence to classification), and many-to-many setups (sequence to sequence, as in translation), unlike fixed-input feed-forward networks.
• Recurrence relation: at each step an RNN combines the previous hidden state and current input through the same function and weights, h_t = f_w(h_{t-1}, x_t), optionally producing an output y_t at each step.
• Vanilla RNN: computes the next hidden state as tanh(W_hh . h_{t-1} + W_xh . x_t), using the same weight matrix at every time step.
• Backpropagation through time: gradients for the shared weight matrix are summed across all time steps; for very long sequences this is approximated with truncated backpropagation through time, which processes fixed-length chunks and carries the hidden state forward between chunks.
• Character-level language modeling: an RNN trained to predict the next character in a sequence can, after training, be sampled one character at a time to generate new text resembling its training data.
• Image captioning: a CNN encodes the image into a vector that initializes an RNN's hidden state, which then generates the caption one word at a time until it samples an end token.
• Soft attention: instead of a single image summary vector, the model attends to different spatial locations of the image at each generated word, learned entirely through training rather than being manually specified.
• Vanishing and exploding gradients: repeatedly multiplying by the same weight matrix during backpropagation through a vanilla RNN causes gradients to explode or vanish depending on the matrix's largest singular value; gradient clipping addresses exploding gradients, but vanishing gradients require a different architecture.
• LSTM: maintains a separate cell state alongside the hidden state and uses input, forget, output and "gate" gates with element-wise (not matrix) interactions, giving gradients a more direct path backward through time, similar in spirit to ResNet's shortcut connections.

Walkthrough

Recap: CNN architectures and gradient flow (0:07)

The lecture opens with administrative notes, then reviews the previous lecture on AlexNet, VGG, GoogleNet and ResNet, adding the observation that VGG and GoogleNet needed training hacks because they predate batch normalization, while ResNet's shortcut connections give gradients a more direct backward path (a "gradient super highway") that also foreshadows the LSTM discussion later in this lecture.

Why recurrent networks, and the recurrence relation (8:09)

The lecture motivates RNNs by listing the kinds of variable-length problems they handle: one-to-many (image captioning), many-to-one (sentiment analysis, video classification), and many-to-many (machine translation, per-frame video labeling). It then defines the core recurrence: an RNN cell takes the previous hidden state and current input, applies the same function and weights at every time step, and produces a new hidden state, optionally with an output. The vanilla RNN implements this as tanh(W_hh h_{t-1} + W_xh x_t). Unrolling the computation across time steps clarifies that the same weight matrix is reused everywhere, and during backpropagation each time step contributes a separate gradient for that shared weight matrix, which are summed to form the final gradient.

Character-level language modeling and sampling (20:17)

Using a toy vocabulary of four letters spelling "hello," the lecture shows how a character-level RNN is trained: each character is one-hot encoded, fed in as x_t, and the model predicts a distribution over the next character, incurring softmax loss against the true next character. At test time, the trained model can be seeded with a starting character and made to generate new text by sampling from its predicted distribution at each step and feeding the sampled character back in as the next input. Truncated backpropagation through time is introduced here as the practical way to train on very long sequences (such as all of Wikipedia) without forward- and back-propagating through the entire sequence at once.

Generating text and interpreting hidden units (31:23)

The lecture shows a minimal (roughly 112-line) implementation, then demonstrates trained models sampling Shakespeare-like text, LaTeX resembling an algebraic topology textbook, and C-like Linux kernel source code, all learned purely from predicting the next character. A follow-up analysis looks inside a trained model's hidden state vector and finds individual units that appear to track meaningful structure, such as whether the model is inside a quoted string, how many characters since a line break, or whether it's inside an if statement condition, despite never being told to learn these concepts explicitly.

Image captioning and attention (38:29)

A CNN encodes an image into a summary vector that initializes the hidden state of an RNN language model, which then generates a caption word by word until it samples a special end token; the whole system is trained jointly and end to end on datasets like Microsoft COCO. Example outputs show both plausible captions and clear failure cases where the model mismatches unfamiliar objects to familiar training patterns. The lecture then introduces soft attention, where the CNN produces a grid of vectors (one per spatial location) instead of a single summary vector, and the RNN learns to shift attention across image regions as it generates each word, without being explicitly told where to look. The same encoder-decoder pattern is shown extended to visual question answering, where a question and an image are separately encoded and combined to predict an answer.

Multi-layer RNNs and the vanishing/exploding gradient problem (50:42)

Stacking RNN layers (typically two to four, rarely deeper) is introduced briefly, followed by a detailed derivation of why vanilla RNNs are hard to train over long sequences: backpropagating through each cell multiplies the gradient by (a portion of) the same weight matrix, so across many time steps the gradient is repeatedly multiplied by that matrix. If its largest singular value exceeds one, gradients explode; if it's below one, they vanish. Gradient clipping is presented as a practical, if inelegant, fix for exploding gradients, while vanishing gradients require an architectural change.

LSTM: a gradient super highway (55:43)

The LSTM, dating to 1997, maintains both a hidden state and a separate cell state, and computes four gates (input, forget, output, and a "gate gate," commonly abbreviated ifog) from the stacked previous hidden state and current input. The forget gate controls how much of the previous cell state to keep, the input and gate gates control how much new information to write, and the output gate controls how much of the cell state to expose as the hidden state. During backpropagation, the cell state's gradient path involves only element-wise multiplication by the (per-time-step, sigmoid-bounded) forget gate rather than repeated full matrix multiplication, giving gradients a much more direct path backward, analogous to ResNet's identity shortcuts. The lecture closes by noting related variants (GRU, highway networks) and research showing that many LSTM-like variants perform similarly, suggesting the key ingredient is managing gradient flow through additive and multiplicative gates rather than any single "magic" equation.

Before you watch

• Review the CNN architecture lecture, especially ResNet's residual connections, since the lecture draws a direct comparison to LSTM gradient flow.
• Be comfortable with backpropagation through computational graphs, including how gradients behave at addition and matrix multiplication nodes.
• Familiarity with softmax loss and one-hot encoding is assumed for the language modeling examples.

Check your understanding

1. What is the recurrence relation of a vanilla RNN, and what stays the same across every time step?
2. Why is truncated backpropagation through time needed for very long sequences, and what is carried forward between truncated batches?
3. In the image captioning model, what initializes the RNN's hidden state, and what signals the model to stop generating words?
4. Why does repeatedly multiplying by the same weight matrix during backpropagation in a vanilla RNN lead to exploding or vanishing gradients?
5. How does the LSTM's forget gate change the gradient path compared to a vanilla RNN, and why does this help with vanishing gradients?