Lecture 18: Sequence Modeling and Recurrent Networks

What this lecture covers

This lecture is the algorithmic half of a two-part unit on sequence modeling (the next lecture covers implementation). It explains why problems like part-of-speech tagging, speech-to-text and language modeling break the independent-and-identically-distributed assumption used everywhere else in the course, then introduces recurrent neural networks (RNNs) as one way to model temporal dependence, works through why plain RNNs are hard to train, and introduces the LSTM as a fix.

After watching, you should be able to write the hidden-state update equations for a vanilla RNN, explain why training deep RNNs causes exploding or vanishing activations and gradients, describe what an LSTM's cell state and gates do differently, and describe sequence-to-sequence and bidirectional RNNs as ways to extend the basic model.

Key ideas

• Sequence dependence: many tasks (part-of-speech tagging, speech-to-text, autoregressive language modeling) have input-output pairs that are ordered in time rather than independent and identically distributed, so temporal context is necessary for correct predictions.
• Autoregressive prediction: a special case of sequence prediction where the target at each step is the next element of the input sequence itself, foundational to language models.
• RNN hidden state: a hidden state h_t is computed as a nonlinear function of the previous hidden state and the current input (h_t = f(W_hh h_{t-1} + W_hx x_t + b_h)), and in principle can summarize all inputs seen so far.
• Training via backpropagation through time (BPTT): rather than deriving RNN gradients by hand, the loop is unrolled and automatic differentiation handles the chain rule through the whole sequence.
• Exploding and vanishing gradients: because RNNs are effectively as deep as they are long, weight initializations that are too large cause activations and gradients to blow up, and initializations that are too small cause them to vanish, mirroring problems seen in very deep MLPs.
• Saturating activations don't fully fix it: bounded nonlinearities like sigmoid or tanh prevent explosion but still cause gradients to vanish, either by stagnating near zero or saturating near the bounds.
• LSTM: splits the hidden state into a hidden state and a cell state, and uses gate terms (input, forget, output, and a cell candidate) to control how much of the previous cell state is kept versus overwritten, which helps preserve long-range dependencies.
• Sequence-to-sequence and bidirectional RNNs: sequence-to-sequence models chain an encoder RNN's final hidden state into a decoder RNN for tasks like translation; bidirectional RNNs stack a forward and backward RNN so predictions can depend on the whole sequence, not just the past.

Walkthrough

Sequence modeling versus IID prediction (2:03)

The lecture contrasts the course's earlier assumption that training examples are independent and identically distributed with tasks where input-output pairs form an ordered sequence, illustrated with part-of-speech tagging, speech-to-text, and autoregressive language prediction, and introduces the notational switch from superscripts (example index) to subscripts (time index).

The recurrent neural network model (11:23)

An RNN is defined by a hidden-state update h_t = f(W_hh h_{t-1} + W_hx x_t + b_h) and an output equation y_t = g(W_yh h_t + b_y), with the hidden state acting as a running summary of everything seen so far in the sequence. The lecture notes RNNs can be stacked into deeper networks without violating the sequence's temporal dependencies.

Training with backpropagation through time (18:51)

Rather than deriving gradients for the RNN by hand, the lecture shows a training loop that initializes the hidden state to zero, unrolls the sequence forward computing hidden states, outputs, and loss at each step, then calls a single loss.backward() to get all gradients via automatic differentiation. This unrolled-training procedure is named backpropagation through time (BPTT).

Exploding and vanishing gradients (27:17)

The lecture connects RNN training instability to the same weight-initialization sensitivity seen in very deep MLPs, but notes RNNs are naturally as deep as their sequence length. Using ReLU activations with W_hh initialized too large causes hidden-state norms to blow up over time (exploding gradients); initializing too small causes them to shrink toward zero (vanishing gradients), and switching to bounded activations like sigmoid or tanh only prevents the explosion, not the vanishing, since gradients still go to zero in the saturated or flat regions of those functions.

LSTMs (38:28)

The LSTM is introduced as a fix that splits the hidden state into a hidden state and a cell state, computing four gate terms (input, forget, cell candidate, output) from one combined weight matrix applied to the previous hidden state and current input. The key update, c_t = c_{t-1} * f_t + i_t * g_t, lets the network preserve or discard the previous cell state via the forget gate, which the lecture argues is the main mechanism that helps mitigate vanishing gradients, while noting the specific gate names are somewhat arbitrary. The lecture also recounts the LSTM's 1997 origin paper and its rise in popularity following Andrej Karpathy's 2015 blog post on character-level RNNs.

Beyond simple sequence prediction (1:00:09)

The lecture closes by describing two extensions: sequence-to-sequence models, which feed an encoder RNN's final hidden state into a decoder RNN to handle tasks like translation where input and output lengths differ, and bidirectional RNNs, which stack a forward-running and backward-running RNN so a prediction at any time step can depend on the entire sequence rather than only the past.

Before you watch

• Review the course's lectures on normalization, regularization, and weight initialization, since this lecture directly reuses the deep-MLP explanation of exploding and vanishing activations.
• Be comfortable with automatic differentiation and computational graphs, as BPTT is presented as ordinary backpropagation over an unrolled graph.

Check your understanding

1. Why does the lecture switch from superscript example indices to subscript time indices for this topic?
2. Why does training a deep RNN reproduce the same exploding/vanishing activation problems seen in very deep MLPs?
3. Why don't bounded activations like sigmoid or tanh fully solve the vanishing gradient problem on their own?
4. What does the forget gate in an LSTM's cell-state update actually control, and why does that help preserve long-range dependencies?
5. How does a sequence-to-sequence model get around the constraint that a vanilla RNN's output at time t only depends on inputs up to time t?