Lecture 6: LSTMs and Neural Machine Translation

What this lecture covers

This lecture asks why simple recurrent neural networks struggle to use information from many steps back, and how a redesigned recurrent unit, the LSTM, fixes that. It continues directly from the previous lecture's introduction of RNNs and language models, first closing out how language models are evaluated with perplexity, then diagnosing vanishing and exploding gradients as the core weakness of plain RNNs.

The second half introduces the long short-term memory (LSTM) architecture and its gating mechanism, surveys other uses of RNNs such as bidirectional and stacked models, and then applies sequence-to-sequence RNNs to machine translation. After watching, you should be able to explain why RNN gradients vanish or explode, describe how an LSTM's gates control what it remembers, and outline how an encoder-decoder network translates a sentence from one language to another.

Key ideas

• Perplexity: the standard evaluation metric for language models, defined as the exponential of the average cross-entropy loss; lower perplexity means better next-word prediction.
• Vanishing gradients: repeatedly multiplying by the same weight matrix during backpropagation through time shrinks gradients from distant time steps, so plain RNNs mostly learn from nearby context only.
• Exploding gradients: the same repeated multiplication can also blow gradients up, producing unstable updates; the standard fix is gradient clipping, which rescales a gradient when its norm exceeds a threshold.
• LSTM (long short-term memory): an RNN variant with a separate cell state and three gates (forget, input, output) that control what is remembered, added, and exposed at each step, making it much easier to preserve information over long sequences.
• Additive updates: the LSTM's cell state is updated by addition rather than full multiplication, which is why gradients flow through it much better than through a plain RNN.
• Bidirectional RNN: runs one RNN forward and one backward over a sequence and concatenates their hidden states, giving each position a representation that reflects both preceding and following context; useful for encoding, not generation.
• Encoder-decoder (sequence-to-sequence) model: one RNN encodes a source sentence into a hidden state, and a second RNN, conditioned on that state, generates the target sequence word by word.
• Neural machine translation (NMT): an end-to-end trained encoder-decoder system that replaced older statistical phrase-based translation systems around 2014-2016 because of a large jump in translation quality.

Walkthrough

Evaluating language models with perplexity (2:07)

The lecture finishes the previous class's topic by explaining how language models are scored rigorously rather than just by eyeballing generated text. A model is shown fresh, human-written text and asked to predict each word; perplexity is the exponential of the average cross-entropy (negative log-likelihood) of those predictions, so a perplexity of 64 is roughly like guessing among 64 equally likely options. The lecture traces perplexity's origin to early IBM speech recognition work and shows how scores fell over decades: n-gram models with smoothing reached perplexities around 67, early RNNs combined with other models did somewhat better, and LSTMs pushed perplexity down into the 30s and below, though modern large language models reach single digits.

Vanishing and exploding gradients (9:16)

Backpropagating a loss through an RNN requires repeatedly multiplying by the same hidden-to-hidden weight matrix at each time step. If the matrix's dominant eigenvalues are less than one, this product shrinks toward zero as it moves backward, so signal from distant time steps barely affects the parameters, a problem called vanishing gradients. If the eigenvalues are greater than one, the product grows explosively instead. The lecture uses an example sentence where the crucial earlier word ("tickets") is about twenty words back, showing that a plain RNN can fail to learn such long-distance dependencies because the gradient from that position never arrives with enough strength; in practice, simple RNNs were found to effectively condition on only about seven prior tokens. Exploding gradients are addressed with gradient clipping, a simple technique that rescales the gradient whenever its norm exceeds a chosen threshold.

LSTMs: gated memory cells (20:25)

To fix vanishing gradients, the lecture introduces the LSTM, proposed by Hochreiter and Schmidhuber in 1997, with a crucial forget gate added by Gers and Schmidhuber in 2000. An LSTM keeps two vectors: a hidden state and a cell state, and computes three gate vectors (values between 0 and 1) at each step: a forget gate controlling how much of the previous cell content to keep, an input gate controlling how much new information to add, and an output gate controlling how much of the cell is exposed to the hidden state. The cell is updated additively, new cell = forget_gate * old_cell + input_gate * candidate_update, and this additive structure, rather than the pure matrix multiplication used by plain RNNs, is what allows gradients to flow across many time steps without vanishing. The lecture also connects this idea to residual and highway networks in deep learning generally, which use similar additive "skip connection" tricks to fix the same gradient problem in very deep networks.

Other uses of RNNs: bidirectional and stacked models (47:54)

RNNs and LSTMs generalize beyond language modeling to tasks like part-of-speech tagging, named entity recognition, and sentence classification, where the final or averaged hidden states serve as a representation of the whole input. Because a standard RNN's hidden state at a position only reflects earlier context, bidirectional RNNs run a forward and a backward RNN and concatenate their states so each position's representation reflects the whole sentence; this works well for encoding but not for generating new text. RNNs can also be stacked into multiple layers for extra representational power, though the lecture notes that in the RNN era, gains typically leveled off after two or three layers, unlike the much deeper networks used with Transformers.

From statistical to neural machine translation (54:59)

Machine translation is described as one of the original goals of computing, dating to Cold War-era code-breaking-inspired research in the 1950s that failed due to limited computing power and poor understanding of language structure. It revived in the 1990s-2000s as statistical machine translation, which split translation probability via Bayes' rule into a simple word-translation model and a separate language model, and powered early systems like Google Translate. The lecture illustrates its weaknesses with a Chinese-to-English example where the system badly mishandles a modifier relationship between clauses, showing that word-order and structural differences between languages were hard for these systems to capture even after a decade of engineering effort.

Neural machine translation with sequence-to-sequence models (1:05:12)

Neural machine translation (NMT), introduced around 2014, trains a single end-to-end network instead of separate translation and language models. An encoder LSTM reads the source sentence and builds a final hidden state summarizing it; a decoder LSTM, initialized from that hidden state, then generates the target sentence one word at a time, feeding each generated word back in as the next input, exactly as in earlier text-generation examples. Training uses parallel sentence pairs, computing the loss at each decoder position and backpropagating through the entire encoder-decoder system at once. This encoder-decoder pattern is described as a general template reused for tasks like summarization and speech recognition. NMT quality was so much better than statistical systems that within about two years it replaced them at Google and across the industry, a shift the lecture frames as one of deep learning's first major NLP successes.

Before you watch

• Watch the prior lecture on language models and recurrent neural networks, since this lecture builds directly on the RNN architecture and training method it introduced.
• Be comfortable with the chain rule and how gradients are computed through repeated matrix multiplications, since the vanishing gradient explanation relies on it.
• Know what cross-entropy loss measures, since perplexity is defined directly from it.

Check your understanding

1. Why does repeatedly multiplying by the same weight matrix during backpropagation through time cause gradients to vanish or explode, and what fixes each problem?
2. How do an LSTM's forget, input, and output gates work together to update the cell state and hidden state at each time step?
3. Why does the additive update in an LSTM cell help gradients flow better than the multiplicative update in a plain RNN?
4. What is the difference between a bidirectional RNN and a standard RNN, and why can't a bidirectional RNN be used to generate new text?
5. How does a sequence-to-sequence encoder-decoder model translate a sentence, and why did neural machine translation outperform statistical phrase-based systems?