Lecture 8: Self-Attention and the Transformer

What this lecture covers

This lecture asks why recurrent neural networks, even with attention bolted on, were replaced almost entirely by a new architecture built purely on attention: the Transformer. It starts by diagnosing two specific weaknesses of RNNs, long-range dependency learning and parallelization, then builds up self-attention as a replacement from first principles, fixing each gap it has (no sense of order, no nonlinearity, no way to prevent looking at the future) one at a time.

The second half assembles these pieces into the full Transformer decoder, adds multi-head attention, residual connections, and layer normalization, and then covers the encoder and encoder-decoder variants along with the original machine translation results. After watching, you should be able to explain why self-attention parallelizes better than recurrence, compute what a single self-attention operation does, and describe the role of each component in a Transformer block.

Key ideas

• Linear interaction distance: in an RNN, two words separated by many positions must interact through many sequential steps, making long-range dependencies hard to learn even with LSTMs.
• Non-parallelizability: an RNN's hidden state at each time step depends on the previous one, forcing O(sequence length) sequential operations that cannot be parallelized on a GPU.
• Self-attention as fuzzy lookup: each word produces a query, and attention softly matches that query against every word's key to produce a weighted average of value vectors, letting any two words interact directly regardless of distance.
• Query, key, value matrices: three learned weight matrices transform each word's embedding into a query, key, and value vector; attention scores come from dot products between queries and keys.
• Position representations: because self-attention has no inherent notion of order, position information (sinusoidal or learned) is added to word embeddings so the model can distinguish word order.
• Masking: for decoders and language modeling, attention scores to future positions are set to negative infinity before the softmax, so a word's representation cannot use information from words that come after it.
• Multi-head attention: running several smaller attention operations in parallel, each with its own query, key, and value matrices, lets the model attend to different kinds of relationships at once.
• Residual connections and layer normalization: adding a layer's input back to its output helps gradients flow during training, and layer normalization rescales each vector's values to stabilize training further.

Walkthrough

Why RNNs limit long-range learning and parallelization (4:08)

The lecture identifies two structural problems with RNNs. First, linear interaction distance: because RNNs process words one step at a time, two related words that are far apart in a sentence must interact through many applications of the recurrent weight matrix, making such dependencies hard to learn even with LSTMs' improved gradient flow. Second, non-parallelizability: computing the hidden state at time step 5 requires first computing steps 1 through 4, so the number of unparallelizable operations grows with sequence length, which wastes the parallel computation that GPUs are good at. Attention, previously used only to connect a decoder to an encoder, is introduced as a way to solve both problems by letting any word interact with any other word in a single, parallelizable step.

Self-attention as a fuzzy lookup (10:11)

Attention is described as a soft version of a key-value lookup table: instead of an exact match between a query and one key, a query is compared against every key, producing similarity scores that are turned into weights via softmax, and the output is a weighted sum of the corresponding values. The lecture illustrates this with a toy example, representing the word "learned" in a sentence by attending, to varying degrees, to other words in the same sentence such as "Stanford" and "CS224N."

Computing queries, keys, and values (14:15)

Formally, each word embedding is transformed by three separate learned matrices into a query, a key, and a value vector. Attention scores are the dot product of a query with every key in the sequence; a softmax over these scores gives attention weights, and the output for a word is the weighted sum of all value vectors using those weights. Using separate query and key matrices, rather than comparing raw embeddings directly, effectively gives a low-rank, learnable way of deciding what should attend to what, including whether a word should attend to itself.

Fixing self-attention's gaps: position, nonlinearity, and masking (21:23)

Plain self-attention has three problems that need fixing before it can replace RNNs. It has no notion of sequence order, since it operates on a set of vectors, so position vectors (either fixed sinusoidal patterns or learned parameters) are added to word embeddings at the input, though learned position vectors limit the model to a maximum sequence length. It has no nonlinearity, since stacking self-attention layers alone just re-averages value vectors, so a feed-forward network is applied independently to each word's output after attention. Finally, for tasks like language modeling where the model must not see future words, attention scores to future positions are masked to negative infinity before the softmax so their weight becomes zero; this masking is used in decoders but not in encoders, which are allowed to see the whole input.

Multi-head attention and scaled dot-product attention (41:44)

A single self-attention operation averages information for one reason at a time, but a word may need to attend to different other words for different reasons at once, such as syntactic role versus topical meaning. Multi-head attention runs several attention operations in parallel, each with its own smaller query, key, and value matrices projecting into a lower-dimensional space, and concatenates their outputs before a final linear transformation combines them. This computation is implemented efficiently as matrix operations over the whole sequence at once, called the sequence-stacked form. Because dot products between vectors grow large as dimensionality increases, which can shrink softmax gradients, scores are divided by a constant based on the model dimensionality, a fix called scaled dot-product attention.

Residual connections, layer normalization, and the full decoder block (58:00)

Two additional optimization tricks are needed for a full Transformer block. Residual connections add each sublayer's input back to its output, giving gradients a direct path through the network and helping avoid vanishing gradients. Layer normalization rescales each word vector to have roughly unit mean and standard deviation (computed independently per word, not shared across the batch or sequence), which helps stabilize training. A Transformer decoder block applies masked multi-head self-attention, then a residual connection and layer normalization, then a feed-forward layer, then another residual connection and layer normalization, and this block is repeated several times to form the full decoder.

Encoder, encoder-decoder, and Transformer results (1:09:09)

A Transformer encoder is nearly identical to the decoder but omits masking, allowing bidirectional context. The original "Attention Is All You Need" architecture is an encoder-decoder model, where the decoder adds a cross-attention step: its own vectors form the queries, while the encoder's output vectors supply the keys and values, letting every decoder position attend over the whole encoded source sentence. Transformers achieved competitive machine translation results while training far more efficiently than RNN-based systems, because parallelization allowed much more data and compute to be used, which later enabled the large-scale pre-training covered in the next lecture. The lecture closes by noting a major drawback: self-attention's compute and memory cost grows quadratically with sequence length, which remains an active area of research, alongside other proposed modifications to position representations and the architecture generally.

Before you watch

• Watch the previous lecture on attention in encoder-decoder machine translation, since this lecture builds directly on that attention mechanism and assumes you know softmax-weighted averaging.
• Be comfortable with matrix multiplication and how linear layers transform vectors, since query, key, and value projections are presented as matrix operations throughout.
• Review vanishing gradients and residual connections from the LSTM lecture, since they motivate why residual connections are used in the Transformer.

Check your understanding

1. What two specific problems with RNNs does self-attention solve, and how does removing recurrence solve each one?
2. Walk through how a self-attention output is computed for one word, from query/key/value projections to the final weighted sum.
3. Why does self-attention need explicit position representations, and what is the tradeoff between sinusoidal and learned position embeddings?
4. Why is masking used in a Transformer decoder but not in a Transformer encoder, and how is masking implemented mathematically?
5. What is the purpose of having multiple attention heads instead of one, and what is the main computational drawback of self-attention as sequence length grows?