Lecture 20: Transformers and Attention

What this lecture covers

This lecture answers why Transformers replaced recurrent networks as the dominant architecture for sequence prediction, and how self-attention actually works as a set of matrix operations. It follows the two lectures on recurrent networks and LSTMs, and precedes a lecture that implements a Transformer in NumPy.

After watching, you can contrast the latent-state approach (RNNs) with the direct-prediction approach (temporal convolutions, Transformers) to time series modeling, derive the self-attention formula from keys, queries, and values, explain why raw self-attention needs a causal mask and positional encoding for sequence tasks, and describe the full Transformer block including residual connections and layer normalization.

Key ideas

• Latent-state vs. direct prediction: RNNs summarize the whole history in one hidden state (unlimited history in theory, but vanishing/exploding gradients in practice); direct-prediction methods map the input sequence to outputs without a compressed running state, trading state-based efficiency for shorter, more direct compute paths.
• Temporal convolutional networks: causal convolutions (no arrows from future to present) give a direct-prediction alternative to RNNs, but their receptive field is limited by kernel size and depth, and widening it (larger kernels, dilation, pooling) adds parameters or loses information.
• Self-attention: defined as softmax(K @ Q.T / sqrt(d)) @ V, where K, Q, V are linear projections of the input; it mixes information across all time steps in a single layer without adding parameters as sequence length grows.
• Permutation equivariance: self-attention treats all positions symmetrically, so permuting the input rows produces the same permutation of the output rows, this is what makes masking and positional encoding necessary for sequence tasks.
• Causal masking: subtracting infinity from the upper-triangular entries of the pre-softmax attention matrix, before the softmax, forces each output position to depend only on current and earlier positions.
• Positional encoding: sine functions at multiple frequencies, added to (a projection of) the input, inject each token's position in the sequence since self-attention itself carries no notion of order.
• Transformer block: self-attention followed by a residual connection and layer norm, then an independent two-layer feed-forward network applied per position, followed by another residual connection and layer norm.
• Compute cost: self-attention requires forming a T x T matrix, giving O(T^2 * d) cost, which is the main scaling limitation and the reason Vision Transformers use patch embeddings instead of per-pixel attention.

Walkthrough

Two approaches to time series modeling (0:00)

The lecture reintroduces the causal time series prediction task: predicting y_1...y_T from x_1...x_T such that y_t depends only on inputs up to time t. It frames RNNs as a latent-state approach that compresses history into a hidden vector, contrasted with a direct-prediction approach that maps the input sequence to outputs without a compressed state, at the cost of having to recompute over the full sequence for each new prediction.

Temporal convolutional networks (10:08)

As a non-Transformer example of direct prediction, the lecture describes causal convolutions, where connections only flow forward in time, giving architectures like WaveNet. It shows that the receptive field of such a network is bounded by kernel size and network depth, and that increasing it via larger kernels, pooling, or dilated convolutions each carries a downside: more parameters, information loss, or awkward architectural tradeoffs.

From attention to self-attention (20:15)

Attention is introduced generally as any mechanism that forms a weighted combination of components. The lecture first shows attention as originally used with RNNs: rather than classifying from only the final hidden state (which is biased toward recent inputs), a weighted sum over all hidden states is formed, with weights computed from a learned scoring function and normalized with softmax. This motivates self-attention as a standalone mechanism rather than an RNN add-on.

Deriving self-attention (28:18)

Self-attention is defined on three matrices, keys K, queries Q, and values V, each of shape T x d, each row formed independently from the corresponding row of the input via a linear transformation. The operation is softmax(K @ Q.T / sqrt(d)) @ V: the inner products form a T x T similarity matrix, softmax normalizes each row to sum to one, and multiplying by V produces, for each output position, a weighted combination of all value rows. The lecture notes self-attention is permutation-equivariant, mixes information across all time steps without adding parameters, but costs O(T^2 * d) to compute and store.

The Transformer block (44:40)

A Transformer block maps z_i to z_i+1: self-attention is applied to K, Q, V formed as linear projections of z_i, the result is added back to z_i and passed through layer norm, then an independent two-layer ReLU feed-forward network is applied to each row (time step) independently, followed by another residual connection and layer norm. This mirrors the standard PyTorch Transformer block and the original Transformer paper's design.

Causal masking (54:55)

Because plain self-attention lets every output depend on every input, a mask matrix with zero on and below the diagonal and negative infinity above it is subtracted before the softmax, since exp(-infinity) = 0. In practice the full T x T inner product is still computed and then masked, rather than only computing the lower-triangular part, because dense matrix multiplication is faster on hardware than exploiting sparsity.

Positional encoding (59:00)

Since self-attention is permutation-equivariant, ordering information is added explicitly: sine functions at a range of frequencies (similar to a Fourier basis), one set of values per position, are added to a projection of the input so the network can distinguish otherwise identical tokens by their position in the sequence.

Transformers beyond time series (1:05:02)

The lecture closes by noting the same pattern (mix data via self-attention, then apply an independent feed-forward network per position) has become dominant well beyond sequence prediction, including Vision Transformers (using patch embeddings instead of per-pixel attention to keep the T x T matrix tractable) and graph Transformers (embedding graph structure into the mask or attention scores).

Before you watch

• Review the recurrent neural network and LSTM lectures, since this lecture is framed as a contrast to the latent-state approach they introduce.
• Be comfortable with matrix multiplication, softmax, and layer normalization.
• Recall the earlier discussion of matrix-matrix multiplication efficiency, which explains why masking is done by zeroing out a dense matrix rather than avoiding the unneeded computation.

Check your understanding

1. What is the fundamental tradeoff between the latent-state (RNN) approach and the direct-prediction approach to time series modeling?
2. Why does self-attention need a mask for causal time series tasks, and how does subtracting infinity before the softmax enforce it?
3. Why is positional encoding necessary for Transformers but not for RNNs?
4. What is the compute and memory cost of self-attention in terms of sequence length T and dimension d, and why does this motivate patch embeddings in Vision Transformers?