Lecture 9: Pretraining

What this lecture covers

This lecture asks how a model can build reusable representations of language before it ever sees a labeled task. It starts with a practical problem: fixed word vocabularies cannot handle new words, typos, or richly inflected languages, which motivates splitting text into subword pieces. It then builds the case for pretraining as a general recipe: mask or predict part of a naturally occurring text and force a network to reconstruct it, which teaches it syntax, co-reference, world knowledge, and sentiment along the way.

The lecture sits right after the Transformers lecture in CS224N and works through the three architecture families introduced there: encoders, encoder-decoders, and decoders. By the end you should be able to explain how BERT, T5, and GPT differ in what they pretrain on and what they are good for, and understand why decoder-only models scaled up into the in-context learning and chain-of-thought behaviors seen in GPT-3.

Key ideas

• Subword tokenization: vocabularies are built from frequently co-occurring character sequences rather than whole words, so rare or novel words split into known pieces instead of becoming an unknown token.
• Pretrain-then-finetune paradigm: a model first learns general parameters from large unlabeled text, then those parameters are adapted with a smaller labeled dataset for a specific task.
• Pretraining is language modeling: nearly all pretraining objectives are versions of predicting missing or upcoming words in real text, which requires no manual labels.
• Masked language modeling (BERT): an encoder is trained by masking roughly 15% of subword tokens and predicting the originals from bidirectional context.
• Span corruption (T5): an encoder-decoder masks contiguous spans in the input and generates the missing spans as output, combining bidirectional input context with generation.
• Decoder-only pretraining (GPT): a decoder is trained purely to predict the next token, which scales into large models used directly for generation.
• In-context learning: sufficiently large decoder-only models can perform new tasks from examples placed in the prompt, without any parameter updates.
• Parameter-efficient fine-tuning: techniques like prefix tuning and low-rank weight updates adapt a pretrained model by training a small number of new parameters instead of all of them.

Walkthrough

Subword tokenization and the limits of fixed vocabularies (2:06)

The lecture opens by revisiting how word2vec-style models assumed a finite vocabulary learned from training data, which breaks down for typos, novel derived words, and languages with heavy morphology (the example given is a Swahili verb with over 300 conjugations). Mapping every unfamiliar string to a single unknown token throws away information. The fix is a subword algorithm that starts from individual characters and iteratively merges the most frequently co-occurring adjacent pairs into new vocabulary entries, so common words stay whole while rare words split into meaningful pieces. This subword approach, originally built for machine translation, underlies tokenization in essentially all modern language models.

From static word embeddings to contextual pretraining (14:20)

The lecture contrasts word2vec, which gives a word like "record" one fixed vector regardless of meaning, with the goal of contextual representations that distinguish "record" the verb from "record" the noun based on surrounding words. Earlier in the course, only word embeddings were pretrained while the rest of the network was randomly initialized; the shift now is to pretrain the entire network, including the Transformer stack, so that fine-tuning for a downstream task needs far less labeled data and starts from far better parameters.

Reconstructing masked input as a general training signal (19:25)

Pretraining is framed as reconstructing an input sequence after part of it is hidden. Worked examples (predicting a missing city name, a missing verb, a missing pronoun's referent, a missing word tied to sentiment) show how this single objective implicitly teaches syntax, co-reference, semantics, and sentiment. The lecture explains why starting fine-tuning from pretrained parameters, rather than random ones, tends to land in a better region after gradient descent, and why the huge scale of unlabeled text available for pretraining makes it more valuable than adding more labeled fine-tuning data.

Pretraining encoders: masked language modeling and BERT (36:37)

Because encoders see bidirectional context, ordinary next-word prediction is trivial for them, so the lecture introduces masked language modeling: replace about 15% of subword tokens (via masking, random substitution, or leaving them unchanged) and predict the true tokens from the surrounding context. This is the objective behind BERT, which also experimented with a next-sentence-prediction task using segment embeddings; the lecture notes that later work (RoBERTa) showed next-sentence prediction was unnecessary and that training on more data helped more. The section also covers using BERT's output vectors for classification tasks, BERT's scale (110M and 340M parameters), and lightweight fine-tuning methods such as prefix tuning and low-rank adaptation that freeze most pretrained parameters.

Pretraining encoder-decoders: span corruption and T5 (58:55)

For encoder-decoder models, the lecture describes span corruption: mask out spans of the input, feed the corrupted sequence to the encoder, and have the decoder generate the missing spans in sequence. This was shown to outperform plain language-modeling-style pretraining for encoder-decoders and is used in T5. The lecture also highlights a striking property of models pretrained with salient-span masking: after fine-tuning on some trivia questions, they can answer new trivia questions correctly some of the time, implying they store and retrieve factual knowledge learned during pretraining, though their answers are fluent even when wrong.

Pretraining decoders: GPT, scale, and in-context learning (1:03:00)

The final and longest section traces decoder-only pretraining through GPT (117M parameters), GPT-2 (1.5B parameters, notable for coherent long-form generation), and GPT-3 (175B parameters, trained on 300 billion words). GPT-3 introduces in-context learning: given a prompt containing examples of a pattern (such as word translations or arithmetic), the model continues the pattern for a new input without any gradient updates. The lecture also introduces chain-of-thought prompting, where prompts include worked reasoning steps, which makes models more likely to reach correct answers on multi-step problems. It closes by noting these are emergent, still poorly understood behaviors, and that pretrained models also learn and can amplify biases present in their training text.

Before you watch

• Come from the CS224N Transformers lecture (self-attention, encoders, decoders, and encoder-decoders), since this lecture assumes that architecture vocabulary.
• Review how language modeling and cross-entropy loss work, since pretraining objectives are framed as variants of language modeling.
• Recall word2vec's fixed, context-independent embeddings, since the lecture uses them as the baseline pretraining improves on.

Check your understanding

1. Why can't an encoder be pretrained with a plain next-word-prediction objective the way a decoder can?
2. What problem does subword tokenization solve that a fixed word-level vocabulary cannot?
3. How does span corruption in T5 differ from masked language modeling in BERT?
4. What does in-context learning let a large decoder-only model do that fine-tuning does not require?
5. Why might starting fine-tuning from pretrained parameters lead to a better outcome than training the same architecture from random initialization?