Lecture 10: Post-training

What this lecture covers

This lecture asks how a model trained only to predict the next word turns into something like ChatGPT, which follows instructions and produces answers people prefer. It traces the path from a pretrained decoder-only model through prompting techniques that coax it into performing tasks, to instruction fine-tuning, which trains it directly on instruction-output pairs, to methods that optimize for human preference rather than just imitating labeled answers.

The lecture assumes familiarity with pretraining and decoder-only Transformer language models from earlier in the course. After watching, you should be able to explain zero-shot and few-shot in-context learning and chain-of-thought prompting, describe why instruction fine-tuning is needed and what its limitations are, and understand at a conceptual level how reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO) turn human preference comparisons into a training signal.

Key ideas

• Scaling and emergent behavior: as pretraining compute, data, and model size grow, models appear to pick up capabilities, such as modeling beliefs and intentions, beyond simple text statistics, though how "emergent" this is remains debated.
• Zero-shot and few-shot prompting: a pretrained model can be coaxed into performing a task by framing it as text completion, either with no examples (zero-shot) or with a few examples placed in the context (few-shot), without any gradient updates.
• Chain-of-thought prompting: showing a model worked reasoning steps, or simply prompting it with "let's think step by step," substantially improves its performance on multi-step reasoning tasks.
• Instruction fine-tuning: training a pretrained model directly on many instruction-output pairs across many tasks makes it responsive to user intent, but it is expensive to collect and treats all token-level errors as equally bad.
• Reward modeling: since humans give inconsistent numeric scores to individual answers, systems instead collect pairwise preference rankings and fit a reward model (using the Bradley-Terry model) to predict which of two completions a human would prefer.
• RLHF: a language model is optimized with reinforcement learning to maximize the learned reward model's score, with a penalty (a KL-divergence term) that keeps it from drifting too far from its starting distribution and hacking the reward model.
• Direct Preference Optimization (DPO): reward can be expressed directly in terms of the language model's own probabilities, letting preference data be used to fine-tune the model directly through a classification loss, without running reinforcement learning.
• Reward hacking and bias: optimizing a learned reward model risks exploiting its errors, and optimizing for human preference can favor confident-sounding but incorrect or overly verbose answers over truthful, concise ones.

Walkthrough

Scaling pretraining and emergent capabilities (1:06)

The lecture opens with the trend of ever-larger pretraining runs, citing recent models trained on roughly 15 trillion tokens, and asks why simply predicting the next token produces models that seem intelligent. It presents examples suggesting that optimizing next-token prediction leads models to implicitly model agents' beliefs and actions, understand mathematical concepts, and generate working code, not just memorize facts and syntax. This sets up the lecture's throughline: models are increasingly described as general-purpose, multitask assistants, and the rest of the lecture explains the techniques that turn a raw pretrained model into that kind of assistant.

Zero-shot and few-shot in-context learning (7:06)

Using the GPT model series as a throughline, the lecture shows how a pretrained decoder-only model can be prompted to perform tasks it was never explicitly trained on. Zero-shot learning, which emerged with GPT-2, works by framing a task as text completion, for example appending "tldr" to a document to induce summarization, or comparing log probabilities of candidate completions to answer questions. Few-shot learning, which became especially strong with GPT-3, places a handful of task examples directly in the prompt with no gradient updates, and performance improves substantially with model scale, approaching task-specific fine-tuned models on some benchmarks.

Chain-of-thought prompting (17:14)

For tasks requiring multi-step reasoning, such as arithmetic word problems, the lecture introduces chain-of-thought prompting: instead of showing only question-answer examples, the prompt includes worked reasoning steps, which teaches the model to reason through new questions rather than jump straight to an answer. This substantially improves accuracy and grows more effective with model scale. A simpler variant, zero-shot chain-of-thought, just prepends "let's think step by step" to induce reasoning without any worked examples, improving performance considerably over plain zero-shot prompting, though not as much as few-shot chain-of-thought.

Instruction fine-tuning (22:18)

Since pretraining optimizes for predicting plausible text rather than helping users, pretrained models often respond unhelpfully to direct instructions. Instruction fine-tuning addresses this by collecting many instruction-output pairs spanning diverse tasks (question answering, summarization, translation, code, reasoning) and fine-tuning the pretrained model on all of them together, evaluating generalization to unseen tasks. The lecture covers evaluation via broad benchmarks such as MMLU, notes that larger models benefit more from instruction tuning, and mentions that strong models can be used to generate instruction-tuning data for smaller models, and that data quality can matter more than data quantity.

Limitations of instruction fine-tuning (34:32)

The lecture identifies three limitations that motivate the next stage: collecting human-written instruction-output pairs is expensive and gets more expensive for specialized tasks, some tasks (like creative writing) have no single correct answer, and standard token-level training loss penalizes all mistakes equally even though some errors are far worse than others. Underlying all of this is a deeper mismatch: instruction fine-tuning still optimizes next-token prediction on a curated dataset, not human preference directly.

Optimizing for human preferences: reward models and RLHF (39:36)

The lecture introduces the goal of directly maximizing a reward that reflects human preference. Because humans give inconsistent absolute scores to individual completions, the practical approach collects pairwise comparisons (which completion is better) and fits a reward model using the Bradley-Terry model, which relates preference probability to the difference in rewards between two completions. Once a reward model exists, the RLHF pipeline optimizes the language model's parameters to maximize the reward model's score, while subtracting a KL-divergence penalty that discourages drifting too far from the model's initial distribution, which otherwise leads to reward hacking, where the model exploits errors in the learned reward model to score artificially high while producing poor text.

Direct Preference Optimization: a simpler alternative to RLHF (56:50)

Because full RLHF is complex and expensive to implement well, the lecture derives Direct Preference Optimization (DPO), which shows mathematically that the reward function implied by the RLHF objective can be rewritten entirely in terms of the language model's own log-probabilities relative to its initial distribution, with the otherwise-intractable normalization term canceling out when comparing a winning and losing completion. This lets preference data be used to fine-tune the model directly through a straightforward classification loss, without sampling from the model or running reinforcement learning. The lecture reports that DPO performs comparably to full RLHF on summarization and notes it has become the dominant approach in many open-source and production models. The lecture closes by tracing this full pipeline (pretraining, instruction tuning, and preference optimization) to models like InstructGPT and ChatGPT, and flags open concerns including reward hacking and the risk that models learn to sound confident and verbose rather than accurate.

Before you watch

• Review decoder-only Transformer pretraining and next-token prediction from earlier in CS224N, since this lecture builds directly on that foundation.
• Recall basic supervised fine-tuning concepts (loss functions, gradient updates), since instruction fine-tuning and the contrast with RLHF/DPO depend on them.
• No reinforcement learning background is assumed; the lecture explains the RLHF objective and policy optimization at a conceptual level.

Check your understanding

1. What distinguishes zero-shot and few-shot in-context learning from traditional fine-tuning, and why does few-shot performance improve with model scale?
2. Why does chain-of-thought prompting improve performance on multi-step reasoning tasks, and how does the zero-shot version differ from the few-shot version?
3. What three limitations of instruction fine-tuning motivate optimizing for human preferences instead?
4. Why do preference-based systems ask humans to rank pairs of completions rather than assign a numeric score to each one individually?
5. What mathematical step lets Direct Preference Optimization avoid the reinforcement learning loop used in RLHF, and what problem (reward hacking) does the KL-divergence penalty in RLHF address?