Lecture 15: Alignment - SFT and RLHF

What this lecture covers

After lectures on pre-training architecture, optimization, and data, the course turns to post-training: how a raw pre-trained model like GPT-3 becomes an instruction-following, safety-conscious system like ChatGPT. The lecture is built around the InstructGPT three-stage pipeline and covers the first two stages in depth: supervised fine-tuning (SFT) on expert demonstrations, and reinforcement learning from human feedback (RLHF) using pairwise preference data.

By the end, you should be able to describe what different styles of instruction-tuning datasets look like and why they matter, explain the risk that SFT can teach a model to hallucinate, describe how RLHF reframes the problem as reward maximization rather than distribution matching, and follow the derivation connecting the Bradley-Terry preference model to Direct Preference Optimization (DPO).

Key ideas

• Post-training as behavior shaping: pre-training packs a model with latent capabilities; post-training uses curated data and feedback to make those capabilities reliably accessible and safe.
• Instruction data paradigms: aggregated NLP-task datasets (FLAN), human-crowdsourced conversational data (OpenAssistant), and model-generated data (Alpaca) each produce noticeably different training examples.
• The knowledge-vs-hallucination trade-off: fine-tuning a model to reproduce responses beyond what it already knows (such as citations it never learned) can teach it to fabricate plausible-looking answers rather than the underlying fact.
• Mid-training blurs pre-training and post-training: modern pipelines increasingly mix instruction-tuning data into the later, decaying-learning-rate phase of pre-training rather than treating SFT as a fully separate stage.
• RLHF reframes the objective: instead of imitating a reference distribution of "correct" outputs, RLHF treats the language model as a policy chosen to maximize a learned reward function.
• Pairwise preference data is easier to collect than demonstrations but still hard to get right: annotators under time pressure struggle to verify facts and math, and can be biased toward longer or more stylistically polished responses.
• AI feedback has become dominant: because strong models agree with human raters nearly as often as humans agree with each other, and are far cheaper, many pipelines (Tulu, Constitutional AI, Zephyr) use model-generated preference labels.
• DPO turns an RL problem into a supervised one: by assuming the optimal RLHF policy has a closed form in terms of a reward, DPO substitutes that form into the pairwise preference model to get a maximum-likelihood loss that avoids training a separate reward model or running on-policy RL.

Walkthrough

From pre-trained model to useful assistant (0:05)

The lecture opens by contrasting GPT-3, which was impressive but not very usable, with ChatGPT, which follows long, compound instructions reliably. Post-training is framed as the process that makes this transition happen: collecting data of desired behaviors and training the model to produce them, while also adding safety and content-moderation guardrails needed for a public product.

Three paradigms of instruction data (4:07)

The lecture compares three representative instruction-tuning datasets: FLAN, built by aggregating many existing NLP task datasets into instruction-formatted examples; OpenAssistant, produced by volunteer annotators writing detailed conversational responses (sometimes including citations); and Alpaca, generated by prompting a language model to create instructions and using another model to generate responses. A live audience exercise, where students write responses to a prompt by hand, is used to show how difficult and time-consuming it is for humans to produce long, high-quality responses compared to a model like GPT-4o.

Why "high quality" data can teach hallucination (18:20)

Using an OpenAssistant example with a citation, the lecture explains a subtle risk: fine-tuning a model to reproduce a response that includes knowledge or citations it doesn't already have can teach it two things at once, the actual fact, and a more general habit of fabricating plausible-sounding references whenever a question looks complex. This links to John Schulman's argument that on-policy reinforcement learning helps because it teaches the model based on what it can already produce, rather than forcing it to imitate answers beyond its knowledge.

Safety tuning and the scale of SFT (24:24)

The lecture briefly covers safety tuning as a related but distinct problem: models need to refuse genuinely harmful requests without over-refusing benign ones that merely sound alarming, and research has shown that even a few hundred well-chosen examples can meaningfully shift this behavior. It then explains how modern pipelines increasingly blur pre-training and post-training by mixing instruction-tuning data into the later, learning-rate-decay phase of pre-training (sometimes called mid-training), which helps avoid catastrophic forgetting and gets more leverage from the data, illustrated with MiniCPM's two-stage data mix.

Reframing the objective for RLHF (42:36)

The lecture marks a conceptual shift: rather than imitating samples from an unknown reference distribution as in SFT, RLHF looks for any policy that maximizes a reward function over completions. Two motivations are given: full demonstrations are expensive to collect, while reward judgments (which output is better) are cheaper, and humans are often better at judging output quality than generating it themselves, illustrated by an annotator who preferred AI-written summaries to her own.

Collecting and using pairwise feedback (46:40)

The lecture walks through how preference data is gathered in practice, referencing InstructGPT and a leaked Google Bard annotation guideline that both emphasize helpfulness, truthfulness, and harmlessness. A second interactive exercise has students judge pairs of responses under time pressure, revealing that longer but factually flawed responses often win, illustrating length and style biases that affect both human and model-based judges. The lecture then covers why AI-generated feedback (as used in Tulu 3, Constitutional AI, and Zephyr) has become common: it is far cheaper than human annotation and agrees with human judgments about as often as humans agree with each other, though it inherits the same length bias.

From PPO to DPO (1:05:00)

The lecture defines the RLHF objective as maximizing expected reward under a learned reward model while staying close (via KL divergence) to the original SFT model, with rewards modeled through the Bradley-Terry pairwise preference formulation. PPO is introduced briefly as the standard algorithm behind InstructGPT, using advantage estimates and clipped probability ratios to keep updates stable, with a fuller treatment deferred to the next lecture. The lecture then derives DPO: by assuming the optimal policy has a closed form involving the reward and the reference policy, the reward can be rewritten in terms of the policy itself, turning the RLHF objective into a supervised maximum-likelihood loss over pairwise comparisons, without a separate reward model or on-policy sampling.

Before you watch

• Watch or review the previous lecture in this course (Lecture 14 on data filtering and deduplication), since this lecture assumes familiarity with pre-training data pipelines.
• Some background in the policy gradient theorem or basic reinforcement learning will make the PPO and DPO sections easier to follow.

Check your understanding

1. Why might fine-tuning on a fully correct but citation-heavy dataset make a model more likely to hallucinate?
2. What are the two main reasons the lecture gives for preferring pairwise feedback over collecting full demonstrations?
3. How does mixing instruction-tuning data into the late stage of pre-training help avoid catastrophic forgetting?
4. What biases showed up in the class's own pairwise-judgment exercise, and how do they relate to biases seen in AI feedback?
5. What assumption about the optimal RLHF policy allows DPO to replace reinforcement learning with a supervised loss?