Guest Lecture: Eric Mitchell on RLHF, Algorithms and Applications

What this lecture covers

This guest lecture by Eric Mitchell (Stanford) explains reinforcement learning from human feedback (RLHF), the technique credited with turning GPT-3-level models into ChatGPT-level assistants. It walks through the full RLHF pipeline used for large language models, then introduces direct preference optimization (DPO), an algorithm Mitchell co-developed that removes the RL training loop entirely while optimizing the same underlying objective. It closes with a case study applying DPO to reduce factual errors in generated text.

After watching, you can describe the four-stage RLHF pipeline (pretraining, supervised fine-tuning, reward modeling, RL fine-tuning), explain why preference comparisons scale better than direct reward labeling, derive the core idea behind DPO, and describe how DPO-style training was used to improve factuality without additional human labels.

Key ideas

• Language models as policies: a language model maps token sequences to a distribution over the next token; for RLHF this is usually treated as a contextual bandit, where a whole prompt is the state and a whole response is a single action.
• Why comparisons beat direct reward labels: asking humans to score a response numerically produces noisy, inconsistent ratings, while asking which of two responses is better produces much higher agreement and is faster to collect.
• Bradley-Terry preference model: preference probability is modeled as a function of the difference between two hidden reward scores, giving a maximum-likelihood objective for fitting a reward model from binary preference data.
• The four-step RLHF pipeline: unsupervised pretraining (produces a GPT-3-like model), supervised fine-tuning on human demonstrations, reward model fitting on preference pairs sampled from the fine-tuned model, and finally RL fine-tuning (typically PPO) against that reward model with a KL penalty back to the supervised model.
• Over-optimization risk: without a KL constraint keeping the policy close to the supervised model, RL fine-tuning can exploit inaccuracies in the reward model, since the reward model is only reliable near the data it was trained on.
• Direct Preference Optimization (DPO): using a specific closed-form relationship between optimal policies and reward functions, the Bradley-Terry preference loss can be rewritten directly in terms of policy parameters, letting the policy be trained by a simple classification-style loss with no reward model, no rollouts, and no separate value function.
• DPO's gradient behavior: the DPO loss pushes up the probability of preferred responses and down the probability of dispreferred ones, weighted more heavily on pairs the implicit reward currently gets wrong, and naturally stops updating on pairs it already ranks correctly.
• RLAIF and AI-generated preferences: preference labels do not have to come from humans; models themselves (or automated fact-checking against references like Wikipedia) can generate preference data, as used both for Claude's harmlessness training and for the factuality application described later.

Walkthrough

Why RLHF, and how language models are framed as RL policies (1:06)

Mitchell frames RLHF as the "one weird trick" that separates the muted reception of GPT-3 from the viral reception of ChatGPT. He defines a language model as an autoregressive policy over tokens and explains the contextual bandit framing used throughout: a prompt is the state, a full response is one action, referencing the "Contrastive Preference Learning" paper for readers interested in per-token versus per-sequence formulations.

The RLHF pipeline: pretraining through reward modeling (5:08)

The lecture walks through unsupervised pretraining (producing a GPT-3-style base model) and supervised fine-tuning on human demonstrations, then asks why this isn't sufficient: demonstrations are laborious to scale and imitation cannot exceed human performance. It argues that a reward signal is needed, and that asking humans to directly assign numeric rewards is unreliable, whereas asking which of two sampled responses is preferred is both faster and more consistent. This preference data is fit to a reward model using the Bradley-Terry choice model via maximum likelihood.

RL fine-tuning, its pitfalls, and why RLHF still works (12:13)

With a reward model in hand, the policy is fine-tuned with an RL algorithm (PPO) to maximize reward. Mitchell highlights the distribution shift risk: since the reward model is only accurate near responses the supervised model assigned high probability to, optimizing without a KL penalty toward that reference model lets the policy exploit reward model errors. He shows InstructGPT results where a 1.3B-parameter RLHF model was preferred by humans over a 175B-parameter purely supervised model, underscoring how much the RL stage contributes. He also briefly covers two RLHF variants: learning diversity metrics from human similarity judgments, and using an RLHF-trained "helpful" model to auto-generate harmlessness labels (RL from AI feedback, as used for Claude).

Direct Preference Optimization (23:23)

Mitchell identifies problems with the PPO pipeline: implementation complexity (juggling a reward model, value function, policy, and reference model), heavy resource requirements, and an unidentifiable reward function (rewards for a given prompt can be shifted by an arbitrary constant without changing the optimal policy). DPO resolves this using a closed-form expression relating any reward function to its optimal KL-constrained policy. Substituting this relationship into the Bradley-Terry loss produces a loss defined directly over policy parameters, no reward model or RL rollouts required. He walks through the resulting gradient, showing it pushes up preferred-response probability and down dispreferred-response probability, weighted by how wrong the implicit reward currently is, which naturally implements the KL constraint by ceasing to update once a pair is already correctly ranked.

Comparing DPO and PPO, and where DPO struggles (41:33)

Using a synthetic sentiment-maximization task with a known ground-truth reward, Mitchell shows DPO achieving a stronger reward-versus-KL-divergence tradeoff than PPO, even when PPO uses the true reward function directly. He also shows that DPO's beta coefficient predictably controls the resulting KL divergence, unlike PPO where the same hyperparameter can produce very different outcomes across runs. A subtler failure mode is described: when the chosen and rejected responses share an identical first token (such as a fixed "TL;DR" marker), DPO's gradient for that token cancels out and nothing is learned about it, showing why the supervised fine-tuning stage remains necessary before preference optimization.

Application: reducing hallucinations with DPO (48:35)

The lecture closes with a case study on factuality. Standard RLHF does not reliably reward truthfulness, since human raters are shown to favor agreement with their existing beliefs over accuracy. Instead, preference data is generated automatically by scoring completions for factual correctness against a reference source, without human labelers, and used to train with DPO. On biography generation and medical question answering, this factuality-tuned model reduces incorrect facts per response and increases correct facts per response, outperforming standard RLHF on the same metric.

Before you watch

• Review the policy gradient and PPO material from earlier in the course, since RLHF fine-tuning is described as running an off-the-shelf RL algorithm on top of a learned reward model.
• Recall the maximum entropy IRL and preference-modeling material from the inverse RL lectures, since the Bradley-Terry preference model reappears here in the same form.

Check your understanding

1. Why do preference comparisons scale better as a source of training signal than asking humans for direct numerical reward scores?
2. What problem does the KL penalty to the reference model solve during RLHF fine-tuning, and what goes wrong without it?
3. How does DPO turn the Bradley-Terry preference loss into a loss over policy parameters, without training a separate reward model?
4. Why did DPO fail to learn anything about the shared first token in the TL;DR summarization example, and what does this imply about needing a supervised fine-tuning stage first?