Lecture 15: After DPO, with Nathan Lambert

What this lecture covers

Nathan Lambert, a post-training researcher at the Allen Institute for AI, gives a guest lecture on what has happened in language model alignment since Direct Preference Optimization (DPO) became popular. The lecture answers a practical question: DPO is now the default way researchers fine-tune models on human preferences, so what should the field work on next? It sits near the end of the course's sequence on post-training, after the class has already covered supervised fine-tuning and the basic RLHF objective.

After watching, you should be able to explain the difference between instruction tuning, RLHF, and DPO, describe how a preference-based reward model is trained and evaluated, and summarize the empirical tradeoffs between DPO and PPO that Lambert's team found when trying to reproduce industry results with limited compute.

Key ideas

• Instruction tuning vs. supervised fine-tuning: instruction tuning trains a model to follow instructions and is closely tied to RLHF, while supervised fine-tuning is a more domain-specific technique.
• RLHF objective: a policy is optimized to maximize a learned reward while a KL-divergence term keeps it close to the original model, preventing over-optimization.
• Reward models: trained on pairwise preference data (a prompt with a chosen and a rejected completion) using a loss that widens the score gap between the two outputs.
• DPO: replaces the separate reward model and RL update with a single loss computed directly from log-probability ratios between a trained policy and a frozen reference model, making it far simpler to implement than PPO.
• RewardBench: an evaluation suite Lambert's team built to score reward models directly, because there was no standard way to check whether a reward model actually captures what it should.
• Online vs. offline data: online methods (like PPO) generate and re-label data from the current policy during training; DPO is typically trained once on a fixed, offline preference dataset.
• Reward hacking: a powerful optimizer will exploit any gap between the true objective and its reward model, which is why reward evaluation and monitoring matter.

Walkthrough

Why post-training is the current frontier (0:05)

Lambert opens by arguing that the balance of effort in large language model development has shifted from pre-training toward post-training. He notes that companies like Meta buy far more human preference data for post-training than exists in public datasets such as Chatbot Arena, which shapes what independent researchers can realistically attempt. He traces a short history of language modeling from early autoregressive models through the Transformer, GPT-2 scaling laws, and the "Stochastic Parrots" critique, up to ChatGPT's release, arguing that pre-training alone was not sufficient to produce a model like ChatGPT: RLHF-style fine-tuning was also necessary.

Definitions: instruction tuning, alignment, and RLHF (7:11)

The lecture distinguishes several overlapping terms. Instruction fine-tuning trains a model to follow instructions using the same autoregressive loss as pre-training, often on stack-overflow- or Reddit-style question-answer data. Alignment is the broader goal of matching model behavior to what a user wants, and RLHF is one specific technique for it, built on preference data collected by asking people (or models) to rank outputs.

From RLHF math to DPO's shortcut (10:14)

Lambert walks through the standard RLHF objective: maximize reward subject to a KL constraint against a reference policy. Traditional RLHF trains a separate reward model using the Bradley-Terry preference model, then optimizes the policy against it with an algorithm like PPO. DPO instead reformulates the same optimization so that gradient ascent can be applied directly, without training a separate reward model network or building the surrounding RL infrastructure. He shows that DPO still implicitly defines a reward as a ratio of log-probabilities between the trained policy and the reference model, which matters for later discussion of evaluating DPO models.

How the field moved from the DPO paper to DPO models (15:17)

The DPO paper was published months before the community started training well-known models with it. Lambert traces the path: early instruction-tuned models like Alpaca and Vicuna used synthetic and ShareGPT-derived data; the Open Assistant project produced one of the first carefully collected human preference datasets; and an RLHF model from CarperAI showed early promise but was not widely adopted. The turning point was the Zephyr model in September 2023, trained on the synthetically labeled UltraFeedback dataset with an unusually low learning rate, which convinced the community DPO worked. AI2's Tulu 2 project then showed the same recipe scaled to 70-billion-parameter models, after which DPO-tuned "instruct" models became standard across many startups.

Evaluating reward models with RewardBench (24:25)

Because no standard tool existed to evaluate reward models directly, Lambert's team built RewardBench: a set of prompts with manually created chosen and rejected answers, used to check whether a reward model agrees with human judgment. He shows how the leaderboard shifted quickly as more labs submitted models, describes using "LLM as a judge" as one baseline, and highlights a subtly hard category called "chat hard" built from prompts and near-miss rephrasings, which remains unsaturated even as easier categories do. He also shows that DPO models can only be scored correctly as reward models if their reference-model checkpoint is available, which many public releases omit, and that safety-focused categories reveal a split between models that refuse too much and models that refuse too little.

Comparing DPO and PPO empirically (39:35)

Lambert describes unpublished work from his team directly comparing DPO and PPO starting from an instruction-tuned Llama 2 13B model. Switching datasets (from Anthropic's helpful-harmless data to UltraFeedback) produced larger gains than switching algorithms: PPO outperformed DPO by roughly one percent on average, at the cost of much greater implementation complexity, slower training (because PPO must generate fresh completions during training), and many more hyperparameters to tune. Scaling up the reward model or adding more code and reasoning prompts did not reliably improve the final policy.

Online data and where the field is heading (46:40)

The lecture closes by distinguishing what makes PPO different from DPO: PPO generates fresh completions from the current policy and can refresh preference labels over time, while DPO is normally trained once on a fixed dataset assembled from many different models' outputs. Lambert surveys recent methods that try to add "online" updates to DPO, such as self-rewarding models and discriminator-guided DPO, and speculates that Meta's Llama 3 post-training pipeline (which combines supervised fine-tuning, rejection sampling, PPO, and DPO) likely mixes methods depending on how much data and time is available. He ends by naming open problems: better evaluation, more diverse preference datasets, alignment of smaller models, and personalization. A Q&A follows covering reward hacking, alternatives to pairwise preferences, and how to build new preference datasets.

Before you watch

• Review the course's earlier lectures on the RLHF training objective and the DPO derivation, since this talk assumes familiarity with both.
• Some background in reinforcement learning terms like policy, reward, and KL divergence will make the middle section easier to follow.

Check your understanding

1. What problem does DPO solve compared to the standard RLHF pipeline of training a reward model and then running PPO?
2. Why did DPO models take several months after the paper's release to become widely used, and what changed with Zephyr?
3. Why does scoring a DPO model as a reward model require access to its reference-model checkpoint?
4. According to the lecture's experiments, what mattered more for model quality: switching from DPO to PPO, or switching the training dataset?
5. What distinguishes "online" preference learning from the offline setting DPO is normally trained in?