Lecture 16: Alignment - RL 1

What this lecture covers

The lecture closes out RLHF and then pivots to reinforcement learning from verifiable rewards (RLVR), the technique behind reasoning models like OpenAI's o1. It first finishes earlier material on DPO variants and a phenomenon called overoptimization, where a policy's measured reward keeps climbing while its real quality plateaus or drops. It then argues that human-preference RL is hard to scale and easy to hack, motivating a shift to domains with fast, checkable rewards such as math and code.

The bulk of the lecture builds up PPO and then simplifies it into GRPO, the algorithm used in the course's own assignment. It closes by walking through three real reasoning-model recipes: DeepSeek-R1, Kimi K1.5, and Qwen 3, comparing how each handles data curation, RL algorithm choice, and length control. After watching, you should be able to explain why GRPO drops PPO's value network, and describe the training pipeline that turns a base model into a reasoning model.

Key ideas

• Overoptimization: as you optimize a policy against a fitted reward model, measured reward keeps rising but real human-preference win rate eventually stalls or falls, because the reward model diverges from true preferences.
• RLHF is a contextual bandit: a language model gets a prompt, produces one output, and receives one reward, with no state transitions to model.
• PPO's clipping objective: bounds how far the updated policy's likelihood ratio can move from the old policy, capping how much reward a single update can capture.
• Generalized advantage estimation (GAE): PPO's variance-reduction machinery, which needs a separate value network roughly as large as the policy itself.
• GRPO's group baseline: for a given prompt, sample several responses, then compute each one's advantage as (reward - group mean) / group std, removing the need for a value model.
• Reward shaping: KL regularization is applied per token, while the actual task reward (correct or not) is applied only at the final token.
• Outcome vs. process rewards: DeepSeek-R1 found outcome-only rewards (correct/incorrect) worked better in practice than process reward models or search-based methods like MCTS.
• Length and format rewards: R1 uses a format reward to force chain-of-thought into tags, and a language-consistency reward to stop the model from mixing languages mid-reasoning.

Walkthrough

Finishing RLHF: DPO variants and overoptimization (0:05)

The lecture recaps DPO as supervised learning on a re-parameterized reward, then covers two variants used in AI2's Tulu 3 work: SimPO, which drops the reference policy and normalizes by response length, and length-normalized DPO. A key empirical point follows: which method "wins" depends heavily on the base model and SFT quality, so single-paper RL results shouldn't be treated as universal conclusions. The section ends on overoptimization and on RLHF models being poorly calibrated, since RL training is not the same probabilistic distribution-matching as pretraining.

Why move to verifiable rewards (11:12)

Human preference signals are expensive to collect and easy to hack. The lecture reframes the goal: borrow from domains like AlphaGo and AlphaFold where a true reward can be computed quickly and cheaply, and apply that mindset to tasks like math where correctness is checkable.

PPO in detail (13:13)

Starting from the plain policy gradient, the lecture introduces TRPO's importance-sampling correction for sampling from a stale policy, then PPO's clipped surrogate objective as a simpler alternative to KL constraints. It walks through a real PPO implementation: an outer loop of rollouts, loss computation, and gradient steps, with a value-function loss alongside the policy loss, per-token KL reward shaping, and generalized advantage estimation with tunable bias-variance tradeoffs.

From PPO to GRPO (27:20)

GRPO keeps PPO's clipping idea but replaces GAE with a group-relative advantage: sample several responses per prompt, subtract the group mean reward, and divide by the group standard deviation. This removes the value network entirely. The lecture also covers a critique of GRPO (from a paper referred to as "Dr. GRPO"): dividing by the group standard deviation over-weights problems that are too easy or too hard, and dividing by response length creates a bad incentive to pad wrong answers with extra length.

DeepSeek-R1 case study (43:31)

R1-Zero applies GRPO directly to a base model with only accuracy and format rewards, reaching near-o1 performance without any chain-of-thought fine-tuning. The full R1 pipeline adds SFT initialization on long chain-of-thought data, a language-consistency reward, and a final RLHF stage. The lecture notes R1's two negative results: process reward models and Monte Carlo tree search did not help, leaving outcome-based GRPO as the working recipe. R1's distillation into smaller Qwen models is also covered.

Kimi K1.5 case study (59:45)

Kimi K1.5 matches R1's results with a different RL algorithm that turns out to be mathematically close to GRPO despite starting from a DPO-style derivation. Its data curation excludes multiple-choice and true/false questions as too easy to hack, and filters out examples the model already solves reliably. Its length reward compresses reasoning chains once training stabilizes, rewarding short correct answers and only mildly penalizing overly long incorrect ones.

Qwen 3 case study (1:13:57)

Qwen 3 follows a similar SFT-then-RL pipeline but adds thinking-mode fusion: fine-tuning a single model to support both a "think" and "no-think" mode via tags, which lets users cap the thinking budget at inference time. The lecture highlights that Qwen 3's reasoning RL used a surprisingly small dataset (under 4,000 examples) and shows a tradeoff where general RL improves instruction-following but can hurt math performance in the thinking mode.

Before you watch

• Know the DPO derivation and RLHF setup from the previous lecture, since this one assumes it.
• Be familiar with basic policy gradient methods and the idea of a baseline for variance reduction.
• Some familiarity with KL divergence and importance sampling will help with the PPO and GRPO derivations.

Check your understanding

1. Why does GRPO not need a separate value network, and what replaces it?
2. What is overoptimization, and why does it appear even when a policy's measured reward keeps increasing?
3. According to the lecture, why does dividing GRPO's advantage by the group standard deviation bias training toward very easy or very hard problems?
4. What two RL techniques did DeepSeek-R1 try and abandon, and why?
5. How does Kimi K1.5's length reward avoid strongly penalizing long incorrect answers?