Lecture 21: RL with Sequence Models & Language Models, Part 3

What this lecture covers

This part moves from single-turn language tasks to multi-turn reinforcement learning problems such as dialogue, where a reward only arrives after several rounds of interaction. Using a visual dialogue benchmark as a running example, it treats a conversation as a POMDP with history-based states and works through the practical choices involved in training an agent to hold a goal-directed conversation.

After watching, you can explain why multi-turn dialogue differs from the single-turn RLHF setting covered earlier, compare per-utterance and per-token time step formulations for value-based RL, and describe how offline RL techniques like CQL and IQL are adapted to train dialogue agents from logged data.

Key ideas

• Multi-turn RL as a POMDP: dialogue and similar tasks (tool use, text games) need a history of past observations and actions as the state, since the reward is delayed until the end of the interaction.
• Contrast with RLHF: reinforcement learning from human feedback optimizes a single-turn response against a learned reward model, while multi-turn dialogue RL must reason about a sequence of turns before any reward appears.
• Cost of online interaction: policy gradient methods work for dialogue but require sampling a real human response for every rollout, making them expensive compared to settings with a static reward model.
• Value-based methods as an alternative: Q-learning style methods can train directly on offline data, such as logged human-human conversations, avoiding repeated live interaction.
• Per-utterance vs. per-token time steps: treating each utterance as one action step keeps the horizon short but gives an exponentially large action space, while treating each token as an action step gives a small discrete action space at the cost of a much longer horizon.
• Q-function design: a per-utterance critic encodes the dialogue history and a candidate utterance through sequence models into a single Q value, while a per-token critic behaves like a language model but outputs Q values instead of next-token probabilities.
• Offline RL machinery: target networks, replay buffers, double Q-learning, and distributional-shift corrections such as CQL or IQL all carry over, with a CQL penalty on per-token Q-values reducing to something close to a standard cross-entropy loss.

Walkthrough

Multi-turn dialogue as a POMDP (0:00)

The lecture introduces visual dialogue, where a questioner bot asks questions to identify a picture the answerer has in mind. Framed as a POMDP, the action is the bot's question, the observation is the answerer's reply, the state is the history of past observations and actions, and the reward only arrives once the questioner makes its final guess. This differs from the earlier RLHF discussion, which optimized single-turn responses against a learned preference reward model.

Training approaches: policy gradients vs. value-based methods (3:14)

Policy gradients remain a viable option and can handle the partial observability using history states, but training against a live human for every rollout is expensive. Value-based methods are appealing because they support offline RL, letting a dialogue agent train on existing conversation logs instead of requiring fresh human interaction for every update.

Choosing a time step: per utterance or per token (5:05)

Two ways to define a time step are compared. Treating each utterance as one time step keeps the horizon short (on the order of the number of conversational turns) but the action space becomes the set of all possible sentences. Treating each token as one time step gives a small, enumerable action space, but the horizon grows to potentially thousands of steps for a short dialogue. Neither choice dominates in the literature.

Value-based methods with per-utterance and per-token actions (7:38)

For per-utterance actions, a sequence model encodes the dialogue history and a candidate utterance into embeddings that a learned critic combines into a Q value; this can pair with an actor network or with decoding methods like beam search to select high-Q actions. For per-token actions, the setup looks like a language model that outputs a Q value for every possible next token instead of a probability, with Bellman backups computed one token at a time.

Implementation details and example papers (12:06)

Standard offline value-based tricks apply: target networks, replay buffers, double Q-learning, and distributional-shift handling via policy constraints, CQL, or IQL. The lecture walks through three papers as concrete instantiations: an actor-critic dialogue agent trained on human sentiment as reward, a CQL-based negotiation chatbot rewarded by sale revenue, and an implicit Q-learning method for natural language generation evaluated on the visual dialogue task.

Before you watch

• Review the RLHF and single-turn language model RL material from earlier in this lecture, since this part explicitly contrasts multi-turn dialogue with that setting.
• Be familiar with offline RL techniques such as CQL, IQL, target networks, and double Q-learning from earlier in the course.

Check your understanding

1. How does the reward structure of multi-turn dialogue RL differ from the single-turn RLHF setting?
2. What is the main practical drawback of using policy gradients to train a dialogue agent that talks to real humans?
3. What is the key trade-off between per-utterance and per-token time step formulations?
4. Why does a CQL penalty on per-token Q-values end up resembling a supervised cross-entropy loss?