Lecture 16, Part 3: Model-Based Offline RL

What this lecture covers

This part of the lecture turns to model-based approaches to offline RL, arguing they are a natural fit because a learned dynamics model can be trained once on a fixed data set and then reused to generate a policy. It explains why the same out-of-distribution problem seen with Q-functions also affects models: a policy can exploit inaccuracies in the model to reach states the model erroneously predicts as high-reward.

The lecture walks through two Dyna-style fixes that penalize the reward according to model uncertainty (MOPO) or directly penalize the critic on model-generated data (COMBO), each with some accompanying theory about how close the learned policy can be guaranteed to get to the true optimum. It closes with a non-Dyna alternative, the trajectory transformer, which plans directly using a large sequence model trained on entire trajectories.

Key ideas

• Out-of-distribution states problem: unlike the out-of-distribution action problem for Q-functions, model-based offline RL faces an additional risk where the policy can exploit the model to reach out-of-distribution states that the model incorrectly predicts as high-reward.
• Uncertainty penalty (MOPO): add a penalty term u, an estimate of model error (for example, disagreement across an ensemble of models), to the reward so that exploiting model inaccuracy is not worth it once the penalty outweighs the fabricated gain.
• Theoretical guarantee: under the assumption that u upper-bounds the true model error, the learned policy's true return is guaranteed to be at least as good as the best policy whose expected model error is bounded, which implies the method should at least match the behavior policy and can approach the optimal policy if the model is accurate where it matters.
• COMBO: applies a CQL-like idea to the model-based setting by pushing down Q-values for model-generated state-action pairs while pushing up Q-values for real data, so if the model produces realistic transitions indistinguishable from real data, the two terms cancel and the model's contribution is trusted.
• Trajectory transformer: a non-Dyna alternative that trains a large sequence model (for example, a causally masked transformer) to estimate the joint probability of entire state-action trajectories, discretized per dimension, and then plans by searching for action sequences with high probability under this model and high reward, using a reward-guided variant of beam search.

Walkthrough

Why the out-of-distribution problem is worse for models (0:02)

The lecture explains that Dyna-style model-based methods, which use real states and actions as starting points for short simulated rollouts, face an analogous but compounded problem to Q-function overestimation: a policy can learn to trick the model into predicting unrealistically high-reward outcomes by driving it into out-of-distribution states, and errors can then compound as the model continues rolling out from those bad states.

MOPO: penalizing uncertainty in the reward (2:02)

The lecture introduces MOPO (model-based offline policy optimization), which adds an uncertainty penalty u to the reward, typically estimated from disagreement across an ensemble of learned dynamics models, so that exploiting the model's inaccuracies becomes less rewarding than staying close to well-understood transitions. It walks through the accompanying theoretical result: if u reliably upper-bounds the true model error, the learned policy's true return is bounded below by the best policy whose expected model error is small, implying the method should be at least as good as the behavior policy and can approach optimal if the model is accurate on the states the optimal policy would visit.

COMBO: applying CQL-style penalties to model data (10:07)

The lecture presents COMBO, which adapts the CQL idea to model-based RL by pushing down Q-values on state-action pairs sampled from the model while pushing up Q-values on real data-set pairs. It explains that if the model produces transitions indistinguishable from real data, these two effects roughly cancel and the model is effectively trusted, while transitions that look clearly synthetic get penalized, without needing to modify the reward function directly.

Planning with the trajectory transformer (12:08)

The lecture describes a non-Dyna approach: train a large sequence model, such as a causally masked transformer, over entire trajectories discretized per state and action dimension, so the model captures the joint probability of state-action sequences under the behavior policy. Because offline training doesn't require active data collection, it becomes practical to use very large, expressive models like transformers. Planning is then done with a reward-guided variant of beam search that samples candidate next tokens from the model (keeping the plan close to what the data supports) and keeps the highest-reward prefixes, illustrated with an animation of long-horizon humanoid predictions.

Before you watch

• Watch Lecture 16, Parts 1 and 2 first for the actor-constraint and CQL solutions to distributional shift in model-free offline RL, which this lecture parallels in the model-based setting.
• Review the model-based RL and Dyna-style methods lecture from earlier in the course, since this lecture assumes familiarity with that framework and terminology.
• Basic familiarity with ensembles for uncertainty estimation, covered in the model-based RL unit, is helpful for understanding the uncertainty penalty.

Check your understanding

1. Why does model-based offline RL face an out-of-distribution states problem in addition to the out-of-distribution actions problem seen in Q-learning methods?
2. What role does the uncertainty penalty u play in MOPO, and what does the accompanying theory guarantee if u upper-bounds the true model error?
3. How does COMBO adapt the CQL idea to model-generated data, and why does this let it trust an accurate model automatically?
4. How does the trajectory transformer's planning approach ensure that generated plans stay close to what the data supports?