Lecture 11, Part 1: Model-Based RL and Distributional Shift

What this lecture covers

Lecture 11 opens the model-based reinforcement learning segment of CS 285, following the known-dynamics planning methods (random shooting, CEM, MCTS, LQR) from Lecture 10. This part asks what happens once the dynamics model itself must be learned from data rather than given, and introduces successive versions of a model-based RL algorithm to address the distributional shift problem that arises.

After watching, you'll be able to describe why the naive "collect data once, fit a model, plan" approach fails with high-capacity models, and explain the iterative data-collection and replanning fixes (model-based RL versions 1.0 and 1.5, the latter being model predictive control) that address it.

Key ideas

• Model-based RL basics: learn a dynamics model f(s_t, a_t) = s_t+1 (or a distribution p(s_t+1 | s_t, a_t)) from data, then plan through it using methods from the previous lecture.
• System identification: the classical robotics analogue, where a known physics model's few unknown parameters (mass, friction, etc.) are fit from data; it works well because the parameter space is small and constrained.
• Version 0.5 (naive): collect data with a random policy, fit a model by supervised learning, then plan through it once; this fails badly with expressive models like neural networks.
• Distributional shift: the planner's induced state distribution p_pi_f(s_t) differs from the data-collection distribution p_pi_0(s_t), so the model is queried on states it never saw and gives unreliable predictions, which the planner then exploits.
• Version 1.0 ('DAgger for models'): iteratively replan, execute, observe the true resulting state, and add it to the dataset, retraining the model repeatedly to reduce distributional shift.
• Version 1.5 (model predictive control, MPC): replan after every single action rather than executing a full open-loop plan, which is far more robust to model errors, at greater computational cost.
• Shorter planning horizons work well with MPC: because replanning continually corrects mistakes, even simple methods like random shooting can work well with short horizons under MPC.

Walkthrough

Motivation and naive model-based RL (0:00)

The lecture sets the agenda: build model-based RL algorithms, understand key design considerations, and compare model classes. Version 0.5 is introduced as the simplest recipe — random exploration, supervised model fitting, then planning through the fitted model — with a nod to how this parallels classical system identification when the model class is small and physics-based.

Why naive model-based RL fails (3:52, around the "does it work" chapter)

Using a mountain-climbing analogy, the lecture shows how a model trained on data from a random policy can make confident but wrong predictions once the planner searches outside the region that random exploration covered — leading the planner toward a cliff edge. This is framed explicitly as distributional shift, closely paralleling the imitation-learning failure modes covered earlier in the course, and it gets worse with more expressive model classes like deep networks.

Fixing it with iterative data collection (10:24, from the "Dagger for Models" chapter)

Borrowing the core idea behind DAgger, version 1.0 closes the distributional shift gap by executing planned actions in the real environment, observing the true next state (no human labeling needed, unlike DAgger), and adding this data back into the training set before retraining. This loop of collect-train-plan-execute-append is described as "DAgger for models."

Model predictive control (13:41)

Version 1.5 improves on 1.0 by replanning after every action instead of executing a whole planned sequence before updating the model — this is model predictive control, briefly previewed at the end of Lecture 10. Because mistakes get corrected immediately rather than compounding, MPC is markedly more robust to model errors, though it is considerably more computationally expensive since planning repeats every time step. The lecture notes that shorter planning horizons, and even simple methods like random shooting, tend to work well under frequent replanning.

Before you watch

• Watch Lecture 10 (all parts) first, since this lecture assumes familiarity with planning methods such as random shooting, CEM, and LQR that are used inside the model-based RL loop.
• Recall the distributional shift discussion from the imitation learning lectures earlier in the course; this lecture draws a direct analogy to DAgger.

Check your understanding

1. Why does the naive "collect once, fit once, plan once" recipe (version 0.5) tend to fail with deep neural network dynamics models but often work for classical system identification?
2. Explain the mountain-climbing example: why does the learned model mislead the planner into a bad outcome?
3. How does model-based RL version 1.0 mitigate distributional shift, and how is it analogous to DAgger?
4. What is model predictive control, and why is it more robust to model errors than executing a full open-loop plan?
5. Why can MPC often get away with short planning horizons and simple planners like random shooting?