Lecture 12, Part 3: Model-Based RL with Policies

What this lecture covers

This part turns the short-rollout design from the previous part into a concrete algorithm family. It starts with Dyna, an online Q-learning method from Richard Sutton that alternates real transitions with short simulated ones, then generalizes it into the "Dyna-style" recipe used by many modern model-based RL algorithms, illustrated as an extension of the deep Q-learning systems diagram.

By the end, you should understand how these algorithms combine a real-data buffer and a model-generated buffer to train a Q-function or actor-critic method, and what benefits and risks that combination brings.

Key ideas

• Dyna: an online Q-learning algorithm that, after each real transition, updates a learned model and reward function, performs a Q-learning update on the real transition, then repeats K simulated one-step updates sampled from the buffer using the learned model.
• Dyna-style generalization: collect data, train a model, then repeatedly sample buffer states, choose actions (from the buffer, latest policy, or an exploration strategy), simulate short rollouts, and use them to train an off-policy algorithm.
• Two buffers: a real-transition buffer and a much larger model-generated buffer, kept separate so the ratio of real to synthetic data can be tuned.
• Buffer eviction: the model-based buffer is often flushed whenever the model changes, since refreshing it costs only compute, not real-world samples.
• Sample efficiency: using a model to amplify a small amount of real data into many synthetic transitions makes Q-learning more sample efficient when the model is accurate.
• Model bias: an inaccurate model teaches the policy to optimize the wrong objective; ensembles can help by averaging out individual model errors.
• Performance ceiling: these methods tend to learn faster early on but can plateau below the level a purely model-free method would reach, because of the bias introduced by the learned model.

Walkthrough

Dyna: the classic recipe (1:01)

The lecture walks through Dyna step by step: pick an action with an exploration policy, observe the resulting transition, update the model and reward function with it, run a standard Q-learning update on that real transition, and then repeat one-step simulated updates K times using states and actions drawn from the buffer and the learned model.

Generalizing Dyna (4:06)

The lecture relaxes Dyna's specific choices (single-step model updates, actions taken from the buffer) into a general template: collect data, train a model, then repeatedly sample states, choose actions flexibly, simulate short rollouts, and feed the results to a model-free algorithm.

A systems diagram for model-based acceleration (5:06)

Building on the earlier deep Q-learning diagram of parallel processes, the lecture adds a model-training process and a model-rollout process that fills a second buffer, explaining how batches for Q-function regression mix real and model-generated transitions.

Design choices, benefits, and risks (9:09)

The lecture surveys related algorithms such as model-based policy optimization (MBPO) and model-based value expansion, then discusses why model-based acceleration helps (more effective data) and why it can hurt (model bias, unfamiliar states, the need to periodically refresh the buffer with real data).

Before you watch

• Watch the previous part of Lecture 12 on short model-based rollouts, since Dyna is presented as a concrete instance of that idea.
• Be familiar with the deep Q-learning systems diagram (buffer, target network, regression process) from the earlier Q-learning lectures.

Check your understanding

1. What are the main steps of the Dyna algorithm, and which step differs from plain online Q-learning?
2. How does the generalized Dyna-style recipe relax the specific choices made by the original Dyna algorithm?
3. Why are the real-transition buffer and model-generated buffer kept separate rather than merged?
4. What causes these methods to plateau at a lower final performance despite learning faster early on?