Lecture 12, Part 2: Model-Based RL with Policies

What this lecture covers

This part continues the model-based reinforcement learning discussion by asking why you would use a model-free algorithm, like the policy gradient, on top of a learned model instead of backpropagating through it directly. It builds on the previous part's derivation of the backpropagation (pathwise) gradient and contrasts it with the likelihood-ratio (policy gradient) estimator.

The lecture then turns to a core practical problem: rollouts through a learned dynamics model accumulate error the longer they run, for the same distributional-shift reasons seen earlier in the course with imitation learning. This sets up the motivation for the short-rollout designs used by modern model-based RL algorithms, covered in the next part.

Key ideas

• Likelihood ratio (policy gradient) estimator: does not require differentiating through the transition dynamics, only through sampling; it is valid for stochastic policies and transitions.
• Pathwise (backpropagation) gradient: computes exact derivatives via the chain rule but requires multiplying a long product of Jacobians across time steps, which can explode or vanish.
• Sampling trade-off: the policy gradient avoids the Jacobian product but needs many samples; with a model, samples are cheap compute rather than costly real-world interaction.
• Model-based RL version 2.5: fit a model, generate many trajectories from it, and improve the policy with the policy gradient instead of backpropagation.
• Distributional shift in rollouts: as in imitation learning, small model errors push the policy into unfamiliar states where errors compound, growing roughly with the square of the rollout length.
• Branched short rollouts: sampling states from real trajectories and rolling out only briefly from each reduces compounding error while still covering later time steps.
• On-policy vs. off-policy: because branched rollouts mix data from the collecting policy and the latest policy, off-policy algorithms like Q-learning handle this state-distribution mismatch better than on-policy policy gradient methods.

Before you watch

• Watch the first part of Lecture 12, which introduces the backpropagation gradient for model-based RL.
• Review the distributional shift argument from the imitation learning lectures, since it is reused here.

Check your understanding

1. Why does the pathwise gradient become numerically unstable for long horizons, while the policy gradient does not?
2. What trade-off do you accept when switching from backpropagation to the policy gradient for training with a learned model?
3. Why does branching short rollouts from real-world states reduce accumulated error compared to full-length model rollouts?
4. Why are off-policy methods generally preferred over on-policy policy gradient methods for training on these branched rollouts?