Lecture 5, Part 4: Off-Policy Policy Gradients with Importance Sampling

What this lecture covers

This part explains why the policy gradient method covered so far is on-policy — requiring fresh samples after every update — and derives a way to use samples from a different policy via importance sampling. It answers: how expensive is the on-policy requirement in deep RL, and can importance sampling make policy gradient work with off-policy data without breaking down?

After watching, you can explain why plain policy gradients discard data after each small update, derive the importance-sampled policy gradient, and describe why naively importance-sampling over full trajectories causes variance to explode exponentially with the horizon.

Key ideas

• On-policy cost: because the policy gradient expectation is taken under p_theta(tau), each parameter update theta requires new samples from that exact theta; neural networks take many small steps, so this means very frequent, potentially expensive resampling.
• Importance sampling: a general technique for estimating an expectation under p(x) using samples from a different distribution q(x), by reweighting with p(x)/q(x); this is unbiased, though it changes variance.
• Trajectory importance weight: because both policies share the same initial state distribution and transitions, the trajectory-level importance weight reduces to a ratio of policy probabilities across time steps, which is computable even without knowing the dynamics.
• Off-policy gradient: differentiating the importance-sampled objective with respect to the new parameters theta' and evaluating at theta = theta' recovers the original on-policy gradient, confirming the derivation is consistent.
• Exponential variance problem: importance-weighting the reward-to-go term over the full past horizon multiplies many probability ratios together, which shrinks toward zero exponentially in the trajectory length T and blows up variance.
• Practical fix (preview): ignoring the state-marginal probability ratio and keeping only the action-probability ratio at each time step avoids the exponential blow-up; this gives bounded error when the new policy is close to the old one, a point the course expands on later under advanced policy gradients.

Before you watch

• Watch the earlier parts of this lecture on the policy gradient derivation, causality, and baselines.
• Know the on-policy/off-policy distinction introduced in the algorithm-comparison lecture.

Check your understanding

1. Why does the policy gradient method require fresh samples after every small parameter update?
2. How does importance sampling let you estimate an expectation under one distribution using samples from another, and why is it unbiased?
3. Why does importance-sampling the reward-to-go term over the full trajectory cause variance to grow exponentially with the horizon?
4. What approximation avoids the exponential variance blow-up, and under what condition is it reasonable?