Lecture 15, Part 2: Offline RL by Importance Sampling

What this lecture covers

This part of the lecture turns to classic offline and batch reinforcement learning techniques based on importance sampling, methods that predate deep offline RL but still shape how the field understands the problem. It picks up directly from the importance-sampled policy gradient covered earlier in the course and shows why naively applying it to offline RL produces an estimator with exponentially exploding variance in the time horizon.

The lecture then works through several classic fixes: dropping parts of the importance weight, the doubly robust estimator (a variance-reduction technique related to baselines), and marginalized importance sampling, which reweights by state or state-action probabilities instead of action probabilities. These are presented as tools for off-policy evaluation, a building block used inside many offline RL algorithms, rather than full offline RL methods themselves.

Key ideas

• Importance-sampled policy gradient: the standard trick of multiplying the policy gradient by the ratio of trajectory probabilities under the target policy and the behavior policy, so a policy can be evaluated or improved using data collected by a different policy.
• Exponential variance problem: because the importance weight is a product of O(T) action probability ratios, the weights become degenerate as the horizon T grows, meaning the estimate effectively collapses onto one arbitrary sample.
• Splitting the importance weight: the weight can be split into a part accounting for the probability of reaching a state under the target policy versus the behavior policy, and a part accounting for the difference in reward-to-go; dropping the first part is the same approximation used in methods like PPO, but it is not valid when the offline policy needs to differ substantially from the behavior policy.
• Doubly robust estimator: a variance-reduction technique, similar in spirit to a baseline in policy gradients, that subtracts an estimated value function from the importance-weighted reward and adds back its expected value; it remains unbiased regardless of how accurate the estimated value function is, and extends recursively from the bandit case to the multi-step case.
• Marginalized importance sampling: instead of weighting by products of action probabilities, this approach estimates importance weights directly as ratios of state or state-action marginal probabilities, avoiding the exponential blow-up, by solving a Bellman-like consistency condition for the weights using only samples.

Walkthrough

The exponential variance problem (0:01)

The lecture recaps the term "batch RL" as an older synonym for offline RL, then reviews importance-sampled policy gradients: multiplying the gradient by the ratio of trajectory probabilities under the target and behavior policies. It shows that this ratio reduces to a product of action probability ratios over the whole trajectory, and that because this product has O(T) terms, the resulting importance weights become degenerate for long horizons, making the estimator technically unbiased but practically useless due to exponential variance.

Splitting the weight and why the usual shortcut doesn't apply offline (4:02)

The lecture splits the importance weight into two parts: one accounting for the different probability of reaching a given state under the two policies, and one accounting for the different expected reward-to-go from that state. It notes that a common simplification in on-policy algorithms with multiple gradient steps, such as PPO, is to drop the first part, but explains this is only justified when the target policy stays close to the data-collecting policy, which defeats the purpose of offline RL, where the whole point is to find a substantially better policy.

The doubly robust estimator (9:04)

The lecture introduces the doubly robust estimator, first in the simpler bandit setting, as a way of subtracting an estimated Q-function from the importance-weighted reward and adding back its expected value under the policy, which acts like a baseline to reduce variance while remaining unbiased regardless of the estimate's accuracy. It then extends this recursively to the multi-step case, showing how it can be built up from a value-function estimate applied at each time step.

Marginalized importance sampling (18:12)

The lecture closes with marginalized importance sampling, which estimates importance weights as ratios of state or state-action marginal probabilities rather than products of per-step action probabilities, avoiding the exponential variance problem entirely. It explains the general recipe: write down a Bellman-like consistency condition that the true weights must satisfy, turn violations of that condition into an expected value under the behavior policy's data, and fit a neural network to satisfy it using only samples from the data set, without ever needing to explicitly model the state marginals themselves.

Before you watch

• Review the earlier lecture on importance-sampled policy gradients, since this lecture builds directly on that notation and derivation.
• Familiarity with baselines and control variates from the policy gradient lectures is useful for understanding the doubly robust estimator.
• Watch Lecture 15, Part 1 first for the definitions of offline RL, the behavior policy, and off-policy evaluation used throughout this part.

Check your understanding

1. Why does the importance-sampled policy gradient estimator have exponentially growing variance as the time horizon increases?
2. Why is it invalid, in the offline RL setting, to drop the part of the importance weight that accounts for reaching a given state?
3. How does the doubly robust estimator reduce variance while remaining unbiased, and what role does the estimated value function play?
4. What is the key advantage of marginalized importance sampling over standard importance sampling, and what kind of consistency condition does it rely on?