Lecture 16, Part 1: Policy Constraints and Implicit Q-Learning

What this lecture covers

This lecture moves from classic offline RL techniques to modern, deep offline RL methods that explicitly address distributional shift. It focuses on the widely studied family of policy constraint methods, which keep the learned policy close to the behavior policy that collected the data, and works through both their intuitive appeal and their practical shortcomings.

The lecture builds up from simple KL-divergence constraints, through explicit ways to enforce them, to an implicit constraint trick that avoids ever needing to know the behavior policy explicitly, yielding the advantage-weighted actor-critic (AWAC) algorithm. It then introduces a different idea based on expectile regression, which leads to implicit Q-learning (IQL), a method that estimates in-support optimal Q-values without ever querying actions outside the data set.

Key ideas

• Policy constraint methods: actor-critic style offline RL algorithms that update the Q-function under the current policy as usual, but constrain the policy update, for example by bounding the KL divergence between the learned policy and the behavior policy pi_beta.
• Too pessimistic, not pessimistic enough: a naive KL constraint can simultaneously fail to prevent errors (since low divergence doesn't guarantee low error for a maximizing policy) and be overly restrictive (for example, forcing the policy to stay random just because the behavior policy was random).
• Support constraints versus divergence constraints: a support constraint, which only allows actions that had reasonable probability under pi_beta, better matches the real goal than a KL constraint, but is much harder to implement exactly; the lecture notes maximum mean discrepancy as one practical approximation.
• Implicit policy constraint: solving the constrained optimization problem in closed form shows the optimal policy is pi_beta reweighted by the exponentiated advantage divided by a Lagrange multiplier, which can be implemented as weighted maximum likelihood ("weighted behavioral cloning") without ever needing to know pi_beta's functional form.
• Advantage-weighted actor-critic (AWAC): alternates a standard critic loss with an actor loss that fits the policy to data-set actions weighted by their exponentiated advantage, implementing the implicit constraint in practice.
• Expectile loss and implicit Q-learning (IQL): instead of estimating the mean of the target-value distribution (as mean-squared error does), an asymmetric expectile loss estimates an upper quantile of it, which corresponds to the best value supported by the data at each state, using only actions that actually appear in the data set and thus never triggering out-of-distribution overestimation.

Walkthrough

The distributional shift problem, recapped (0:01)

The lecture briefly recaps on-policy, off-policy, and offline RL, then restates why value-based offline methods suffer from distributional shift: the Q-function is accurate under the behavior policy pi_beta but is queried under the improving policy pi_nu, which is explicitly incentivized to find actions that inflate the Q-function's output, as shown again in the plots of wildly overestimated Q-values.

Policy constraint methods and their trade-offs (4:01)

The lecture introduces policy constraint methods, which bound the divergence between the learned policy and pi_beta, and walks through why this reasonable-sounding idea has real limitations in practice: it doesn't guarantee low error even for in-distribution actions, and a KL constraint specifically can be both too loose and too restrictive depending on the situation. It contrasts KL divergence constraints, which are easy to implement but not quite what is wanted, with support constraints, which better match the intuition but are much harder to enforce exactly, illustrated with a picture of a policy trying to concentrate on a high-value action while staying within its divergence budget.

Implicit constraints and advantage-weighted actor-critic (17:10)

Rather than explicitly estimating pi_beta for the constraint, the lecture derives the closed-form solution to the constrained optimization problem, showing the optimal policy reweights pi_beta by the exponentiated advantage over a Lagrange multiplier. This can be approximated using only samples from the data set with a weighted maximum-likelihood loss, leading to the advantage-weighted actor-critic (AWAC) or advantage-weighted regression algorithms, which alternate a standard critic update with this weighted behavioral-cloning actor update.

Expectile regression and implicit Q-learning (23:12)

The lecture identifies a remaining problem: even AWAC still needs to query the Q-function with actions sampled from the current policy when computing target values, which can be out of distribution during training. To avoid this entirely, it introduces the expectile loss, an asymmetric variant of squared error that estimates an upper quantile rather than the mean of the target-value distribution at each state, using only the states and actions that appear in the data set. This yields implicit Q-learning (IQL), which trains Q- and value-functions purely from in-data transitions and only extracts an explicit policy as a final, separate step using the advantage-weighted approach from before.

Before you watch

• Watch Lecture 15, Parts 1 through 3 first, since this lecture assumes familiarity with the offline RL setup, distributional shift, and the classic value-based methods already covered.
• Review actor-critic methods and Lagrangian duality from earlier lectures, since both are used directly in deriving the implicit constraint.

Check your understanding

1. Why can a KL-divergence policy constraint be simultaneously too permissive and too restrictive?
2. How does the implicit constraint trick avoid the need to explicitly estimate the behavior policy pi_beta?
3. What is the key difference between a mean-squared error loss and an expectile loss when fitting a value function, and why does this matter for avoiding out-of-distribution actions?
4. Why does AWAC still risk querying out-of-distribution actions even though it implements the constraint implicitly, and how does implicit Q-learning address this?