Lecture 16, Part 2: Conservative Q-Learning (CQL)

What this lecture covers

This short segment introduces a different family of offline RL methods that address distributional shift by directly repairing the Q-function rather than constraining the actor, as the previous part's policy-constraint methods did. The central idea is conservative Q-learning (CQL): add terms to the Q-function's training objective that explicitly push down Q-values for actions the policy would try to exploit, while pushing up Q-values for actions actually seen in the data set.

The lecture explains the intuition behind this combined objective, why it produces a feedback effect that pulls the policy's preferred actions back toward the data distribution, and how to implement it efficiently for both discrete and continuous action spaces without needing to explicitly construct the adversarial action distribution.

Key ideas

• Direct Q-value repair: instead of constraining the policy, add a term to the critic's loss that finds actions with high (likely overestimated) Q-values and explicitly minimizes them, addressing the same erroneous peaks that policy constraint methods try to avoid indirectly.
• Lower bound guarantee: with an appropriate weighting alpha, pushing down Q-values this way can be shown to produce a Q-function that lower-bounds the true Q-function, though this alone tends to be overly pessimistic.
• Balancing push-down and push-up terms: adding a second term that pushes up on Q-values for actions actually in the data set causes the two terms to cancel out when the policy already favors in-distribution actions, while still suppressing genuinely out-of-distribution overestimation, and it remains a valid lower bound in expectation over the policy rather than pointwise everywhere.
• Maximum entropy regularization trick: choosing the regularizer on the "push down" distribution to be its entropy makes the optimal push-down distribution proportional to the exponentiated Q-values, which means its expected value reduces to a convenient log-sum-exp of the Q-values for discrete actions.
• Practical implementation: for discrete actions, the extra CQL terms can be computed directly via log-sum-exp; for continuous actions, they can be estimated with single-step importance sampling, which avoids the exponential variance problems seen in multi-step importance sampling because it only reweights over one time step.

Walkthrough

Repairing the Q-function directly (0:01)

The lecture reintroduces the picture of a fitted Q-function whose maximum coincides with its largest positive error, and proposes a direct fix: add a term to the critic loss that selects actions with high Q-values under a distribution mu and explicitly minimizes their value, which can be shown to yield a Q-function that lower-bounds the truth when the weighting alpha is chosen appropriately.

Balancing overestimation against pessimism (1:01)

The lecture points out that pushing down all high Q-values indiscriminately makes the method too pessimistic to ever recover the correct Q-function, and proposes adding a second term that pushes up on Q-values for actions actually present in the data set. It explains the resulting feedback dynamic: if high Q-values are mostly for in-data actions, the two terms roughly cancel; if they are for out-of-distribution actions, the push-down and push-up terms combine to pull the policy's preferred actions back toward the data over training. This combined objective is referred to as CQL, still only guaranteed to be a lower bound in expectation over the policy rather than pointwise.

Implementing CQL in practice (4:05)

The lecture explains how to implement the push-down term efficiently without explicitly constructing mu: adding an entropy regularizer to mu makes its optimal form proportional to the exponentiated Q-values, and its expected Q-value collapses to a log-sum-exp over discrete actions, which can be computed directly. For continuous actions, the same idea is estimated using importance sampling over sampled actions reweighted by exponentiated Q-values, which works reasonably well because, unlike multi-step importance sampling, it only involves a single time step and so does not suffer from exponential variance.

Before you watch

• Watch Lecture 16, Part 1 first, since this segment explicitly contrasts CQL with the policy-constraint methods (AWAC, implicit Q-learning) covered there.
• Review the distributional shift and out-of-distribution action problems from Lecture 15, since CQL is presented as a direct solution to the same issue.

Check your understanding

1. How does conservative Q-learning differ from policy constraint methods in how it addresses distributional shift?
2. Why is pushing down all high Q-values by itself too pessimistic, and how does adding a push-up term for in-data actions fix this?
3. What role does the entropy regularizer on mu play in making CQL practical to implement?
4. Why does the single-step importance sampling used for continuous actions in CQL avoid the exponential variance problems seen in multi-step importance sampling?