Lecture 19, Control as Inference, Part 4

What this lecture covers

This closing part of the control-as-inference lecture instantiates the soft-optimality framework as two concrete algorithms: soft Q-learning, which replaces the hard max in the Q-learning target with a soft max, and entropy-regularized policy gradient, which adds an entropy bonus to the standard policy gradient objective. It also shows that the two are closely related once the policy is written as the exponentiated soft advantage.

Key ideas

• Soft Q-learning: identical to standard Q-learning except the target value uses a soft max (log-sum-exp of next Q-values) instead of a hard max over actions; the resulting policy is the exponentiated advantage rather than the greedy action.
• Entropy-regularized policy gradient: adds the policy's entropy to the expected-reward objective; its gradient equals the standard policy gradient with r - log pi used as the reward, and the extra term from differentiating log pi reduces to a constant that acts as a baseline and can be ignored.
• Connection to soft Q-learning: substituting log pi = Q - V into the entropy-regularized policy gradient produces an expression close to the soft Q-learning update, differing mainly by an off-policy correction term.
• Practical benefits: entropy regularization prevents premature collapse of policy stochasticity (which would otherwise harm exploration), gives a principled way to break ties between equally good actions, improves robustness by covering multiple ways to solve a task, and makes policies easier to fine-tune for related tasks.
• Recovering hard optimality: as reward magnitude increases, or as a temperature parameter is driven toward zero, the soft-optimal policy approaches the standard deterministic optimal policy.

Before you watch

• Watch parts 1 through 3 of this lecture, which derive the soft value iteration and the entropy-augmented RL objective that these algorithms optimize.
• Review standard Q-learning and policy gradient from earlier lectures in this course, since both algorithms here are direct modifications of them.

Check your understanding

1. What is the only change needed to turn standard Q-learning into soft Q-learning?
2. Why does adding an entropy term to the policy gradient objective end up looking like subtracting log pi from the reward, with no other change to the gradient expression?
3. What practical problem does entropy regularization solve for on-policy policy gradient methods?
4. Under what conditions does the soft-optimal policy reduce back to the standard, deterministic-optimal policy?