Lecture 19: Control as Inference, Part 5

What this lecture covers

This closing part of the control-as-inference lecture connects the variational inference framework built earlier in the course to real algorithms and results. It asks why running the same deep RL algorithm twice can produce very different, sometimes worse, behavior, and shows how treating the policy as a distribution proportional to exponentiated Q-values fixes this by keeping multiple good strategies alive instead of collapsing onto one.

By the end, you can explain how soft Q-learning and soft actor-critic follow from the soft optimality model, why entropy-seeking pretraining speeds up fine-tuning, and where to find the original papers behind these methods.

Key ideas

• Local optimum problem in RL: two runs of the same algorithm can commit to different, unequally good solutions because early random advantages get amplified during learning.
• Soft Q-learning: choosing actions proportional to the exponentiated Q-value keeps probability mass on multiple promising strategies instead of committing early to one.
• Entropy-seeking pretraining: training with a broad reward (for example, run fast in any direction) under soft Q-learning produces policies that later specialize faster than deterministic algorithms like DDPG, which must unlearn a wrong direction first.
• Soft actor-critic (SAC): the actor-critic counterpart of soft Q-learning; its Q-update looks like ordinary actor-critic but subtracts a -log pi entropy term, and its policy update fits the variational distribution to the Q-function.
• Message passing view: updating the Q-function corresponds to inference in the graphical model, while updating the policy corresponds to fitting the variational approximation to the posterior.
• Robustness from entropy: because SAC learns many ways to solve a task, resulting policies (a robotic arm stacking Lego, a legged robot walking) tolerate physical perturbations better than policies trained without entropy.

Before you watch

• Be familiar with the variational inference and soft optimality framework from the earlier parts of this control-as-inference lecture.
• Know the basics of Q-learning and actor-critic methods, since both are extended here with an entropy term.

Check your understanding

1. Why can two runs of the same deep RL algorithm converge to noticeably different behaviors?
2. How does choosing actions proportional to the exponentiated Q-value change what the agent explores?
3. Why does a policy pretrained with soft Q-learning fine-tune faster than one pretrained with DDPG?
4. What term is added to the standard actor-critic Q-update in soft actor-critic, and what does it account for?