Lecture 19, Control as Inference, Part 2

What this lecture covers

Building on the optimality-variable graphical model from part 1, this lecture derives the three inference operations needed to make the framework useful: backward messages, the resulting near-optimal policy, and forward messages. It shows that the backward recursion is a "soft" version of value iteration, using log-sum-exp (a soft max) in place of the hard max, and that the near-optimal policy turns out to be a Boltzmann distribution over a soft advantage function.

By the end, you should be able to state the backward-message recursion, explain how it reduces to the ordinary Bellman backup in deterministic environments, describe why an action prior can be folded into the reward without loss of generality, and explain how backward and forward messages combine to give the state marginal under near-optimal behavior.

Key ideas

• Backward message recursion: beta_t(s_t,a_t) = p(O_t|s_t,a_t) * E[beta_{t+1}(s_{t+1})], computed backward from the end of the trajectory, with beta_T initialized from the last reward.
• Soft value and Q functions: defining V_t = log(beta_t(s_t)) and Q_t = log(beta_t(s_t,a_t)) turns the recursion into V_t(s_t) = log(integral of exp(Q_t(s_t,a_t))), a soft max, and Q_t(s_t,a_t) = r(s_t,a_t) + log E[exp(V_{t+1}(s_{t+1}))], a soft Bellman backup.
• Deterministic transitions recover the Bellman equation: when the next state is a deterministic function of the current state and action, the soft backup collapses exactly to the standard Bellman backup, but with soft max instead of hard max.
• Optimism problem under stochastic transitions: the soft backup's log-sum-exp over next states is dominated by the highest-value outcome, which makes the model act as though it can choose to get lucky rather than merely tolerate luck.
• Action prior: an assumed-uniform prior over actions given a state can, if actually non-uniform, be folded into the reward as log p(a|s) without loss of generality, so uniformity can be assumed without losing expressiveness.
• Recovered policy: pi(a_t|s_t) = exp(Q_t(s_t,a_t) - V_t(s_t)), a Boltzmann distribution over the soft advantage; a temperature parameter can interpolate between this soft policy and the deterministic greedy policy.
• Forward messages and state marginals: forward messages (probability of reaching a state given past optimality) are derived with a similar recursive, Bayes'-rule argument, and combined with backward messages give p(s_t | O_{1:T}) ∝ beta_t(s_t) * (forward message), matching the "two cones intersecting" intuition and observed patterns in human reaching behavior.

Before you watch

• Watch part 1 of this lecture, which sets up the optimality-variable graphical model this part performs inference on.
• Review value iteration and the Bellman backup, and basic Bayes' rule manipulations, since the derivations rely heavily on both.

Check your understanding

1. Why does the backward message recursion reduce exactly to the standard Bellman backup only in the deterministic-transition case?
2. What is the "optimism problem" that arises when applying this soft backup to stochastic transitions, and why does it happen?
3. Why can a non-uniform action prior always be absorbed into the reward function instead of being modeled separately?
4. How do backward and forward messages combine to produce the state marginal p(s_t | O_{1:T}), and what intuitive picture does the lecture give for this (the two "cones")?