Lecture 19, Control as Inference, Part 1

What this lecture covers

This lecture opens a new topic: reframing optimal control and reinforcement learning as probabilistic inference. It starts from an empirical observation about behavior, that humans and animals reaching a goal usually get there but rarely take the exact optimal trajectory, and asks whether a better model of "near optimal" behavior than the standard deterministic-optimal-policy framework exists.

By the end, you should understand why standard RL formulations cannot explain stochastic behavior (since fully observed MDPs admit deterministic optimal policies), how adding binary "optimality" variables to a graphical model produces trajectory probabilities that favor high reward but assign non-zero probability to any trajectory, and why this view sets up both better exploration and, in the next lecture, inverse reinforcement learning.

Key ideas

• Near-optimal, not perfectly optimal, behavior: animals and humans tend to reach a goal reliably but vary in exactly how, especially on aspects of the task that barely affect the outcome.
• Deterministic optimal policies: for any fully observed MDP with a reward linear in the state-action marginal, there exists a deterministic optimal policy, so standard RL cannot naturally explain randomized behavior.
• Optimality variable: a binary variable O_t added to the graphical model at each timestep, with p(O_t = true | s_t, a_t) = exp(r(s_t, a_t)), requiring rewards to be non-positive (achievable without loss of generality by shifting the reward).
• Trajectory probability under optimality: conditioning on all O_t being true gives p(tau | O_{1:T}) ∝ p(tau) * exp(sum of rewards along tau), so higher-reward trajectories are more likely but lower-reward ones remain possible with exponentially decreasing probability.
• Explaining stochastic behavior: this model naturally explains why an agent might behave randomly among near-equally-good options while strongly avoiding clearly bad ones.
• Motivations beyond modeling animals: this framework helps explain suboptimal demonstrator behavior for inverse reinforcement learning, enables solving control problems with inference algorithms, and helps explain why stochastic policies can aid exploration and transfer.
• Backward and forward messages: inference in this chain-structured graphical model uses backward messages (probability of future optimality given the current state and action, from which the policy can be recovered) and forward messages (probability of reaching a state given past optimality), both to be derived in the next part.

Before you watch

• Review the definition of an MDP and the fact that optimal policies in fully observed MDPs can be taken to be deterministic, covered earlier in the course.
• Some familiarity with graphical models, message passing, or hidden Markov models is helpful since the inference approach parallels HMM-style forward-backward algorithms.

Check your understanding

1. Why can't a standard, fully-observed MDP with a deterministic optimal policy explain the kind of variable, near-optimal behavior seen in animals?
2. What role does the constraint that rewards must be negative play in defining p(O_t = true | s_t, a_t), and why is this not a real limitation?
3. Under the optimality-variable model, why does a trajectory with much lower reward become exponentially less likely rather than simply impossible?
4. What is the practical difference between a backward message and a forward message in this graphical model?