Lecture 7, Part 1: From Actor-Critic to Policy Iteration

What this lecture covers

Opening Berkeley CS285's lecture on value function methods, this part asks whether the explicit policy network used in actor-critic is even necessary. It shows that taking the action that maximizes the advantage function at every state produces a policy at least as good as the one that advantage was computed for, no matter how bad the original policy was.

You come away understanding the idea behind policy iteration, an algorithm that alternates between evaluating a value function and setting a new implicit policy via an argmax, and how this reduces to a dynamic-programming procedure in the special case of a small, tabular MDP with known transition probabilities.

Key ideas

• Implicit policy via argmax: taking argmax_a A^pi(s,a) gives an action at least as good as one sampled from pi, for any policy pi, which means acting greedily on the advantage improves the policy without needing an explicit policy network.
• Policy iteration: alternates between evaluating the advantage (or value function) of the current policy and constructing a new deterministic policy that assigns probability 1 to the argmax action.
• Tabular setting: when the state and action spaces are small and discrete and the transition probabilities are known, the value function can be stored explicitly as a table rather than approximated with a neural network.
• Policy evaluation by dynamic programming: repeatedly applying the Bellman backup V(s) = r(s, pi(s)) + gamma * E[V(s')] across the whole table converges to the true value function V^pi.
• Value iteration: a simplification that skips explicit policy representation by building a table of Q values (Q(s,a) = r(s,a) + gamma * E[V(s')]) and then setting V(s) to the max over actions in each row, which is equivalent to the argmax-based update.

Before you watch

• Watch the actor-critic lecture (Lecture 6) in full, since this lecture opens by directly extending its batch actor-critic algorithm.
• Be comfortable with the definitions of the advantage, value, and Q functions from that lecture.

Check your understanding

1. Why does argmax_a A^pi(s,a) always produce an action at least as good as sampling from pi, regardless of how good pi is?
2. What are the two steps of policy iteration, and what does each one compute?
3. Why does the tabular setting allow the value function to be stored exactly rather than approximated?
4. How does value iteration simplify policy iteration by working with the Q function instead of alternating value and policy updates?