Lecture 6, Part 1: Actor-Critic and Value Functions

What this lecture covers

This part opens Berkeley CS285's lecture on actor-critic algorithms by asking how to lower the variance of the policy gradient estimator introduced in the previous lecture. It recaps the REINFORCE algorithm and its reward-to-go term, then shows that this term is a high-variance single-sample estimate of an expectation that a learned function can approximate more accurately.

By the end you should understand why the advantage function is the natural quantity to multiply into the policy gradient, why fitting the value function alone is more convenient than fitting the Q function, and how Monte Carlo and bootstrapped targets differ when training that value function.

Key ideas

• Reward-to-go (q_hat): the sum of rewards from a time step onward in one sampled trajectory; it is an unbiased but high-variance estimate of the true expected future reward.
• Advantage function: A(s,a) = Q(s,a) - V(s), measuring how much better an action is than the policy's average behavior in that state.
• Why fit V instead of Q: Q(s,a) can be rewritten as the current reward plus the expected next-state value, so approximating the next state with the one actually observed lets the advantage be expressed using only V, which depends on fewer inputs and is easier to learn.
• Policy evaluation: the process of estimating V^pi, the value of the current policy at every state; the RL objective itself is the expected value of V^pi over the initial states.
• Monte Carlo target: summing actual observed rewards after a state gives a target for fitting V; it stays low bias but only reduces variance through the function approximator's generalization across similar states.
• Bootstrapped target: using the observed reward plus the current value estimate of the next state gives a lower-variance but more biased target.
• Bias-variance trade-off: approximate value functions introduce some bias into the policy gradient, but this is usually worth it for the large drop in variance.

Before you watch

• Review the REINFORCE algorithm and the "orange/green/blue box" anatomy of policy gradient methods from the previous lecture, since this lecture modifies the green box.
• Be comfortable with the baseline trick for policy gradients, since the advantage function extends that idea to a state-dependent baseline.

Check your understanding

1. Why does using a single sampled trajectory to estimate the reward-to-go lead to high variance?
2. Derive how the advantage function follows from subtracting the value function from the Q function as a baseline.
3. Why is it more convenient to fit V(s) than Q(s,a) or A(s,a) directly?
4. Compare the Monte Carlo and bootstrapped targets for fitting a value function: what does each trade off?
5. How does policy evaluation relate to computing the RL objective J(theta)?