Lecture 6, Part 3: Implementing Actor-Critic

What this lecture covers

This part turns the actor-critic algorithm from earlier parts into something that works in practice. It first compares two neural network designs, separate actor and critic networks versus a shared trunk with two heads, then addresses the fact that fully online actor-critic updates on a single transition have too much variance for stochastic gradient descent to work well.

You come away understanding why batching matters, how synchronous and asynchronous parallel actor-critic get larger batches, and how switching to a replay buffer forces the algorithm to learn a Q function instead of a value function so it can handle transitions from older policies.

Key ideas

• Separate networks: simplest to implement and more stable to train, at the cost of not sharing learned features between actor and critic.
• Shared network with two heads: can be more sample-efficient when representations transfer, such as with image inputs, but harder to stabilize because the actor and critic gradients differ in scale.
• Synchronous parallel actor-critic: multiple simulator workers each take one step, and the resulting transitions form a batch used for a synchronized update.
• Asynchronous parallel actor-critic: workers update without waiting for each other, introducing slight bias from using parameters that are marginally older than the latest ones, which in practice tends to be an acceptable trade-off.
• Off-policy actor-critic with a replay buffer: instead of running multiple threads, transitions from many past policies are stored and sampled to form a batch, but naive reuse of old actions breaks both the value target and the policy gradient.
• Fix 1, learn Q instead of V: because old actions in the buffer are valid inputs to a Q function (unlike a V function that assumes on-policy actions), the critic is switched to Q(s,a), and target values are computed by querying the latest policy for the action it would take at the next state.
• Fix 2, resample actions for the policy gradient: the policy gradient uses an action sampled from the current policy at the buffer's state, not the action originally stored, which keeps the gradient estimate unbiased with respect to pi_theta.
• Remaining bias: the states themselves still come from older policies' visitation distributions, which is accepted as a source of bias that tends to be benign in practice.

Before you watch

• Watch Parts 1 and 2 of this lecture, since they introduce the value function, advantage estimate, and discounting used throughout.
• Recall the Q function definition from earlier in the course, since this part relies on it to fix the off-policy update.

Check your understanding

1. What is the trade-off between separate actor/critic networks and a shared-trunk design?
2. Why does a fully online, single-sample actor-critic update perform poorly with deep neural networks?
3. What is the key difference between synchronous and asynchronous parallel actor-critic?
4. Why does off-policy actor-critic need to learn a Q function instead of a value function?
5. How does resampling the action at the buffer's state keep the policy gradient estimator unbiased?