Lecture 6, Part 5: Actor-Critic Summary and Examples

What this lecture covers

This short closing part of the actor-critic lecture reviews the full set of ideas covered across the previous parts: the actor-critic algorithm, discount factors, network design choices, and the various baseline and n-step tricks used to control variance. It then points to specific papers that put these ideas into practice.

Key ideas

• Actor-critic recap: the actor is the policy, the critic is the value function, and the algorithm follows the sample, evaluate, and gradient-update structure with substantially lower variance than plain policy gradients.
• Discount factor: makes policy evaluation feasible over infinite horizons and can be interpreted either as a preference for near-term reward or as a variance-reduction trick.
• Architecture and batching choices: shared versus separate actor/critic networks, and batch versus online updates, including using parallelism to get larger effective batch sizes.
• Baselines and control variates: state-dependent baselines keep the gradient unbiased, and action-dependent control variates can reduce variance further at the cost of needing a correction term.
• GAE paper: "High-Dimensional Continuous Control with Generalized Advantage Estimation" applies the GAE trick in a batch-mode actor-critic setting on continuous control tasks such as a running humanoid.
• A3C paper: "Asynchronous Methods for Deep Reinforcement Learning" uses online, asynchronous, parallel actor-critic with image inputs, an n-step return with n=4, and a shared network for actor and critic.
• Foundational reading: the classic "Policy Gradient Methods for Reinforcement Learning with Function Approximation" paper underlies much of the lecture's material, including the causality trick.

Before you watch

• Watch Parts 1 through 4 of this lecture, since this part is a review and reference list rather than new material.

Check your understanding

1. What are the three boxes in the general recipe for actor-critic algorithms, and what does each one do in this setting?
2. How does the asynchronous methods (A3C) paper's use of n-step returns and a shared network relate to the design choices discussed earlier in the lecture?
3. What does the GAE paper's estimator combine to control bias and variance?