Lecture 8, Part 5: Q-Learning with Continuous Actions

What this lecture covers

Q-learning as discussed so far assumes discrete actions, where the max over actions needed for the policy and for target values is just an exhaustive evaluation. This part asks how to perform that same max when actions are continuous, since it appears both in action selection and, more critically, in the inner-loop computation of target values.

The lecture presents three families of solutions and ends with the full pseudocode for DDPG, a continuous-action Q-learning algorithm. This extends the target-network and double Q-learning material from earlier in Lecture 8 to continuous control problems, which recur throughout the rest of the course.

Key ideas

• The core difficulty: an exact argmax over a continuous action space has no closed form, and it must be computed cheaply because it sits inside the training loop.
• Option 1, stochastic optimization: approximate the max by sampling a set of candidate actions (or using CEM/CMA-ES for a more refined search) and picking the best one; simple, easy to parallelize, less accurate in high dimensions.
• Option 2, easy-to-optimize function classes: design the Q-function so the action-dependent part is quadratic, as in NAF (normalized advantage function), giving a closed-form optimum at the cost of representational power.
• Option 3, learned maximizer: train a second network mu_theta(s) to approximate the argmax of Q_phi, updated by backpropagating the Q-function's gradient through mu_theta via the chain rule.
• DDPG: combines option 3 with target networks for both Q_phi' and mu_theta', alternating a Q-function update with an update to mu_theta that maximizes Q_phi(s, mu_theta(s)).
• Related algorithms: DDPG is closely related to the earlier NFQCA algorithm and can also be viewed as a deterministic actor-critic method; later variants include TD3 and SAC.

Walkthrough

The continuous action problem (1:56)

With discrete actions the max is a simple enumeration; with continuous actions it becomes an optimization problem that must run efficiently inside the training loop, both for the policy's action selection and, especially, for the target-value computation.

Option 1: random sampling and stochastic optimization (4:20)

The simplest approach samples a batch of candidate actions and takes the one with the highest Q-value; it's inexact but fast and trivially parallelized, and its imprecision can even reduce overestimation. More accurate options include the cross-entropy method (CEM), which iteratively refines the sampling distribution toward good regions, and CMA-ES, a more elaborate variant; both work reasonably well up to roughly 40-dimensional action spaces.

Option 2: normalized advantage functions (6:14)

Instead of approximating the max, this approach restricts the Q-function's shape so the max has a closed form: NAF has a network output a bias, a vector, and a positive-definite matrix that define a function quadratic in the action for each state. The maximizing action is then simply the vector output, and the algorithm otherwise runs unchanged, at the cost of being unable to represent Q-functions that are not quadratic in the action.

Option 3: learning an approximate maximizer (DDPG) (9:34)

A second network mu_theta(s) is trained to output the action that approximately maximizes Q_phi(s, a), found by pushing gradients through the Q-function and back through mu_theta using the chain rule. Target values then use mu_theta' in place of an explicit argmax: y = r + gamma * Q_phi'(s', mu_theta'(s')). The full algorithm mirrors standard Q-learning, adding a gradient update for theta and target-network updates (for example via Polyak averaging) for both phi' and theta'. This is essentially the DDPG algorithm, closely related to the earlier NFQCA method, with more recent successors including TD3 and SAC.

Before you watch

• Be familiar with target networks and the target-value computation from earlier in Lecture 8, since all three options here modify how that max is computed.
• Knowing the actor-critic framework helps, since DDPG can also be understood as a deterministic actor-critic method.

Check your understanding

1. Why is computing the max over actions harder for continuous action spaces than discrete ones?
2. What is the trade-off between random-sampling approximation and CEM/CMA-ES for approximating the max?
3. How does the normalized advantage function (NAF) get a closed-form max, and what does it give up to do so?
4. In DDPG, what role does the network mu_theta(s) play, and how is it trained?