Lecture 7, Part 3: Q-Learning and Exploration

What this lecture covers

This part explains why fitted Q-iteration counts as an off-policy algorithm: the policy only enters through the max inside the Q-function target, which acts like a simulated lookup rather than requiring a new rollout, so old transitions stay valid for training. It also clarifies what fitted Q-iteration actually optimizes, the Bellman error, and what that error implies about policy quality.

The lecture then derives online Q-learning as a special case of fitted Q-iteration with a single transition per update, and addresses a practical problem: using a purely greedy policy to collect data can get the agent stuck taking a bad action forever. This motivates exploration strategies, in particular epsilon-greedy and Boltzmann (softmax) exploration.

Key ideas

• Why fitted Q-iteration is off-policy: the current policy only appears as an argument to the Q function inside a max, so changing the policy does not require generating new rollouts, since the transition (s,a,s') does not depend on it.
• Bellman error: fitted Q-iteration minimizes the difference between Q_phi(s,a) and target values y; zero error implies the optimal Q function and policy, but nonzero error gives no guarantees about policy quality.
• Online Q-learning: a special case of fitted Q-iteration using one transition at a time: take an action, compute one target value, and take one gradient step on the temporal difference error Q_phi(s,a) - y.
• Temporal difference error: the difference between a Q function's current prediction and its bootstrapped target value.
• Problem with a purely greedy policy during learning: because the argmax policy is deterministic, a poor initial Q function can cause the agent to repeatedly take the same suboptimal action and never discover better ones.
• Epsilon-greedy exploration: take the greedy action with probability 1 - epsilon and a uniformly random other action with probability epsilon, often decreasing epsilon over training as the Q function improves.
• Boltzmann (softmax) exploration: choose actions in proportion to a positive transformation (typically exponentiation) of their Q values, so similarly good actions get similar probability and very bad actions are rarely chosen.

Before you watch

• Watch Part 2 of this lecture on fitted Q-iteration, since this part builds directly on its algorithm and target value computation.

Check your understanding

1. Why doesn't fitted Q-iteration need new rollouts when the policy changes?
2. What does zero Bellman error imply about the resulting policy, and what can be said when the error is not zero?
3. How is online Q-learning derived as a special case of fitted Q-iteration?
4. Why can using a purely greedy policy during learning prevent the agent from discovering better actions?
5. How does Boltzmann exploration differ from epsilon-greedy in how it treats two similarly good actions?