Lecture 14: Deep Reinforcement Learning

What this lecture covers

The lecture moves from supervised and unsupervised learning, covered earlier in the course, into a third problem setup: an agent that takes actions in an environment and learns from rewards rather than labels. It formalizes this as a Markov decision process (MDP) and builds up the two main families of deep reinforcement learning algorithms, Q-learning and policy gradients, showing how each one is derived, trained, and where it succeeds or struggles.

By the end, you can describe an RL problem as states, actions, and rewards, explain the Bellman equation and why plain Q-learning does not scale, and follow how deep Q-networks, policy gradient methods (REINFORCE), and actor-critic combine the two. The lecture closes with two applied examples, a recurrent visual attention model and DeepMind's AlphaGo, that show these ideas used in vision and game-playing systems.

Key ideas

• Markov decision process (MDP): a tuple of states, actions, a reward distribution, a transition probability distribution, and a discount factor, used to formalize the agent-environment loop.
• Policy: a function from states to actions; the objective is to find the policy that maximizes expected cumulative discounted reward.
• Value function and Q-value function: the value function estimates expected future reward from a state under a policy; the Q-value function estimates it for a state-action pair.
• Bellman equation: the optimal Q-value of a state-action pair equals the immediate reward plus the discounted value of the best action in the next state.
• Deep Q-learning: approximates the Q-function with a neural network trained to satisfy the Bellman equation, made stable in practice with experience replay.
• Policy gradients (REINFORCE): directly optimize policy parameters with gradient ascent on expected reward, using a Monte Carlo estimator that avoids needing transition probabilities.
• Variance reduction: techniques such as using future rewards only, discounting, and subtracting a baseline (including a learned value function) reduce the high variance of policy gradient estimates.
• Actor-critic: combines a policy network (actor) with a learned Q or value function (critic) that scores the actor's actions using the advantage function.

Walkthrough

The reinforcement learning problem (1:09)

The lecture contrasts RL with supervised and unsupervised learning: here an agent receives a state, takes an action, and gets back a reward and a new state, repeating until a terminal state ends the episode. It walks through examples including cart-pole, robot locomotion, Atari games, and Go, identifying the state, action, and reward for each.

Markov Decision Process (6:15)

The MDP formalism is introduced as a tuple of states, actions, a reward distribution, a transition distribution, and a discount factor. The lecture works through the agent-environment loop mathematically and defines a policy as a mapping from states to actions, with the goal of finding the policy that maximizes cumulative discounted reward.

Grid World, value functions and Q-learning (8:19)

A simple Grid World example illustrates random versus optimal policies. The lecture then defines the value function and Q-value function, derives the Bellman equation for the optimal Q-function, and explains why naive value iteration over all state-action pairs is computationally infeasible for large state spaces such as raw game pixels.

Deep Q-learning with experience replay (15:28)

The fix is to approximate the Q-function with a neural network trained to reduce the error against the Bellman target, which is deep Q-learning. Applied to Atari, the network takes a stack of the last four preprocessed frames and outputs one Q-value per action in a single forward pass. Experience replay, storing transitions and training on random mini-batches instead of consecutive ones, is introduced to break correlations between samples and improve data efficiency. The full deep Q-learning with experience replay algorithm is assembled step by step, and a video shows a DQN agent learning to play Breakout over the course of training.

Policy gradients and REINFORCE (28:38)

When the Q-function is too complex to learn but the policy is simpler, the lecture introduces policy gradients: parametrize the policy directly and take gradient ascent on expected reward. It derives the REINFORCE gradient estimator using the log-derivative trick to avoid differentiating through unknown transition probabilities, and explains the intuition of pushing up the probability of actions from high-reward trajectories and down for low-reward ones.

Reducing variance and actor-critic (36:48)

Because the basic REINFORCE estimator has high variance, the lecture presents variance-reduction tricks: scaling by future reward only, discounting, and subtracting a baseline. It connects a good baseline to the value function, arrives at the advantage function Q(s,a) - V(s), and combines policy gradients with a learned critic in the actor-critic algorithm, which can reuse Q-learning tricks like experience replay.

Applications: recurrent attention and AlphaGo (46:55)

Two case studies close the lecture. The recurrent attention model uses policy gradients to choose where to glimpse next in an image for classification, since choosing glimpse locations is non-differentiable; it has also been applied to fine-grained recognition, captioning, and visual question answering. AlphaGo is presented as combining supervised pretraining on professional games, self-play policy gradient training, a learned value network, and Monte Carlo tree search to select moves.

Before you watch

• Review supervised and unsupervised learning from earlier lectures, since this lecture defines RL by contrast with both.
• Be comfortable with gradient descent and neural network training, as Q-learning and policy gradients both reduce to gradient-based optimization.
• Basic familiarity with probability distributions and expectations helps with the Bellman equation and the REINFORCE derivation.

Check your understanding

1. What are the components of a Markov decision process, and what does each one represent?
2. How does the Bellman equation relate the optimal Q-value of a state-action pair to the value of the next state?
3. Why does plain Q-learning become infeasible for large state spaces, and how does deep Q-learning with experience replay address this?
4. What problem does the log-derivative trick solve in deriving the policy gradient estimator?
5. Why does the actor-critic algorithm need both a policy network and a critic, and how does the advantage function fit in?