Seyed Masoud Hosseini · Overview · Study log · Weekly summaries · Ideas · Search · Transcript · RSS feed

Deep Reinforcement Learning · Lecture 10 of 99 · 7:23

Lecture 4: Introduction to RL Algorithms, Part 2

Study guide

What this lecture covers

This short lecture describes the high-level structure common to nearly all reinforcement learning algorithms covered in the course: a loop of generating samples, estimating something about the current policy, and improving the policy, and discusses which of these steps tend to be computationally expensive.

After watching, you should be able to name the three boxes in the RL algorithm anatomy, give an example of each for a simple policy gradient method and a model-based method, and explain what makes sample generation cheap or expensive.

Key ideas

Three-part anatomy: every RL algorithm in the course generates samples (orange box) by running the policy in the environment, fits a model or estimate (green box) of how well the policy is doing, and improves the policy (blue box) — then repeats.
Policy gradient example: the green box simply sums rewards along sampled trajectories; the blue box computes a policy gradient and takes a gradient step, making good trajectories more likely and bad ones less likely.
Model-based example ("RL by backprop"): the green box trains a full dynamics model f_phi(s_t, a_t) ≈ s_{t+1} via supervised learning; the blue box backpropagates through the model and reward to optimize the policy directly.
Cost of the orange box (sampling): collecting trajectories from a real-world system (robot, car, power grid) can be extremely expensive and slow, while simulators like MuJoCo can run at up to 10,000x real time, making sampling nearly free.
Cost of the green and blue boxes varies too: summing rewards or taking a single gradient step is cheap; training an entire dynamics model or backpropagating through a model and policy can be expensive.
Different algorithms emphasize different boxes: for example, Q-learning (covered later) concentrates most computation in the green box, with a cheap argmax as its blue box.

Before you watch

Watch Lecture 4 Part 1 first for the MDP and RL objective definitions this framework builds on.

Check your understanding

What are the three boxes in the shared anatomy of RL algorithms, and what does each one do?
How does the green box differ between a simple policy gradient method and a model-based "RL by backprop" method?
Why might the cost of the orange (sampling) box vary so much between a real robot and a MuJoCo simulator?
In what sense does Q-learning concentrate its effort mostly in the green box?

Vocabulary

anatomy (noun): The basic structure or parts that make up something.
This lecture describes the anatomy of an RL algorithm.
generate samples (phrase): To collect data by running a policy and observing outcomes.
The orange box generates samples from the environment.
fit (verb): To adjust a model so it matches observed data well.
The green box fits an estimate of how good the policy is.
policy gradient (noun): A method that directly computes how to change a policy's parameters to increase reward.
Policy gradient methods take a gradient step to improve the policy.
dynamics model (noun): A model that predicts how the environment's state changes given an action.
A dynamics model predicts the next state from the current one.
backpropagate (verb): To send gradients backward through a computation to compute how to improve it.
The method backpropagates through the model to optimize the policy.
simulator (noun): Software that mimics a real environment for testing or training.
A simulator can run much faster than real time.
real time (noun): The actual speed at which events happen in the physical world.
Some simulators run thousands of times faster than real time.
Q-learning (noun): A reinforcement learning method that learns the value of taking each action in each state.
Q-learning concentrates most of its computation in the green box.
argmax (noun): The input value that produces the largest output of a function.
The policy picks the argmax action from the Q-function.
computationally expensive (phrase): Requiring a lot of processing time or resources.
Training a full dynamics model can be computationally expensive.
loop (noun): A repeating sequence of steps.
RL algorithms repeat a loop of sampling, fitting, and improving.
improve (verb): To make something better through a change.
The blue box improves the policy using the fitted estimate.
estimate (noun): An approximate calculation or judgment of a value.
The green box produces an estimate of the policy's performance.
supervised learning (noun): Training a model using data that comes with known correct answers.
The dynamics model is trained via supervised learning.
gradient step (noun): One update to a model's parameters based on a computed gradient.
Policy gradient methods take a gradient step to improve the policy.
trajectories (noun): Recorded sequences of states and actions produced by a policy.
The green box sums rewards along sampled trajectories.
cheap (adjective): Requiring little time, effort, or resources.
A fast simulator makes sampling nearly free and cheap.
environment (noun): The world or system an agent interacts with and receives feedback from.
The policy is run in the environment to generate samples.
power grid (noun): The network that generates and distributes electricity.
Sampling from a real power grid can be very expensive.

Chapters

0:00 The anatomy of a reinforcement learning algorithm
2:12 A simple example
3:40 Another example: RL by backprop
5:11 Which parts are expensive?

← Lecture 4: Introduction to RL Algorithms, Part 1 · Lecture 4, Part 3: Q-Functions and Value Functions →