Lecture 4, Part 6: Examples of Deep RL Algorithms

What this lecture covers

This short closing part of the lecture rounds things out with concrete examples rather than new theory. It names specific algorithms within each family covered earlier (value-function fitting, policy gradient, actor-critic, model-based) and shows short video results so you can see what these methods actually produce.

After watching, you can recognize the names of common deep RL algorithms grouped by family, and connect each to a real result: playing Atari from pixels, robotic manipulation, and learning to walk.

Key ideas

• Value function fitting methods: named examples include Q-learning, DQN, and temporal difference learning.
• Policy gradient methods: named examples include REINFORCE, natural gradient, TRPO (trust region policy optimization), and PPO.
• Actor-critic methods: named examples include A3C (asynchronous advantage actor-critic), soft actor-critic, and DDPG.
• Model-based methods: named examples include Dyna, guided policy search, MPPI, and SVG.
• Atari from pixels: a 2013 DQN-style result uses Q-learning with convolutional networks to output a Q-value per discrete action, then selects the action with the highest value.
• Robot grasping and walking: a guided policy search result (model-based, using learned dynamics plus image-based convolutional networks) performs robotic skills, and a TRPO-based actor-critic result trains a simulated humanoid to walk; a continuous-action variant of Q-learning is used for a grasping robot.

Before you watch

• Watch the earlier parts of this lecture on Q-functions/value functions and the four algorithm families, since this part assumes you already know what value-based, policy gradient, actor-critic, and model-based methods are.

Check your understanding

1. Which family does DQN belong to, and how does it choose actions in a discrete-action game like Atari?
2. What two components does the guided policy search example combine to perform robotic skills?
3. Which algorithm type is used in the humanoid walking example, and why is it described as a hybrid?