Lecture 4, Part 4: Types of RL Algorithms

What this lecture covers

This part gives a map of the reinforcement learning landscape before the course dives into each family in detail. It answers: what are the main categories of RL algorithms, and how does each one fill in the "green box" (fitting/learning) and "blue box" (policy improvement) steps of the generic RL algorithm anatomy?

After watching, you can name the four main algorithm families, describe what each one learns and how it uses that to improve the policy, and recognize terms like rollout, value-based method, and model-based RL that recur throughout the rest of the course.

Key ideas

• Policy gradient methods: directly estimate the derivative of the RL objective with respect to the policy parameters theta and take a gradient ascent step; the green box just sums rewards along sampled trajectories (rollouts).
• Value-based methods: fit V(s) or Q(s, a), usually with a neural network, and often represent the policy only implicitly as arg max_a Q(s, a) rather than as an explicit distribution.
• Actor-critic methods: a hybrid that fits a value function or Q-function like value-based methods, then uses it in the blue box to compute a more accurate policy gradient step.
• Model-based RL: learns a transition model p(s_{t+1} | st, at), then either plans directly with it, backpropagates rewards through it, uses it to learn a value/Q-function via dynamic programming, or uses it to generate extra synthetic data for a model-free algorithm.
• Rollout: sampling from the policy step by step in the environment; the term comes from "unrolling" the policy over time.
• Second-order tricks: backpropagating rewards through a learned model for policy improvement is simple in principle but often needs second-order optimization methods for numerical stability.

Before you watch

• Review the Q-function and value function definitions and the green-box/blue-box anatomy of an RL algorithm from the previous part of this lecture.

Check your understanding

1. What does a policy gradient algorithm compute in the blue box that a value-based method does not?
2. Name three different ways a learned transition model can be used to improve a policy.
3. How does an actor-critic method combine ideas from value-based and policy gradient methods?
4. In a pure value-based method, how is the policy represented?