Lecture 4, Part 3: Q-Functions and Value Functions

What this lecture covers

This part answers a specific question: what mathematical object captures "how good is this state or action under my current policy," and why is that object useful? It follows directly from the definition of the reinforcement learning objective as an expectation over trajectories, which the lecture rewrites as a chain of nested expectations.

After watching, you can write down the Q-function and value function from the RL objective, explain how they relate to each other, and describe two ways they can be used to make a policy better: acting greedily with respect to the Q-function, or nudging the policy toward actions whose Q-value beats the state's average value.

Key ideas

• Nested expectations: the RL objective (expected sum of rewards) can be expanded, via the chain rule, into an expectation over s1, inside an expectation over a1, inside an expectation over s2, and so on.
• Q-function: Q(st, at) is the expected sum of future rewards if you start in state st, take action at, and then follow the policy.
• Value function: V(st) is the expected sum of future rewards if you start in state st and follow the policy; it equals the expectation of Q(st, at) over actions drawn from the policy.
• Objective from V: the full RL objective equals the expectation of V(s1) over the initial state distribution.
• Policy improvement via argmax: given a full Q^pi(s, a), a new policy that puts all probability on arg max_a Q^pi(s, a) at every state is at least as good as pi, and usually better.
• Better-than-average actions: if Q^pi(s, a) > V^pi(s), action a does better than the policy's average, which is a signal used later for gradient-based updates (policy gradients).
• Where they fit: in the general anatomy of an RL algorithm, Q-functions and value functions are typically learned in the "green box" (fitting step) and used in the "blue box" (policy improvement step).

Before you watch

• Know the RL objective as an expectation of summed rewards over the trajectory distribution, and the "orange/green/blue box" anatomy of an RL algorithm from earlier in this lecture.
• Be comfortable with the chain rule of probability applied to factorize a joint distribution.

Check your understanding

1. How is Q^pi(st, at) defined, and how does it differ from V^pi(st)?
2. Why does knowing Q^pi(s1, a1) make it easy to pick the best policy at the first time step?
3. If Q^pi(s, a) is greater than V^pi(s), what does that tell you about action a?
4. In the green-box/blue-box view of RL algorithms, where do Q-functions and value functions get used?