Lecture 14, Part 1

What this lecture covers

This lecture opens a different angle on exploration than the previous lecture's count- and bonus-based methods: what if there is no reward signal at all? The motivating idea is that an agent (or a child at play) could acquire a broad repertoire of skills through unsupervised exploration, then repurpose that knowledge quickly once a real goal is given, similar to a home robot practicing in a kitchen before being asked to do the dishes.

To build toward the algorithms covered in later parts (goal-reaching without reward, state-distribution matching, coverage-based objectives, and skill discovery), this part reviews the information-theoretic building blocks: entropy and mutual information, and introduces "empowerment" as a concrete example of how mutual information can express a meaningful RL objective.

Key ideas

• Reward-free exploration: instead of seeking reward, the agent aims to acquire diverse, reusable skills or knowledge that can later be repurposed for arbitrary goals.
• Entropy H(p(x)): measures how broad a distribution is; a uniform distribution has maximal entropy, a distribution peaked on one value has minimal entropy.
• Mutual information I(x;y): the KL divergence between the joint distribution p(x,y) and the product of marginals p(x)p(y); it is zero when x and y are independent and grows as they become more dependent.
• Mutual information as entropy reduction: I(x;y) = H(y) - H(y|x), the reduction in uncertainty about y from observing x, connecting it to the information-gain idea from the previous lecture.
• State marginal entropy H(pi(s)): the entropy of the states a policy visits, used later in the lecture as a measure of how much coverage a policy achieves.
• Empowerment: the mutual information between the next state and the current action, I(s_{t+1}; a_t); maximizing it favors states with many available actions that reliably lead to many different, controllable future states.

Before you watch

• Watch the previous exploration lecture (Lecture 13), since this lecture explicitly contrasts reward-free exploration with the count- and information-gain-based bonus methods covered there.
• A basic grasp of probability distributions and KL divergence is assumed, though the lecture reviews entropy and mutual information from scratch.

Check your understanding

1. Why might an agent benefit from acquiring skills before it knows what its future goals will be?
2. How does mutual information relate to entropy, and why does that make it a natural measure of "informativeness"?
3. Why does empowerment require both high entropy of the next state and low entropy of the next state given the action?
4. What real-world intuition does the "standing in the middle of a room" example illustrate about empowerment?