Lecture 13, Part 4: Exploration

What this lecture covers

This part continues the survey of optimism-based exploration methods, moving past density-model pseudo-counts to three other techniques for measuring state novelty. It covers hash-based counting, where similar states are compressed to the same code so exact counting becomes feasible again, classifier-based density estimation (EX2), which sidesteps generative modeling entirely, and heuristic prediction-error methods that use the error of a learned model as a novelty signal.

Together these give a sense of the range of practical tricks used to approximate "have I seen something like this before" without needing an exact count or a high-quality generative model.

Key ideas

• Hashing exploration: compress each state into a short k-bit code via an encoder, then count occurrences of each code; shorter codes cause more collisions, which broadens the notion of "similar" states.
• Learned hashing via autoencoders: training the encoder for reconstruction accuracy makes collisions more likely between states that actually look alike, rather than colliding arbitrarily.
• EX2 classifier-based density: train a classifier to distinguish a given state from all previously seen states; an easily distinguished (novel) state gets a low estimated density, derived from the classifier's output via the Bayes-optimal classifier formula.
• Amortized classifier: instead of training a separate classifier per state, train one network conditioned on both the exemplar state and the state being classified, updated online as new states arrive.
• Prediction-error novelty: fit a function f_hat to approximate some target function f_star on observed states and actions; large prediction error signals states far from what has been seen, so error itself becomes the novelty bonus.
• Random network target: a particularly simple and effective choice is to let f_star be a randomly initialized, untrained neural network, since it only needs to vary meaningfully across the state space, not be semantically meaningful.

Before you watch

• Watch the previous part of Lecture 13, which introduces pseudo-counts from density models; this part presents these methods as alternatives to that approach.

Check your understanding

1. How does hash-based exploration make exact counting feasible again in large state spaces?
2. Why does training the hash encoder for reconstruction accuracy improve which states collide into the same code?
3. How does the EX2 method turn a classifier's output into a density estimate for a state?
4. Why can a randomly initialized, untrained network serve as a useful novelty-detection target?