Lecture 11, Part 3: Estimating Epistemic Uncertainty with Neural Networks

What this lecture covers

Following Part 2's argument that uncertainty-aware planning helps model-based RL, this part addresses how to actually get calibrated model uncertainty out of a neural network dynamics model. It distinguishes two kinds of uncertainty and works through two practical techniques: Bayesian neural networks and bootstrap ensembles.

After watching, you'll be able to explain why the entropy of a model's output distribution is the wrong signal for model uncertainty, define the difference between aleatoric and epistemic uncertainty, and describe how bootstrap ensembles approximate a posterior over model parameters in practice.

Key ideas

• Naive idea (and why it fails): using the entropy of a network's own output distribution (Gaussian variance or softmax logits) as "uncertainty" doesn't work, because an overfit model can be highly confident and completely wrong on out-of-distribution inputs.
• Aleatoric (statistical) uncertainty: irreducible randomness in the true function itself, such as the outcome of a dice roll; it doesn't shrink with more data.
• Epistemic (model) uncertainty: uncertainty about which model is correct, arising from limited data; it shrinks as more data is collected, and it's the kind relevant to safe planning.
• Parameter posterior: instead of a single best-fit parameter theta, estimate a distribution p(theta | D) and integrate (average) predictions over it to capture epistemic uncertainty.
• Bayesian neural networks: place a distribution over every weight (commonly approximated as independent Gaussians per weight) instead of a single value, then sample networks from this distribution to get a posterior over predictions.
• Bootstrap ensembles: train multiple independent networks and treat their disagreement as an uncertainty estimate; mathematically this approximates the parameter posterior as a mixture of Dirac deltas centered at each network's parameters.
• Practical simplification: resampling training data with replacement for each ensemble member is theoretically motivated but often unnecessary in practice — different random initialization and SGD noise alone tend to produce sufficiently diverse models.

Walkthrough

Why output entropy is the wrong signal (0:00)

The lecture first shows why a plausible-seeming shortcut — using the variance or entropy that a neural net already outputs as its predictive distribution — fails as an uncertainty measure. An overfit model can predict a near-zero variance (high confidence) at training points while being completely wrong at nearby test points, because this entropy reflects how noisy the model thinks the dynamics are, not how uncertain we should be about the model itself.

Two types of uncertainty (3:22)

The lecture draws the key distinction between aleatoric uncertainty (irreducible randomness in the true dynamics, like a dice roll) and epistemic uncertainty (uncertainty about which model is correct, which shrinks with more data). Epistemic uncertainty is what matters for cautious planning in the cliff-edge and mountain-climbing examples from earlier, and it requires estimating a distribution over model parameters theta rather than a single point estimate.

Bayesian neural networks (7:32)

The lecture sketches the high-level idea of Bayesian neural networks: place a distribution (often independent Gaussians as a tractable approximation) over each weight, then sample whole networks from this distribution to get a posterior over predictions. It notes this requires variational inference machinery covered later in the course, and points to papers on weight uncertainty and concrete dropout for further reading.

Bootstrap ensembles (10:55)

As a simpler and often better-performing alternative, the lecture presents bootstrap ensembles: train several independent models and use their disagreement as an uncertainty signal. This formally approximates the parameter posterior as a mixture of Dirac deltas, one per ensemble member — meaning predictions should be combined as a mixture of distributions, not by averaging predicted means. The bootstrap idea of resampling the training set with replacement per model is explained, along with the practical note that this resampling often isn't necessary for deep networks.

Before you watch

• Watch Lecture 11, Parts 1 and 2 first, since this part builds directly on the cliff-edge uncertainty motivation and the HalfCheetah performance gap discussed there.
• Basic familiarity with Gaussian and softmax output distributions, as used elsewhere in the course, is assumed.

Check your understanding

1. Why does a network's own predicted variance fail as a measure of epistemic uncertainty?
2. Give an example each of aleatoric and epistemic uncertainty, and explain why only one of them shrinks with more data.
3. What does it mean to "integrate out theta" when making a prediction, and why is this normally intractable for neural networks?
4. How does a bootstrap ensemble approximate the parameter posterior, and why should predictions be mixed rather than averaged as means?
5. Why does the lecture say resampling with replacement is often unnecessary when training a bootstrap ensemble of deep networks?