Lecture 22: Bayesian Statistical Inference II

What this lecture covers

This lecture completes the course's Bayesian inference unit by working through a concrete estimation problem end to end, then developing the theoretical properties of the conditional-expectation estimator and introducing a computationally simpler alternative: the linear least mean squares estimator. It follows directly from the previous lecture's introduction of Bayesian estimation and the optimality of conditional expectation under squared error.

By the end, you should be able to derive a posterior distribution and conditional-expectation estimator from a joint density, compute the resulting mean squared error as a function of the observation, prove that an optimal estimator's error is unbiased and uncorrelated with the estimate, and derive and interpret the linear least mean squares estimator, including how it generalizes to multiple noisy measurements.

Key ideas

• Posterior-based estimation recap: given a prior on theta and a conditional model P(X|theta), Bayes' rule yields the posterior P(theta|X), and the conditional expectation E[theta|X] minimizes mean squared error, both on average and for every specific observed value of X.
• Worked example: with theta uniform on [4,10] and X = theta + U for U uniform on [-1,1], the posterior of theta given X is uniform over the feasible interval, so the optimal estimator g(X) is the midpoint of that interval, a nonlinear, piecewise-linear function of X.
• Observation-dependent error: the mean squared error (posterior variance) varies with which X was observed; some observations pin down theta almost exactly, while others leave much more uncertainty.
• Unbiasedness: for the optimal estimator, E[theta_hat - theta | X] = 0 for every value of X, so the estimation error has zero mean, both conditionally and unconditionally.
• Error uncorrelated with any function of the data: the covariance between the error and any function h(X) of the observation, including the estimate theta_hat itself, is 0 — extreme estimates give no systematic reason to expect the error to be positive or negative.
• Variance decomposition: Var(theta) = Var(theta_hat) + Var(error), since the estimate and error are uncorrelated; this is a version of the law of total variance, showing how much of theta's original uncertainty the estimator removes.
• Linear least mean squares estimator: restricting to estimators of the form theta_hat = aX + b and minimizing mean squared error gives a closed-form solution using only means, variances, and covariance, not the full joint distribution.
• Interpreting the linear estimator: the optimal linear estimate corrects the prior mean by the deviation of X from its expected value, scaled by the covariance between X and theta; when X and theta are uncorrelated, no correction is made.
• Resulting error formula: the mean squared error of the linear estimator is Var(theta)*(1 - rho^2), where rho is the correlation coefficient; rho = 1 gives zero error, rho = 0 gives no improvement over the prior.
• Multiple measurements: for several independent noisy measurements of a scalar theta, the optimal linear estimator is a variance-weighted average of the measurements and the prior mean, with the prior mean treated as an extra "data point."
• Normal special case: when all variables are normal, the optimal linear estimator coincides with the true conditional expectation, since conditional expectations of normal variables are linear.
• Invariance and its limits: optimal (conditional-expectation) estimation is unaffected by invertible transformations of the data (e.g., observing X vs X^3), but linear estimation is not invariant to such transformations, since a linear function of X differs from a linear function of X^3.

Walkthrough

Recap of the Bayesian estimation framework (0:00)

The lecture reviews the prior, conditional model, joint density, and posterior distribution, and restates that the conditional expectation E[theta|X] is optimal under mean squared error, both for a specific observation and on average.

Worked example: uniform prior, uniform noise (6:05)

With theta uniform on [4,10] and X = theta + U, the lecture constructs the joint density, derives the posterior as uniform over the feasible interval for each X, and shows the resulting estimator g(X) is the midpoint of that interval — a nonlinear, piecewise curve.

Observation-dependent mean squared error (11:16)

The lecture computes the posterior variance as a function of X, showing it's constant (1/3) in the middle of the range but drops to 0 at the boundary observations (where theta is known exactly), illustrating that some measurements are more informative than others.

Proving the estimator is unbiased (17:27)

Using linearity of conditional expectation and the fact that theta_hat is a function of X, the lecture proves E[theta_hat - theta | X] = 0 for every X, establishing that the optimal estimator is unbiased.

Error uncorrelated with the estimate (21:33)

Extending the argument to any function h(X) and using the law of iterated expectations, the lecture shows the covariance between the error and h(X) — including theta_hat itself — is 0, meaning a large or small estimate gives no reason to suspect a systematic error in either direction.

Variance decomposition (27:53)

Because the error and estimate are uncorrelated, Var(theta) = Var(theta_hat) + Var(error), giving an interpretation of how much of the original uncertainty in theta the estimator has "absorbed."

Deriving the linear least mean squares estimator (30:55)

To avoid the complexity of computing a full conditional expectation, the lecture restricts to estimators theta_hat = aX + b, minimizes the resulting quadratic mean squared error over a and b, and shows the optimal coefficients depend only on means, variances, and the covariance between X and theta.

Interpreting the linear estimator and its error (34:03)

The lecture explains the resulting estimator as a correction to the prior mean based on how far X deviates from its expected value, scaled by the covariance, and gives the clean formula for the resulting mean squared error in terms of the correlation coefficient rho.

Extending to multiple measurements (39:17)

For a scalar theta observed through several independent noisy measurements, the lecture shows the optimal linear estimator is a variance-weighted average of the measurements and the prior mean, with less reliable (higher-variance) measurements given less weight.

Invariance under transformations (48:35)

The lecture closes by noting that optimal (conditional-expectation) estimation is unaffected by relabeling the data via an invertible transformation like X^3, but linear estimation is not, since a linear function of X is not equivalent to a linear function of X^3 — a caution when choosing what scale to apply linear estimation on.

Before you watch

• Watch the previous lecture on Bayesian estimation, which introduces the posterior distribution, MAP estimate, and the optimality of conditional expectation.
• Review joint densities, conditional distributions, and the law of iterated expectations.

Check your understanding

1. In the worked example, why is the posterior distribution of theta given X uniform, and why does the resulting estimator turn out to be piecewise linear rather than a single straight line?
2. How does the proof that E[theta_hat - theta | X] = 0 establish that the optimal estimator is unbiased?
3. Why does the fact that the error is uncorrelated with the estimate mean the estimator cannot be improved?
4. How does the linear least mean squares estimator's weighting of multiple measurements depend on each measurement's variance, and why does that weighting make sense?
5. Why is optimal (conditional-expectation) estimation invariant to invertible data transformations, while linear estimation is not?