Inferring a Parameter of the Uniform Distribution, Part 2

What this lecture covers

This is the second half of a two-part review problem on Bayesian inference (see Part 1, which derives the posterior for an unknown uniform-distribution parameter theta from observed delays). Here the lecture uses that posterior to evaluate and compare estimators: it computes the conditional mean squared error (MSE) for the MAP and LMS estimators found earlier, then derives a third estimator, the linear LMS, and compares all three.

After watching, you should be able to compute conditional MSE for an estimator given a posterior distribution, explain why the LMS estimator always achieves the lowest MSE by construction, and derive a linear LMS estimator from expectation, variance and covariance calculations.

Key ideas

• Conditional MSE: for an estimator theta_hat, it is defined as the expected squared error E[(theta_hat - theta)^2 | X = x], computed by integrating against the posterior.
• Expanding the square: conditional MSE splits into three integrals involving theta_hat^2, theta_hat * theta, and theta^2, each evaluated against the posterior density.
• MAP vs. LMS comparison: plotting both conditional MSE curves shows the LMS estimator has lower error everywhere, because it is designed specifically to minimize mean squared error.
• Linear LMS estimator: the formula E[theta] + (Cov(theta, X) / Var(X)) * (X - E[X]) restricts the estimator to a linear function of the data, trading some accuracy for simplicity.
• Law of iterated expectations and total variance: both are used to compute E[X] and Var(X) when X's distribution depends on the random parameter theta.
• Estimator artifacts: the linear LMS estimator can produce theta estimates greater than 1, which is impossible given the model, illustrating a known limitation of forcing linearity.
• Performance trade-off: despite this flaw, the linear LMS estimator tracks the LMS estimator's MSE closely except near the boundary, where the impossible estimates increase error.

Walkthrough

Setting up conditional MSE (0:23)

The lecture defines conditional MSE generically for any estimator theta_hat and expands the square into three integral terms, each weighted by the posterior distribution 1/theta * |log x| derived in Part 1.

Computing MSE for MAP and LMS (2:32)

Plugging in theta_hat = x (the MAP estimate) and theta_hat = (1-x)/|log x| (the LMS estimate) into the general MSE formula produces two explicit expressions. Plotting them confirms the LMS estimator has uniformly lower conditional MSE, consistent with its role as the minimum-MSE estimator.

Introducing the linear LMS estimator (6:11)

Because the LMS estimator is not linear in x, the lecture introduces the linear LMS estimator, of the form a*x + b, computed from E[theta], E[X], Var(X), and Cov(theta, X).

Calculating the four required quantities (7:16)

Using the law of iterated expectations and the law of total variance, the lecture finds E[theta] = 1/2, E[X] = 1/4, Var(X) = 7/144, and Cov(theta, X) = 1/24, combining conditional facts about the uniform distribution at each layer.

Assembling and interpreting the linear LMS estimator (15:31)

Substituting the four quantities gives theta_hat = (6/7)x + 2/7. The lecture points out that for x close to 1, this estimator can exceed 1, which is impossible for theta, an artifact of forcing a linear form.

Comparing all three estimators (18:36)

The lecture closes by comparing the conditional MSE of the linear LMS estimator against the LMS estimator, showing they are close except near x = 1, where the linear estimator's out-of-range values hurt its performance, and summarizes the full set of tools from Chapter 8 used across both parts.

Before you watch

• Watch Part 1 of this problem first, since it derives the posterior distribution used throughout.
• Review the definitions of MAP, LMS, and linear LMS estimators, and the formulas for covariance and variance.
• Be comfortable with the law of iterated expectations and the law of total variance.

Check your understanding

1. Why does the LMS estimator always achieve the lowest conditional mean squared error among all estimators?
2. What four quantities are needed to compute a linear LMS estimator, and why?
3. Why can the linear LMS estimator produce a value of theta greater than 1, and what does that reveal about its limitations?
4. How does the law of total variance help compute Var(X) when X's distribution depends on a random parameter?
5. Under what circumstances might a linear LMS estimator still be preferred over the LMS estimator despite its higher error?