An Inference Example

What this lecture covers

This review problem starts from a joint PDF of two random variables X and Y that is constant (2/3) over an irregular region and works through several estimation techniques applied to it. The lecture shows how to read a conditional expectation directly off a picture of the joint density when the density is piecewise uniform, then formalizes the answer algebraically.

Building on that estimator, the lecture computes its conditional and unconditional mean squared error, marginalizes the joint PDF to find the distribution of X, derives the linear LMS estimator for comparison, and explains why a MAP estimator breaks down in this particular setting. It is a consolidation lecture, pulling together tools of conditional expectation, variance, and estimation from earlier in the course.

Key ideas

• Reading conditional expectation from a picture: when a joint PDF is piecewise uniform, slicing the region at a fixed X value gives a uniform conditional distribution of Y, whose expectation is just the midpoint of the slice.
• Piecewise LMS estimator: here the LMS estimator of Y given X is X/2 for X between 0 and 1, and X - 1/2 for X between 1 and 2, with a kink at X = 1.
• Conditional variance equals conditional MSE for the LMS estimator: because the LMS estimator is the conditional expectation, the conditional mean squared error reduces to the conditional variance of Y given X.
• Marginalizing a joint PDF: integrating the joint density over Y for each region of X gives the marginal PDF of X, needed to compute unconditional expectations.
• Law of iterated expectations for MSE: the overall mean squared error equals the expectation, over X, of the conditional mean squared error.
• Linear LMS estimator: computed from E[Y], E[X], Var(X), and Cov(X,Y), it approximates the true (kinked) LMS estimator with a single straight line, at the cost of allowing impossible negative estimates of Y.
• MAP estimator can be ill-defined: when the conditional distribution of Y given X is uniform, every value of Y has equal posterior density, so there is no unique maximum and no sensible MAP estimate.

Walkthrough

Reading the joint PDF and finding the LMS estimator visually (0:00)

The lecture presents a joint PDF of X and Y that is a constant 2/3 over a bounded region, and shows that because it is uniform, the conditional expectation of Y given X can be found visually as the midpoint of each vertical slice, giving a piecewise linear function with a kink at X = 1.

Computing the conditional mean squared error (4:08)

The lecture defines conditional MSE and shows it equals the conditional variance of Y given X, since the LMS estimator is the conditional mean. Using the uniform-distribution variance formula (width squared over 12), it computes this variance for both regions of X.

Marginalizing to find the distribution of X and the overall MSE (8:13)

To get the unconditional mean squared error, the lecture marginalizes the joint PDF over Y to find f_X(x), piecewise 2/3 x and 2/3, then applies the law of iterated expectations to combine the conditional MSE with this marginal density, arriving at a final MSE of 5/72.

Deriving the linear LMS estimator (13:27)

The lecture computes E[X], Var(X), E[Y] (via iterated expectations using the earlier LMS result), and Cov(X,Y) (via a double integral over the joint PDF), then assembles the linear LMS formula to get an explicit straight-line estimator.

Comparing estimators and their mean squared errors (21:43)

Plotting the linear LMS estimator against the LMS estimator shows they are close but the linear version lacks the kink and can dip below zero for small X, an impossible value for Y. The lecture explains that the LMS estimator must have the smallest MSE by construction, so the linear LMS estimator's MSE can only be equal or larger.

Why the MAP estimator fails here (25:50)

Because each conditional slice of Y given X is uniform, every value of Y in the slice has the same conditional density, so there is no single maximizing value and the MAP rule gives no unique estimate.

Before you watch

• Review conditional expectation and conditional variance for jointly distributed random variables.
• Be familiar with the formulas for LMS and linear LMS estimators and how covariance and variance are computed from a joint PDF.
• Recall how to marginalize a joint PDF to find a single variable's distribution.

Check your understanding

1. Why can the conditional expectation of Y given X be read directly off a picture when the joint PDF is piecewise uniform?
2. Why does the conditional MSE of the LMS estimator equal the conditional variance of Y given X?
3. How is the law of iterated expectations used to go from conditional MSE to overall MSE?
4. Why does forcing the estimator to be linear allow it to produce impossible values, and why is that acceptable in some situations?
5. Why is the MAP estimator undefined in this problem, and what does that reveal about when MAP is a useful estimation method?