Lecture 21: Bayesian Statistical Inference I

What this lecture covers

This lecture opens the course's final unit on statistical inference: how you go from real-world data to a model of what's happening, and how to use that model to make estimates or decisions. It surveys where inference shows up (polling, medical trials, recommendation systems, signal processing, curve fitting) and frames the field conceptually as an application of probability tools already covered in the course, while noting that, unlike ordinary probability problems, inference problems don't have a single "correct" method.

The lecture develops the Bayesian approach specifically: treating an unknown quantity as a random variable with a prior distribution, then using Bayes' rule to compute its posterior distribution given data. It introduces two ways to report a single point estimate from a posterior — the maximum a posteriori (MAP) estimate and the conditional expectation — and proves that the conditional expectation minimizes mean squared error. After watching, you should be able to distinguish hypothesis testing from estimation problems, distinguish Bayesian from classical (non-Bayesian) approaches to an unknown quantity, compute MAP and conditional-expectation estimates from a posterior distribution, and explain why conditional expectation is optimal under squared error.

Key ideas

• Inference vs. probability: probability problems have a single correct answer given the model; inference problems involve choosing among multiple reasonable methods to fit a model to observed data, with no universally "correct" choice.
• Hypothesis testing vs. estimation: hypothesis testing problems involve an unknown quantity from a small discrete set of options, and aim to maximize the probability of a correct decision; estimation problems involve a continuous unknown quantity, and aim to minimize the size of the error.
• Bayesian vs. classical inference: the classical approach treats the unknown quantity theta as a fixed but unknown constant; the Bayesian approach treats theta as a random variable with a subjective prior distribution reflecting initial beliefs, updated via Bayes' rule after observing data.
• Posterior distribution: applying Bayes' rule with a prior and a model of the data-generating process (P(X | theta)) yields the posterior distribution or density of theta given the observed data.
• MAP estimate: reporting the value of theta with the highest posterior probability (discrete case) or highest posterior density (continuous case) minimizes the probability of an incorrect decision in the discrete case; in the continuous case its optimality rationale is weaker since any single continuous value has probability zero.
• Conditional expectation estimate: reporting E[theta | X] is a different, commonly used point estimate, especially natural for continuous unknowns.
• Conditional expectation minimizes mean squared error: for the "no data" warm-up case, minimizing E[(theta - c)^2] over c gives c = E[theta], with resulting error equal to Var(theta); conditioning on data, the same argument shows E[theta | X] minimizes mean squared error in the conditional universe, and this optimality holds on average over all data too.
• Practical difficulties: real problems require choosing a plausible prior (a judgment call) and often involve multi-dimensional integrals over vectors of unknowns and data, which can be computationally difficult even though the underlying formula is simple.

Walkthrough

From real phenomena to models (0:00)

The lecture frames statistical inference as the bridge between real-world data and probabilistic models, giving examples across polling, medical trials, the Netflix recommendation competition, financial data, and signal processing, and noting the field's long history in astronomical curve-fitting.

Why inference differs from probability, and the risk of misuse (5:11)

Unlike probability problems with a unique correct answer, inference problems admit multiple reasonable methods. The lecture warns that statistics is frequently misapplied when people plug data into a method without understanding its assumptions and guarantees.

System identification and communication as the same problem (9:16)

Using a signal-plus-noise example (X = A*S + noise), the lecture shows that estimating an unknown channel gain A (system identification) and decoding an unknown transmitted signal S (communication) are mathematically the same linear inference problem, just with different unknowns.

Classifying inference problems (12:19)

The lecture distinguishes hypothesis testing (discrete unknown, minimize probability of error) from estimation (continuous unknown, minimize size of error), using the airplane-radar and polling examples respectively.

Bayesian vs. classical philosophies (14:20)

Two philosophical approaches to an unknown theta are contrasted: classical statistics treats it as a fixed unknown number; Bayesian statistics models it as a random variable with a prior distribution reflecting subjective belief, even when the quantity isn't inherently random.

Bayes' rule and the posterior (18:24)

The lecture shows how Bayes' rule combines a prior P(theta) with a model of the data given theta to compute the posterior P(theta | X), for discrete and continuous cases, and applies this to estimating the coefficients of a parabolic trajectory from noisy position measurements, noting that unknowns and data are typically vectors rather than single variables.

MAP estimates and reporting a single answer (25:31)

Using a coin-bias example, the lecture contrasts the classical estimator (sample proportion of heads) with the Bayesian approach (choosing a prior, then computing the posterior). It introduces the MAP estimate as the posterior mode, which minimizes the probability of an incorrect decision in the discrete case, and notes the weaker justification for this choice with continuous densities.

Conditional expectation and the least-squares warm-up (34:43)

As a warm-up without data, the lecture minimizes E[(theta - c)^2] over constants c, showing the optimal choice is c = E[theta], with mean squared error equal to Var(theta).

Proving conditional expectation is optimal given data (38:50)

Applying the same argument within the "conditional universe" created by observing X, the lecture shows the optimal point estimate given data is E[theta | X]. It then proves this beats any other estimator g(X) in mean squared error for every value of X, and therefore on average, establishing conditional expectation as the optimal estimator under squared error.

Practical complications (45:58)

The lecture closes by noting that while the theory is simple, applying it involves choosing a defensible prior and computing potentially difficult multi-dimensional integrals, motivating a simpler alternative to be introduced in the next lecture.

Before you watch

• Review Bayes' rule for both discrete and continuous random variables, and conditional expectation.
• Recall the weak law of large numbers and the polling example, referenced here as a classical estimator.

Check your understanding

1. How does the Bayesian approach to an unknown quantity differ philosophically from the classical approach, and how does this affect the tools each uses?
2. What is the maximum a posteriori (MAP) estimate, and why is its optimality rationale different in discrete versus continuous settings?
3. How is it proven that E[theta | X] minimizes mean squared error, and why does this proof extend from a fixed value of X to an average over all values of X?
4. Why can implementing Bayesian estimation in practice be difficult even though the guiding formula is simple?