Lecture 23: Classical Statistical Inference I

What this lecture covers

This lecture opens a three-lecture unit on classical statistics, where the unknown quantity being estimated is treated as a fixed constant rather than a random variable with a prior distribution, in contrast to the Bayesian approach covered earlier in the course. The lecture asks how to build an estimator from observed data and how to judge whether that estimator is any good, introducing maximum likelihood estimation (MLE) as the general-purpose method and comparing it to Bayesian MAP estimation.

The second half of the lecture applies these ideas to the simplest and most common estimation problem: estimating the mean of a distribution using the sample mean. It develops the properties an estimator should have (unbiasedness, consistency, small mean squared error) and finishes by constructing a confidence interval, explaining carefully what a 95% confidence interval does and does not claim about the unknown parameter. This lecture assumes familiarity with the law of large numbers, the central limit theorem, and Bayesian estimation from earlier in the course, and sets up hypothesis testing covered in the following two lectures.

Key ideas

• Classical vs. Bayesian statistics: in classical statistics the unknown parameter theta is a fixed constant, not a random variable, so there is no prior distribution over it, and probability statements are written with a semicolon (f(x; theta)) instead of a conditioning bar.
• Maximum likelihood estimation: choose the value of theta that makes the observed data most likely, by maximizing the joint density (or PMF) of the data as a function of theta; in practice this is done by maximizing the log-likelihood.
• MLE as a limiting case of MAP: maximum likelihood estimation is mathematically what Bayesian MAP estimation reduces to when the prior on theta is uniform (constant), since a constant prior drops out of the maximization.
• Bias: an estimator is unbiased if, on average across repeated experiments, it equals the true parameter for every possible true value; the lecture shows the MLE for an exponential distribution's rate parameter is biased.
• Consistency: an estimator is consistent if it converges in probability to the true parameter as the amount of data grows, typically shown using the weak law of large numbers.
• Mean squared error decomposition: MSE = Var(theta_hat) + bias^2, so a good estimator must control both variance and bias, not just one.
• The sample mean as an estimator: averaging observed data to estimate an unknown mean is unbiased, consistent, and has mean squared error variance/n, and in several cases coincides with the MLE.
• Confidence intervals: a 95% confidence interval is a random interval, constructed from the data, that has a 95% probability of covering the fixed true parameter; the true parameter itself has no probability of being anywhere, since it is not random.

Walkthrough

The classical estimation framework (1:00)

The lecture sets up the general picture: an unknown constant theta, a noisy measurement process, and an estimator box that turns observed data X into an estimate theta_hat. It stresses that f(x; theta) denotes a family of distributions indexed by theta, not a conditional distribution, since theta has no probability distribution of its own.

Maximum likelihood estimation and its link to Bayesian MAP (6:03)

Maximum likelihood estimation picks the theta under which the observed data was most likely to occur. The lecture compares this to the Bayesian MAP estimator, which maximizes the posterior f(theta|x), and shows the two coincide when the Bayesian prior on theta is uniform, since the prior term then contributes nothing to the maximization.

Worked example: exponential distribution (9:05)

For independent, identically distributed exponential data, the lecture derives the log-likelihood, differentiates it with respect to theta, and solves to find that the MLE is the reciprocal of the sample mean, matching the known relationship between an exponential distribution's rate and its mean.

Bias and consistency of the exponential MLE (13:14)

The lecture defines bias as the deviation of E[theta_hat] from the true theta and shows the exponential-distribution MLE is biased, even infinitely biased with a single data point, because it involves a reciprocal of a random variable that has density near zero. It then shows the same estimator is asymptotically consistent, converging in probability to the true theta via the weak law of large numbers applied to the sample mean and the continuity of the reciprocal function.

Mean squared error and the bias-variance trade-off (21:30)

Mean squared error is shown to decompose into variance plus squared bias. A deliberately bad estimator, always reporting the number 100 regardless of the data, illustrates that zero variance can come at the cost of arbitrarily large bias, motivating estimators that balance both terms.

The sample mean estimator (26:37)

For estimating an unknown mean, the sample mean is shown to be unbiased, consistent by the weak law of large numbers, and to have mean squared error equal to the true variance divided by n. The lecture notes it often coincides with the MLE, for example under a normal distribution.

Constructing and interpreting confidence intervals (30:49)

The lecture explains that a confidence interval's endpoints are random variables built from the data, and carefully distinguishes the correct interpretation (95% of constructed intervals, across repeated experiments, will contain the fixed true theta) from the incorrect one (theta has a 95% chance of lying in one particular reported interval). Using the central limit theorem, it derives the standard sample_mean +/- 1.96 * sigma/sqrt(n) interval for a 95% confidence level.

Estimating the unknown standard deviation (42:03)

Since the true standard deviation sigma is usually unknown, the lecture presents options: use a conservative upper bound (as with Bernoulli data, where sigma is at most 1/2), or estimate sigma from the data itself, plugging the estimated mean into the variance formula (dividing by n-1 for an unbiased estimate) to produce a confidence interval that remains approximately valid for large n.

Before you watch

• Review Bayesian inference and MAP estimation, since this lecture contrasts them directly with maximum likelihood estimation.
• Be comfortable with the weak law of large numbers and the central limit theorem, both used repeatedly to justify estimator properties and to construct confidence intervals.
• Recall the standard normal table value 1.96 used for a 95% confidence level.

Check your understanding

1. Why does classical statistics treat the unknown parameter as a constant rather than a random variable, and how does this change the meaning of f(x; theta)?
2. Under what condition does maximum likelihood estimation give the same result as Bayesian MAP estimation?
3. Why can an estimator be biased even though it becomes consistent as the sample size grows?
4. How does mean squared error combine variance and bias, and why is it possible to reduce one at the expense of the other?
5. What is the correct interpretation of a 95% confidence interval, and why is it incorrect to say the true parameter has a 95% probability of lying inside one specific reported interval?