Lecture 24: Classical Inference II

What this lecture covers

This lecture continues the classical statistics unit, briefly reviewing maximum likelihood estimation and confidence intervals for a sample mean before moving into two major topics: linear regression and the start of hypothesis testing. It shows that fitting a line to data by minimizing squared prediction errors is not an arbitrary choice but can be derived as maximum likelihood estimation under a specific probabilistic model where the noise around the line is normally distributed.

The lecture then works through the mechanics of simple and multiple linear regression, discusses how to extend a model with nonlinear explanatory variables, and warns about common pitfalls: overfitting with too many parameters, heteroscedasticity, collinear explanatory variables, and mistaking correlation for causation. It closes by introducing the hypothesis testing framework, defining the two types of error (false rejection and false acceptance) and the likelihood ratio test, a topic that continues into the next lecture.

Key ideas

• Linear regression as maximum likelihood estimation: assuming Y = theta0 + theta1 * X + W with W independent, zero-mean, normally distributed noise, maximizing the likelihood of the observed data is mathematically equivalent to minimizing the sum of squared prediction errors.
• Closed-form regression estimates: for a single explanatory variable, theta1_hat is estimated from the ratio of the sample covariance of X and Y to the sample variance of X, and theta0_hat follows from the sample means of X and Y.
• Multiple linear regression: with several explanatory variables, the same least-squares idea applies, but no simple closed-form formula exists without matrix notation; the optimal parameters are found by solving a system of linear equations numerically.
• Nonlinear features, linear model: using a transformed variable like h(X) = X^2 as the explanatory variable keeps the model "linear" in its parameters, letting you fit curved relationships without changing the underlying method.
• Overfitting risk: using too many explanatory variables or too high a degree relative to the amount of data lets a model fit the data almost perfectly while producing unreliable, unstable predictions.
• R-squared: measures how much of the variance in Y is explained by the model, comparing the unconditional variance of Y to the variance of Y conditional on the explanatory variables.
• Common pitfalls: heteroscedasticity (noise variance that changes across the range of X), collinearity (redundant, highly correlated explanatory variables that make coefficient estimates unstable), and the general warning that a good linear fit shows association, not causation.
• Hypothesis testing setup: given two candidate distributions (the null hypothesis H0 and an alternative H1), the observed data space is split into a rejection region and an acceptance region, with two types of error (false rejection of H0, probability alpha, and false acceptance, probability beta) that trade off against each other.

Walkthrough

Review of classical estimation and confidence intervals (1:01)

The lecture briefly recaps maximum likelihood estimation, its relationship to Bayesian MAP under a flat prior, and the correct interpretation of a confidence interval as a property of the random interval rather than a probability statement about the fixed true parameter.

Motivating linear regression (9:12)

Using an SAT-score-versus-college-GPA example, the lecture introduces the idea of fitting a line theta0 + theta1 * X to data by minimizing the sum of squared prediction errors, first presented simply as a reasonable engineering choice.

Deriving regression as maximum likelihood estimation (13:18)

Assuming the noise term W is independent, identically distributed, and normal, the lecture writes out the likelihood of the observed Y values and shows that maximizing this likelihood reduces exactly to minimizing the sum of squared errors, giving least-squares regression a probabilistic justification.

Interpreting the closed-form estimator formulas (18:28)

The lecture explains where the regression formulas come from: assuming the linear model holds exactly in expectation, the true theta1 relates to the covariance of X and Y divided by the variance of X, and the observed formulas are simply data-based estimates of those covariance and variance quantities.

Multiple regression and nonlinear features (25:47)

Extending the model to several explanatory variables (for example, SAT score, family income, and other factors) keeps the same least-squares philosophy but requires solving a linear system numerically rather than using a simple formula. The lecture also shows that replacing X with a transformed variable like X^2 allows fitting curved relationships while remaining a linear model in its parameters.

Pitfalls: overfitting, heteroscedasticity, and collinearity (30:59)

The lecture warns that too many parameters relative to the data lets a model fit every point perfectly but produce unreliable predictions. It then describes heteroscedasticity, where noise variance varies across the range of X and skews the least-squares fit toward noisy regions, and collinearity, where redundant explanatory variables make coefficient estimates unstable.

Correlation versus causation, and R-squared (34:02)

The lecture defines R-squared as the fraction of the variance in Y explained by the explanatory variables, and stresses that a strong linear fit only demonstrates association between variables, not a causal mechanism.

Introducing hypothesis testing (42:13)

The lecture switches topics to hypothesis testing, defining a null hypothesis H0 and alternative H1, splitting the data space into a rejection region and its complement, and naming the two error types: false rejection of H0 (probability alpha) and false acceptance of H0 when H1 is true (probability beta).

The likelihood ratio test (47:18)

Drawing an analogy to Bayesian hypothesis testing, the lecture introduces the likelihood ratio, the ratio of the data's likelihood under H1 to its likelihood under H0, compared against a threshold. Without prior probabilities to set that threshold, the lecture explains that it is instead chosen by fixing the false-rejection probability alpha at a desired level, leaving beta determined by that choice.

Before you watch

• Watch Lecture 23 (Classical Statistical Inference I) first, since this lecture opens with a review of maximum likelihood estimation and confidence intervals covered there.
• Be comfortable with covariance, variance, and the normal distribution's density function.
• Review the earlier course material on Bayesian hypothesis testing, which this lecture draws on by analogy for the likelihood ratio test.

Check your understanding

1. Under what probabilistic assumptions does minimizing the sum of squared errors give the same result as maximum likelihood estimation?
2. Why can using a transformed variable like X^2 still count as a "linear" regression model?
3. What goes wrong when a model has too many parameters relative to the amount of data available?
4. Why does a strong linear relationship between two variables not establish that one causes the other?
5. How are the two types of error in hypothesis testing (alpha and beta) related, and why can't both be minimized simultaneously?