Lecture 25: Classical Inference III

What this lecture covers

This final lecture of the course's statistics unit is a tour through hypothesis testing examples of increasing complexity, closing the classical inference material started two lectures earlier. It starts by formalizing the likelihood ratio test: the general recipe of computing a likelihood ratio statistic, choosing the shape of the rejection region from that ratio, and then picking a threshold so the false-rejection probability equals a chosen significance level such as 5%.

The lecture then works through several concrete cases: distinguishing two normal distributions with different means, distinguishing two normal distributions with different variances (introducing the chi-square distribution), testing whether a coin is fair, testing whether a die is fair (the chi-square goodness-of-fit test), and testing whether data follow an assumed continuous distribution using the empirical CDF (the Kolmogorov-Smirnov idea). It closes with broader reflections on the "art" involved in statistics, the difference between failing to reject and proving a hypothesis, and a caution about selective reporting and multiple testing, illustrated by the well-known paper "Why Most Published Research Findings Are False."

Key ideas

• Likelihood ratio test structure: computing the ratio of data likelihoods under two hypotheses gives a single statistic; committing to this test reduces a complicated multi-dimensional decision region to a one-dimensional threshold problem.
• Statistic: a scalar (or low-dimensional) summary of the data, such as a sum, sample variance, or count, used as the basis for a decision instead of the raw data.
• Alpha and beta trade-off: alpha is the probability of falsely rejecting a true null hypothesis, beta is the probability of failing to reject a false null hypothesis, and moving the decision threshold trades one against the other; likelihood ratio tests achieve the best possible trade-off for a given alpha.
• Chi-square distribution: arises as the distribution of a sum of squared normal random variables, and is used to set thresholds in tests such as comparing variances or testing goodness of fit.
• Coin- and die-fairness tests: reduce the data to a simple statistic (number of heads, or a sum-of-squared-deviations statistic across die faces) and compare it to a threshold chosen from its approximate (often normal or chi-square) distribution under the null hypothesis.
• "Not rejected" versus "proved": failing to reject a null hypothesis means the data are compatible with it, not that the hypothesis has been proven true; only evidence that contradicts a hypothesis can falsify it.
• Empirical CDF comparison: to test whether data follow an assumed continuous distribution, compare the assumed CDF to the empirical CDF built from the data, using the maximum distance between them as the test statistic.
• Multiple testing and publication bias: running many hypothesis tests, or choosing which hypothesis to test after seeing the data, inflates the true rate of false discoveries well beyond the nominal significance level of any single test.

Walkthrough

Formalizing the likelihood ratio test and the alpha-beta trade-off (2:00)

The lecture reviews how a likelihood ratio statistic reduces the problem of dividing the data space into acceptance and rejection regions to choosing a single threshold, and explains the fundamental trade-off between the two error probabilities, alpha (false rejection) and beta (false acceptance), noting that likelihood ratio tests give the best achievable trade-off for a fixed alpha.

Comparing two normal distributions with different means (10:13)

Working out the likelihood ratio for two normal distributions differing only in mean shows that the decision reduces to comparing the sum of the observed data to a threshold, which is then found using the normal tables so the false-rejection probability equals a chosen alpha.

Comparing two normal distributions with different variances (13:19)

For distributions differing in variance, the likelihood ratio test reduces to comparing the sum of squared observations to a threshold. Because this statistic follows a chi-square distribution, the lecture introduces chi-square tables as the standard way to locate the required threshold.

Testing whether a coin is fair (18:23)

With an alternative hypothesis covering many possible biases rather than one specific value, the lecture reduces the data to the number of observed heads, treats it as approximately normal by the central limit theorem, and rejects fairness if the count is an outlier relative to the expected value under a fair coin.

Testing whether a die is fair (28:38)

Extending the idea to six outcomes, the lecture defines a chi-square-style statistic summing squared deviations of observed counts from expected counts, scaled by the expected counts, and explains that its distribution (derived and tabulated by others) sets the threshold for a chosen significance level.

Testing whether data follow an assumed distribution (35:48)

The lecture generalizes further to testing whether continuous data match an assumed PDF, first by discretizing into a histogram and reusing the chi-square approach, then by comparing the assumed CDF to the data's empirical CDF and using the maximum gap between the two curves as a test statistic with its own tabulated threshold.

Reflections on statistics as an art, and the risk of selective testing (44:06)

The lecture closes by noting how much judgment goes into choosing statistics and rejection-region shapes, contrasts the solidity of probability theory with the messiness of applied statistics, and warns that running many hypothesis tests or selecting hypotheses after seeing the data inflates the real rate of false conclusions well beyond any single test's stated significance level.

Before you watch

• Watch Lecture 24 (Classical Inference II) first, since it introduces the likelihood ratio test structure this lecture builds on.
• Review the normal distribution tables and the basic setup of a hypothesis test (null versus alternative hypothesis, rejection region).
• Be comfortable with the central limit theorem, since several tests rely on approximating a statistic as normally distributed.

Check your understanding

1. Why does committing to a likelihood ratio test simplify the problem of designing a rejection region?
2. What is the trade-off between alpha and beta, and why can't both be made arbitrarily small at the same time?
3. Why does failing to reject a null hypothesis not mean the hypothesis has been proven true?
4. How does the chi-square distribution arise in the die-fairness test, and why is it useful there?
5. Why does running many hypothesis tests, or choosing a hypothesis after looking at the data, increase the real risk of a false conclusion beyond the stated significance level of any one test?