Lecture 7: Discrete Random Variables III

What this lecture covers

This lecture closes out the chapter on discrete random variables in MIT 6.041, the last lecture before the first quiz. It generalizes the joint and conditional PMF notation from two random variables to three or more, defines independence of random variables, and reviews conditional independence, before turning to expectation and variance: linearity of expectation, when expectation of a product factors, and when variance of a sum adds.

After watching, you'll be able to write joint and conditional PMFs for multiple random variables, test independence from a table, and use the indicator-variable trick to compute the mean and variance of the binomial distribution and the classic hat-check problem without brute-force summation.

Key ideas

• Joint, marginal and conditional PMFs are the same old probabilities in new notation: the multiplication rule and the law of total probability carry over directly from events to random variables.
• Independence of random variables: three random variables are independent if and only if their joint PMF factors into the product of the marginal PMFs, for every combination of values.
• Conditional independence: independence can hold inside a restricted universe (after conditioning on an event) even when it doesn't hold unconditionally, or vice versa.
• Linearity of expectation: E[X+Y+Z] = E[X] + E[Y] + E[Z] always holds, with no independence assumption required.
• Expectation of a product needs independence: E[XY] = E[X]E[Y] only when X and Y are independent, and the same holds for E[g(X)h(Y)].
• Variance of a sum needs independence: Var(X+Y) = Var(X) + Var(Y) only holds when X and Y are independent; dependent variables can make the sum's variance much larger or smaller.
• Indicator variables: writing a complicated random variable as a sum of simple 0/1 variables turns a hard direct calculation into an easy application of linearity.

Walkthrough

Joint, conditional and marginal PMFs for multiple variables (1:28)

The lecture reviews the joint PMF of two random variables as the probability that both take specific values simultaneously, and the conditional PMF as a probability distribution over one variable given a fixed value of the other. It walks through how to recover the marginal PMF of X by summing the joint PMF over all values of Y, and how the multiplication rule connects the joint, marginal and conditional PMFs, mirroring the identities already known for events.

Conditional probability and generalizing to three variables (2:30)

The same ideas extend to three random variables X, Y, Z: the joint PMF gives the probability all three take specific values, the marginal of X is found by summing over all y and z, and a three-variable multiplication rule breaks the joint probability into a chain of conditional probabilities, exactly as with three events.

Independence of random variables (9:00)

Independence of several random variables is defined by a single condition: the joint PMF factors into the product of the individual marginal PMFs for every possible combination of values. The lecture stresses this compact definition versus the many conditions needed for independence of multiple events, and gives the intuitive reading: knowing the value of some variables doesn't change your beliefs about the others.

Conditional independence example (15:07)

A worked table example shows two variables that are not independent overall but become independent once you condition on a specific event, illustrated by checking whether the conditional joint PMF factors into conditional marginals within the restricted universe.

Expectation: linearity and products of independent variables (19:28)

Expectation is introduced as a long-run average, computable directly from a joint PMF without first finding a marginal PMF. Linearity of expectation is shown to hold generally, illustrated with SAT section scores. The lecture then proves that for independent X and Y, E[XY] = E[X]E[Y], and extends this, via an information argument, to E[g(X)h(Y)] = E[g(X)]E[h(Y)].

Variance rules (27:15)

The lecture derives Var(aX) = a^2 Var(X) and shows that adding a constant leaves variance unchanged. It then shows Var(X+Y) = Var(X) + Var(Y) holds only for independent variables, using the extreme cases Y=X and Y=-X to show why dependence breaks additivity, and closes with a drill computing the variance of a linear combination like X - 3Y.

Mean and variance of the binomial via indicator variables (31:44)

Rather than summing the binomial PMF directly, the number of successes in n independent trials is written as a sum of 0/1 indicator variables, one per trial. Linearity of expectation immediately gives E[X] = np, and the variance shortcut Var(Xi) = E[Xi^2] - E[Xi]^2 gives Var(X) = np(1-p), with the shape of p(1-p) explaining why a fair coin is the most uncertain.

The hat-check problem (40:05)

The lecture applies the same indicator trick to a harder, dependent case: n people randomly grab hats, and X counts how many get their own hat back. Even though the indicator variables are dependent, linearity of expectation still gives E[X] = 1. Computing the variance requires expanding X^2 into indicator terms and cross-terms Xi*Xj, using the probability that two specific people both get their own hats, which works out so that Var(X) = 1 as well.

Before you watch

• Review the joint PMF of two random variables and the multiplication rule from the previous lecture in this course.
• Be comfortable with the definitions of expectation and variance for a single discrete random variable.
• Know the binomial distribution's PMF formula from earlier in the course.

Check your understanding

1. Why does linearity of expectation hold even when the random variables being summed are dependent?
2. How would you check, from a joint PMF table, whether three random variables are independent?
3. Why does the hat-check problem require calculating E[Xi * Xj] for i != j to find the variance, while the binomial case does not?
4. What does the shape of p(1-p) tell you about when a coin flip has the most uncertainty?