11. Derived Distributions (ctd.); Covariance

What this lecture covers

This lecture continues the study of derived distributions, moving from functions of a single random variable to functions of two random variables. It answers a practical question: given the joint behavior of two random variables, how do you find the distribution of a quantity built from both of them, such as their ratio or their sum? It sits in MIT 6.041's chapter on derived distributions and follows directly from the previous lecture's cookbook method (find the CDF, then differentiate).

The second half introduces covariance and the correlation coefficient, which describe how two random variables move together. After watching, you should be able to derive the density of a ratio or sum of two random variables, use the convolution formula for both discrete and continuous cases, and compute or interpret a covariance and correlation coefficient from a scatter diagram.

Key ideas

• Cookbook method for two variables: to find the density of a function of two random variables, first find its CDF as a probability of an event in the (x, y) plane, then differentiate.
• Convolution formula: for W = X + Y with X and Y independent, the PMF (or density) of W is obtained by flipping one distribution, shifting it by w, multiplying pointwise, and summing (or integrating).
• Sum of independent normals is normal: convolving two normal densities gives another normal density, with mean equal to the sum of means and variance equal to the sum of variances.
• Monotonic transformation shortcut: when y = g(x) is strictly monotonic, f_Y(y) = f_X(x) / |g'(x)|, evaluated at the corresponding x; flat regions of g produce high density in Y.
• Covariance: Cov(X,Y) = E[(X - E[X])(Y - E[Y])] measures whether large values of X tend to occur with large (positive covariance) or small (negative covariance) values of Y.
• Correlation coefficient: covariance divided by the product of the standard deviations, always between -1 and 1; a magnitude of 1 means one variable is an exact linear function of the other.
• Variance of a sum: Var(X1 + ... + Xn) equals the sum of variances plus twice the sum of pairwise covariances; the covariance terms vanish when the variables are independent.

Walkthrough

Distribution of a ratio of two uniforms (1:00)

The lecture opens with X and Y independent and uniform on the unit square, and derives the distribution of Z = Y/X. Working through the CDF geometrically (areas of triangles under lines of slope z), it shows the cumulative has one form for z between 0 and 1 and a different form for z above 1, then differentiates to get the density. A follow-up discussion shows that E[Z] is infinite here, and that you cannot compute E[Y/X] by naively dividing E[Y] by E[X] (though E[Y · (1/X)] does factor into E[Y]·E[1/X] because X and Y are independent).

Monotonic transformations from densities directly (13:14)

For a strictly monotonic function y = g(x), the lecture shows you can skip the CDF step and relate densities directly through the slope of g, using the idea that a small interval around x and its corresponding interval around y must carry equal probability. It works the example y = x^3, arriving at f_Y(y) = f_X(y^(1/3)) / (3 y^(2/3)), and explains intuitively why flat parts of g concentrate probability in Y while steep parts spread it thin.

The convolution formula, discrete then continuous (24:34)

For the sum W = X + Y of independent random variables, the lecture derives the discrete convolution formula and demonstrates a graphical technique: flip the PMF of Y, shift it by w, multiply term by term against the PMF of X, and add. It then states the continuous analogue, replacing PMFs with densities and sums with an integral.

Sum of two independent normals (33:44)

Using the convolution integral, the lecture shows that the joint density of two independent normals has elliptical contours, and that carrying out the convolution integral for their sum produces another normal density, with mean and variance additive across the two variables.

Covariance and the scatter diagram (40:55)

Using scatter diagrams, the lecture defines covariance as capturing whether large deviations in X go together with large deviations in Y (positive covariance) or with small/negative deviations (negative covariance). It notes the shortcut formula for covariance analogous to the one for variance, and that independence implies zero covariance (though not the converse).

Correlation coefficient and variance of a sum (48:14)

The lecture normalizes covariance by the standard deviations of X and Y to get the dimensionless correlation coefficient, bounded between -1 and 1, and explains that a magnitude of 1 corresponds to an exact linear relationship. It closes by extending the variance formula for a sum of random variables to include covariance cross terms.

Before you watch

• Be comfortable with the CDF-then-differentiate method for finding the distribution of a function of a single random variable, covered in the previous lecture.
• Know the definitions of variance and independence of random variables.
• Some familiarity with the normal distribution's density formula is helpful for the convolution example.

Check your understanding

1. Why does the density of Y = g(X) involve dividing by the slope of g rather than multiplying by it?
2. Walk through the convolution formula's graphical procedure (flip, shift, multiply, sum) for two discrete PMFs.
3. Why does independence between X and Y imply zero covariance, but zero covariance not imply independence?
4. If two random variables have a correlation coefficient of exactly -1, what does that tell you about their relationship?
5. How does the variance of a sum of dependent random variables differ from the variance of a sum of independent ones?