Lecture 15: PCA and ICA

What this lecture covers

The lecture continues the unsupervised learning unit by introducing principal components analysis (PCA), a non-probabilistic alternative to the factor analysis model from the previous lecture, and then begins independent components analysis (ICA). The guiding question for PCA is: if your data appears to live near a lower-dimensional subspace of a high-dimensional space, how do you find that subspace and use it to compress the data, without building a full probabilistic model of p(x)?

This builds directly on the factor analysis lecture, contrasting PCA's non-probabilistic approach with factor analysis's probabilistic one, and it assumes comfort with covariance matrices, eigenvectors, and eigenvalues. After watching, you should be able to derive PCA as the eigenvectors of a data's covariance matrix, explain why mean-centering and variance-standardization matter as preprocessing steps, choose how many components to keep, judge when PCA is a good or questionable choice for a project, and describe what problem ICA is designed to solve.

Key ideas

• PCA finds a lower-dimensional subspace: given high-dimensional but not necessarily probabilistic data, PCA finds the directions along which the data varies most and projects the data onto them.
• Two equivalent derivations: PCA can be derived either by minimizing the sum of squared distances between points and their projections, or by maximizing the variance of the projected points; the lecture uses the variance-maximization view.
• Eigenvectors of the covariance matrix: the direction that maximizes projected variance, subject to being a unit vector, is the top eigenvector of the data's covariance matrix Sigma; the top k eigenvectors define the best k-dimensional subspace.
• Preprocessing matters: subtracting the mean and standardizing each feature's variance before running PCA is necessary, since features on different scales, or data not centered at zero, will distort which direction PCA treats as principal.
• Compression and reconstruction: a k-dimensional representation y_i is computed by projecting x_i onto the top k eigenvectors, and an approximate reconstruction of x_i can be recovered as a weighted sum of those eigenvectors.
• Good vs questionable uses: visualization of high-dimensional data and compression for computational efficiency are solid uses of PCA; using it to reduce overfitting or for outlier detection by distance in a reduced space is described as unreliable, working roughly as often as it fails.
• Choosing k: a common metric is the ratio of the sum of the top k eigenvalues to the sum of all eigenvalues, often reported as the fraction of variance retained (for example, 90% to 99%).
• ICA and the cocktail party problem: given several microphone recordings, each a linear mixture of independent speaker signals, ICA recovers an unmixing matrix that separates the original sources, up to an unavoidable ambiguity in their order and sign.

Walkthrough

Motivating PCA with a correlated-features example (2:06)

The lecture opens with a simple case: measuring children's heights in both centimeters and inches produces two-dimensional data that is really only one-dimensional, since the two measurements are almost perfectly correlated, with a small amount of rounding noise. PCA is introduced as an algorithm that can discover this principal axis of variation and project the noisy two-dimensional data onto a cleaner one-dimensional line, without ever modeling a probability density. Additional motivating examples include two vibration sensors on the same machine and combining pilot-skill and pilot-enjoyment scores into a single aptitude measure.

Deriving PCA by maximizing projected variance (13:20)

After standardizing the data to zero mean and unit variance, the lecture sets up the optimization: find a unit vector U that maximizes the sum of squared lengths of the projections U^T x_i across the training set. Expanding this sum shows it equals U^T * Sigma * U, where Sigma is the data's covariance matrix. Maximizing this quantity subject to U being a unit vector, via a Lagrangian argument, shows that the maximizing U must satisfy Sigma*U = lambda*U, meaning U is an eigenvector of Sigma and the best choice is the eigenvector with the largest eigenvalue. The lecture notes this is one of several equivalent derivations of PCA, and that it is also equivalent to minimizing the sum of squared projection distances.

Reducing to k dimensions and reconstructing the data (22:32)

To project onto more than one dimension, the lecture generalizes to taking the top k eigenvectors of Sigma, each paired with its eigenvalue, and represents each example x_i with the k-dimensional vector y_i of projections onto these eigenvectors. Given the compressed representation y_i, the original high-dimensional example can be approximately reconstructed as a weighted sum of the eigenvectors using the y_i values as weights. Eigenvectors are chosen by taking those with the largest eigenvalues, and most numerical libraries already return them sorted this way.

Applications, and where PCA is a questionable choice (31:46)

Solid applications of PCA highlighted in the lecture include visualizing high-dimensional data in two or three dimensions, illustrated with a Stanford neuroscience project that reduced 50-dimensional brain electrode recordings to three dimensions to visualize a monkey planning a reaching movement, and compressing data to speed up downstream learning algorithms. The lecture is more skeptical of two other common uses: using PCA to reduce overfitting before training a classifier, and using distances in a PCA-reduced space for tasks like early face-matching systems (Eigenfaces); both are described as succeeding roughly as often as they fail, and Ng recommends regularization instead of PCA for overfitting, and testing whether a project's algorithm even needs PCA before adding it.

Choosing between PCA, factor analysis, k-means, and EM, and how many components to keep (49:11)

The lecture organizes the four unsupervised algorithms covered in the unit along two axes: whether the algorithm models a probability density p(x) and whether the data is assumed to live in a low-dimensional subspace or in clusters. This places factor analysis (probabilistic, subspace), PCA (non-probabilistic, subspace), mixture of Gaussians and EM (probabilistic, clusters or general latent-variable density estimation), and k-means (non-probabilistic, clusters) into a single framework. The lecture also addresses how many principal components to keep in practice, using the ratio of the sum of the top k eigenvalues to the sum of all eigenvalues as the fraction of variance retained, with values like 90% to 99% commonly reported after compressing high-dimensional data by a large factor.

Introducing ICA and the cocktail party problem (1:01:32)

The lecture pivots to independent components analysis, motivated by the cocktail party problem: several microphones each record an overlapping linear mixture of multiple speakers' voices, and the goal is to recover the original, separate voice signals from only the mixed recordings, played in a live audio demonstration. The model assumes observed signals x_i = A*s_i, where s_i are the true source signals and A is an unknown mixing matrix, and the algorithm's goal is to find an unmixing matrix W (approximately A inverse) that recovers the sources. The lecture closes by noting two unavoidable ambiguities in what ICA can recover: it cannot determine the original order of the sources, and it cannot determine their sign, though for audio this does not matter since a sound and its negation are indistinguishable to the ear.

Before you watch

• Review the factor analysis lecture, since this lecture explicitly contrasts PCA's non-probabilistic approach with factor analysis's probabilistic one.
• Be comfortable with covariance matrices, eigenvectors, and eigenvalues; the core PCA derivation depends on this linear algebra.
• Familiarity with the method of Lagrange multipliers is helpful for following the constrained-optimization argument, though the lecture notes it is not required to follow the result.

Check your understanding

1. Why must the data be mean-centered and variance-standardized before running PCA?
2. How does maximizing U^T * Sigma * U subject to U being a unit vector lead to the conclusion that U must be an eigenvector of the covariance matrix?
3. What does it mean to say a subspace spanned by the top k eigenvectors is stable while individual eigenvectors are not?
4. Why does the lecture recommend against using PCA to reduce overfitting, and what alternative is suggested instead?
5. What are the two sources of ambiguity in what ICA can recover from a mixed audio signal, and why does the sign ambiguity not matter in practice for sound?