Lecture 16: Independent Component Analysis & Reinforcement Learning

What this lecture covers

This lecture closes out CS229's unit on unsupervised learning by finishing independent component analysis (ICA), then opens the course's final major topic, reinforcement learning. The first half answers a specific question left over from the previous lecture: why must the source signals in ICA be non-Gaussian, and how do you turn an assumed source density into a maximum-likelihood algorithm that unmixes them? The second half asks a different question: how do you train a system to act well when there is no single correct answer to imitate, only a signal for when it did well or poorly?

By the end you should be able to explain why Gaussian sources make ICA unidentifiable, derive the density of the observed signal x from the density of the sources s and the unmixing matrix W, and describe how the ICA gradient-ascent update recovers a matrix that separates mixed voices. You should also be able to state the five components of a Markov decision process (MDP) and explain, using the grid-world example, what a policy and a discount factor do.

Key ideas

• Rotational ambiguity: if the sources were Gaussian, their joint distribution would be rotationally symmetric, so no unmixing matrix could tell which axis is which source; ICA therefore requires non-Gaussian sources.
• CDF vs PDF: a random variable's cumulative distribution function F(s) gives the probability of being below a value; differentiating it recovers the probability density function, and ICA is built by choosing a CDF for the sources rather than a PDF directly.
• Density transformation: because x = As = W^-1 s, the density of x is not simply the density of s evaluated pointwise; it must be scaled by the absolute value of the determinant of W to stay normalized.
• Sigmoid as source CDF: choosing the sigmoid function as the CDF of each source gives a density with fatter tails than a Gaussian, which fits real signals such as speech better than a Gaussian would; the Laplacian (double exponential) density also works.
• Maximum likelihood update: stochastic gradient ascent on the log-likelihood of W, using the derivative given in the lecture notes, converges to an unmixing matrix that recovers the original sources as s = Wx.
• State transition probabilities: P_sa gives the probability of moving to each possible next state s' after taking action a in state s; the grid-world example models a robot that only reaches its intended cell 80% of the time.
• Discount factor: gamma, a number just below 1, shrinks the weight of rewards further in the future, which both encourages getting to positive rewards sooner and makes many reinforcement learning algorithms provably converge.
• Policy: the output of a reinforcement learning algorithm is typically a policy, a mapping from states to actions, which is executed by always taking the action the policy assigns to the current state.

Walkthrough

Why ICA needs non-Gaussian sources (3:07)

The lecture reopens the cocktail-party picture from before: sources uniform between -1 and 1 can be recovered because their joint distribution has corners that pin down the correct unmixing directions. If the sources were Gaussian instead, their joint density would be rotationally symmetric, so any rotation of the axes would look equally valid, and no algorithm could recover the true speaker directions. This is stated as a general fact: ICA is only possible when the source distribution is non-Gaussian.

From CDF to the density of x (8:09)

To build the model, the lecture introduces the cumulative distribution function as an alternative to specifying a density directly, since ICA will define the source density indirectly through a CDF. Using a worked one-dimensional example (s uniform on [0,1], x = 2s), it shows that naively substituting s = Wx into the source density gives the wrong answer for continuous variables; the correct density of x must be multiplied by the absolute value of the determinant of W to remain a valid, normalized density.

Choosing a source density and deriving the ICA update (18:32)

Since Gaussian sources are ruled out, the lecture picks the sigmoid function as the CDF for each speaker's voice; its derivative gives a fatter-tailed density that better matches real signals, and the Laplacian density is offered as an equally workable alternative. Combining the per-speaker densities (assuming independence) with the determinant correction gives the density of x as a function of W. Maximizing the resulting log-likelihood by stochastic gradient ascent yields an update rule for W; after it converges, the sources are recovered as s = Wx.

ICA in practice: EEG and natural images (33:57)

The lecture shows ICA cleaning EEG recordings, treating overlapping brain signals (heartbeat, eye blinks, cognitive activity) as independent sources mixed across scalp electrodes; removing the recovered heartbeat and eye-blink components leaves a cleaner signal for downstream analysis. It also describes applying ICA to natural image patches, where the independent components come out looking like edge detectors, a result connected to earlier hierarchical sparse-coding work, including the well-known "Google cat" neural network result, and to open questions about nonlinear extensions of ICA.

Number of sources versus number of sensors (40:01)

The lecture addresses a practical question: what if the number of microphones and speakers differ? More microphones than speakers can be handled by allowing some recovered sources to come out silent, or by explicitly modeling fewer sources than sensors. Fewer microphones than speakers is described as a genuinely hard, still-active research problem, workable in limited cases (for example, separating one male and one female voice from a single microphone) but unreliable in general.

Introducing reinforcement learning (49:23)

The lecture motivates reinforcement learning with tasks like flying a helicopter or playing chess or Go, where there is no single correct action to imitate at each step, so supervised learning does not directly apply. Instead, a designer specifies a reward function that scores outcomes as good or bad (for example, +1 for a win, -1 for a loss), and the algorithm must learn by itself which actions lead to more reward. The credit assignment problem is introduced: when a late reward arrives, the algorithm has to work out which earlier actions actually caused it.

MDPs, the grid-world example, and discounting (56:29)

Reinforcement learning problems are formalized as Markov decision processes, defined by a five-tuple: states, actions, state transition probabilities, a discount factor gamma, and a reward function. Using a simple grid-world robot with an 80% chance of moving as commanded and 10% chances of slipping sideways, the lecture shows how rewards (+1 at a goal cell, -1 at a hazard cell, small negative rewards elsewhere) shape behavior, and how the discount factor pulls positive rewards earlier and pushes penalties later while also helping convergence proofs. It closes by defining a policy as a mapping from states to actions and showing the optimal policy for the example grid, where the robot sometimes takes a longer but safer route to avoid a chance of stepping into the penalty cell.

Before you watch

• Review the earlier ICA lecture's cocktail-party setup (mixing matrix A, unmixing matrix W, sources s, observations x) and its two harmless ambiguities (speaker order and sign).
• Be comfortable with the relationship between a probability density function and its cumulative distribution function.
• Recall the earlier unsupervised learning methods in this course (k-means, mixture of Gaussians and EM, factor analysis, PCA), since this lecture explicitly places ICA alongside them.
• Basic familiarity with matrix determinants and stochastic gradient ascent will make the ICA derivation easier to follow.

Check your understanding

1. Why does a Gaussian source distribution make independent component analysis impossible, while a uniform or Laplacian distribution does not?
2. In the derivation of the density of x, why can't you simply substitute s = Wx into the density of s, and what correction factor fixes this?
3. What role does the sigmoid function play in the ICA model, and why is a Gaussian CDF not a valid choice?
4. What is the credit assignment problem, and why does it make reinforcement learning harder than supervised learning?
5. In the grid-world example, why might the optimal policy choose a longer route toward the goal instead of the shortest one?