Lecture 13: Generative Models

What this lecture covers

The course shifts from supervised learning, where every previous lecture assumed labeled data, to unsupervised learning, and specifically to generative models: models that learn to produce new samples resembling a training distribution rather than predicting a label. The lecture frames generative modeling as a density estimation problem and works through three major approaches used in research at the time: PixelRNN and PixelCNN, variational autoencoders (VAEs), and generative adversarial networks (GANs).

Each approach makes a different tradeoff between tractability, sample quality, and training stability. After watching, you should be able to explain how each of the three models defines and optimizes (or avoids optimizing) a density function, describe the encoder/decoder structure shared by autoencoders and VAEs, and explain the two-player minimax game that trains a GAN.

Key ideas

• Generative model: given training data drawn from a distribution P_data, learn a model P_model that can generate new samples resembling that distribution.
• Explicit vs implicit density: some models (PixelRNN/CNN, VAE) explicitly define a density function to optimize; GANs only learn to produce samples, without ever writing down a density.
• Chain rule decomposition: PixelRNN and PixelCNN write the likelihood of an image as a product of per-pixel conditional distributions, each conditioned on previously generated pixels, making the density tractable but generation sequential and slow.
• Autoencoder: an unsupervised encoder/decoder network trained to reconstruct its input, producing a compressed feature representation Z useful for downstream tasks, but not designed to generate new data.
• Variational autoencoder: a probabilistic version of the autoencoder with a latent variable Z and a Gaussian prior; because the true data likelihood is intractable, training instead maximizes a tractable lower bound on it.
• Reconstruction term and KL term: the VAE's training objective balances reconstructing the input well against keeping the learned latent distribution close to the chosen prior.
• GAN two-player game: a generator network tries to produce images realistic enough to fool a discriminator network, while the discriminator tries to correctly classify images as real or generated.
• Minimax objective: GANs are trained by alternating gradient steps that maximize the discriminator's ability to distinguish real from fake and, using an adjusted objective for stability, maximize the generator's ability to fool it.

Walkthrough

Unsupervised learning and generative models (2:03)

The lecture contrasts supervised learning, which maps labeled data X to labels Y, with unsupervised learning, which works with unlabeled data to uncover hidden structure such as clusters, reduced-dimensionality axes of variation, learned feature representations, or an estimated density. Generative models are introduced as an unsupervised task focused specifically on density estimation: learning a model that can produce new samples from the same distribution as the training data. The lecture motivates this with applications including sample generation, super-resolution and colorization, simulation for reinforcement learning, and learning useful latent representations, and sketches a taxonomy dividing generative models into explicit-density and implicit-density families.

PixelRNN and PixelCNN: explicit tractable density (11:02)

PixelRNN models the likelihood of an image by decomposing it, via the chain rule, into a product of conditional distributions, one per pixel, each conditioned on all previously generated pixels in a fixed order starting from a corner of the image. It uses an LSTM to model these per-pixel dependencies, generating pixels one at a time, which makes both training and generation slow. PixelCNN keeps the same corner-outward generation order and tractable likelihood but replaces the RNN with a CNN applied to the already-generated context region around each pixel, which speeds up training (since ground-truth pixel values are known during training) while generation at test time remains sequential and slow. Both models are trained by maximizing the likelihood of training images under a softmax loss over pixel values, using no external labels since the training data itself supplies the target values.

Autoencoders and variational autoencoders (20:43)

Before introducing VAEs, the lecture reviews the plain autoencoder: an encoder network maps input X to a lower-dimensional feature Z, and a decoder network maps Z back to a reconstruction of X, trained with an L2 reconstruction loss and no external labels. The point of forcing Z to be smaller than X is to learn features that capture the data's most meaningful factors of variation, and the trained encoder can later be reused to initialize a supervised model. Variational autoencoders reframe this as a probabilistic generative process: data is assumed to be generated by first sampling a latent Z from a simple prior (typically Gaussian) and then sampling X from a conditional distribution P(X|Z) modeled by a decoder network. Because the resulting data likelihood involves an intractable integral over Z, training instead introduces an encoder network approximating the posterior Q(Z|X), which lets the lecture derive a tractable variational lower bound made up of a reconstruction term and a KL-divergence term keeping Q(Z|X) close to the prior.

Training a VAE and generating data (27:19)

During training, an input batch passes through the encoder to produce a distribution over Z, a sample is drawn from that distribution, then passed through the decoder to produce a distribution over X, and the loss combines the reconstruction likelihood with the KL term, all differentiable and trainable by backpropagation. At generation time, only the decoder is needed: sampling Z from the prior and passing it through the decoder produces new images. The lecture shows that varying individual dimensions of Z on models trained on MNIST and face datasets produces smooth, interpretable changes, such as head pose or amount of smile, illustrating that the latent dimensions can capture disentangled factors of variation. VAE samples are described as recognizable but noticeably blurrier than other generative approaches, which motivates ongoing work on richer posterior approximations.

Generative adversarial networks (50:15)

GANs abandon explicit density modeling entirely and instead learn to transform samples from a simple noise distribution into samples resembling the training distribution, using a generator network. Training is framed as a two-player minimax game: a discriminator network tries to correctly classify real training images as real and generator-produced images as fake, while the generator tries to produce images that fool the discriminator. The lecture shows that the generator's original minimize-log(1-D(G(z))) objective has a poor gradient signal early in training and is replaced in practice with a maximize-log(D(G(z))) objective that learns faster from bad samples. Training alternates between several discriminator gradient steps and generator gradient steps, and the lecture notes this joint training is unstable, an active research problem addressed by later work such as Wasserstein GAN. It closes with example results, including convolutional GAN architectures generating realistic bedroom images, vector arithmetic on the noise space producing semantically meaningful combinations, and applications like image-to-image domain transfer.

Before you watch

• Be comfortable with the supervised classification pipeline and loss functions covered earlier in the course, since generative models are repeatedly contrasted against that setup.
• Recall convolutional and recurrent architectures (CNNs, LSTMs) from earlier lectures, since PixelCNN and PixelRNN reuse them directly.
• Basic familiarity with probability concepts like conditional distributions, Bayes' rule, and KL divergence will help with the VAE derivation, though the lecture explains each term as it introduces it.

Check your understanding

1. Why does the chain-rule decomposition used by PixelRNN and PixelCNN make the data likelihood tractable, and what is the main practical cost of this approach?
2. What problem does the intractable integral over Z create for training a variational autoencoder directly, and how does introducing an encoder network Q(Z|X) address it?
3. What are the two terms in the VAE's variational lower bound, and what does each one encourage the model to do?
4. Explain the roles of the generator and discriminator in a GAN, and why the generator's original training objective was replaced with an alternative one.
5. Compare PixelRNN/CNN, VAEs, and GANs on tractability of the density function, sample quality, and training stability.