Lecture 2: Building a Bigram Language Model (makemore, Part 1)

What this lecture covers

This lecture starts the makemore project, a character-level language model that generates new, name-like words by learning which characters tend to follow which in a dataset of about 32,000 real names. It answers a focused question: what is the simplest possible language model, and how do you train, sample from, and evaluate one? The lecture builds a bigram model, which only ever looks at the single preceding character to predict the next one, first by directly counting character pairs and then by training an equivalent single-layer neural network with gradient descent.

This is the second lecture in the Neural Networks: Zero to Hero series, following the backpropagation engine (micrograd) built in the first. It reuses that same gradient-based machinery, now implemented with PyTorch tensors instead of scalar Value objects, and introduces the tools (tensor indexing, broadcasting, one-hot encoding, softmax, negative log likelihood) that later lectures reuse when building multi-layer perceptrons and transformers. After watching, you should be able to count bigram statistics into a table, sample from a probability distribution with torch.multinomial, and explain why a linear layer followed by a softmax and negative log likelihood loss ends up learning the exact same thing as counting.

Key ideas

• Character-level language model: predicts the next character in a sequence given the characters seen so far; makemore treats each name as a sequence with implicit start and end markers.
• Bigram model: the simplest such model, which predicts the next character using only the single previous character, ignoring everything earlier in the word.
• Counting as training: building a 27-by-27 table of how often each character follows each other character, then normalizing each row into a probability distribution, is itself a way of training a model.
• Broadcasting: PyTorch's rule for combining tensors of different shapes by aligning dimensions from the right; using it incorrectly (for example, forgetting keepdim=True when summing) silently normalizes the wrong axis and produces a working-looking but wrong result.
• Negative log likelihood: the loss function used to score the model, computed as the negative average of the log probabilities the model assigns to the actual next characters; lower is better, and zero is the theoretical minimum.
• One-hot encoding: turning a character index into a vector of zeros with a single one, so it can be fed into a neural network layer instead of being multiplied directly as an integer.
• Logits, softmax, and counts: a linear layer produces logits (interpreted as log counts); exponentiating gives count-like positive numbers, and normalizing those gives probabilities, exactly mirroring the counting approach.
• Model smoothing and regularization: adding fake counts to every bigram (or, in the neural network version, penalizing large weights) prevents the model from assigning zero probability to unseen combinations, which would otherwise produce infinite loss.

Walkthrough

Reading and exploring the dataset (3:03)

The names dataset is loaded as a list of about 32,000 lowercase words. The lecture checks basic statistics, such as the shortest and longest names, and explains that every word encodes many bigram examples, including an implicit signal for which character starts a word and which one ends it.

Counting bigrams into a 2D tensor (12:45)

After first counting bigrams in a plain Python dictionary, the lecture switches to a 27-by-27 PyTorch integer tensor, one row and column per character plus a single special token for start and end. A character-to-integer lookup table is built, and the counts are visualized with matplotlib so each cell shows a bigram and how often it occurs.

Sampling from the counting-based model (24:02)

Each row of counts is normalized into a probability distribution, and torch.multinomial is used with a seeded generator to draw a character index according to those probabilities. Repeating this, always feeding the newly sampled character back in, generates full names. The results are name-like but noticeably weak, because a bigram model has no memory beyond the single previous character.

Efficient normalization and broadcasting (36:17)

Instead of re-normalizing a row every time it's needed, the whole counts matrix is converted to probabilities at once by dividing by row sums. This requires torch.sum with the correct dim and keepdim=True; the lecture deliberately introduces a bug by dropping keepdim to show how broadcasting can silently normalize columns instead of rows, and works through PyTorch's broadcasting rules to explain why.

The loss function: negative log likelihood (50:14)

To turn the model's per-bigram probabilities into a single quality score, the lecture multiplies all the assigned probabilities together to get a likelihood, converts that product into a sum of logs for numerical convenience, and negates and averages it to get the negative log likelihood loss. It shows that an unseen bigram produces zero probability and infinite loss, which motivates adding fake counts as smoothing.

Part 2: the neural network approach (1:02:57)

The same bigram problem is reframed as a neural network task: a character index is one-hot encoded, fed through a single linear layer of 27 neurons (no bias, no nonlinearity) to produce logits, which are exponentiated and normalized (a softmax) into probabilities, exactly mirroring the counting pipeline but now built from differentiable operations.

Putting it together: training, regularization, and sampling (1:42:55)

Gradients of the loss with respect to the weight matrix are computed with loss.backward() and used to update the weights, first on a single word and then on the full 228,000-bigram dataset, converging to roughly the same loss as the counting-based model. The lecture shows that the weight matrix is mathematically the log-count table in disguise, and that penalizing large weights (an L2 regularization term added to the loss) has the same smoothing effect as adding fake counts, before sampling names from the trained network exactly as before.

Before you watch

• Complete or review Lecture 1 (micrograd), since this lecture reuses Value-style gradient descent, now expressed with PyTorch tensors.
• Be comfortable with basic Python and simple array indexing.
• A rough sense of probability (what a probability distribution is) is helpful before the counting section.

Check your understanding

1. Why does normalizing counts row by row produce a valid probability distribution for the next character?
2. What goes wrong if you sum a tensor along the wrong dimension or forget keepdim=True, and why does the result still "run" without an error?
3. Why is an unseen bigram assigned a probability of zero, and how does smoothing fix that?
4. In what sense are the trained neural network's weights equivalent to the table of bigram counts?
5. Why is negative log likelihood used as the loss instead of, say, mean squared error, for this classification-style problem?