Seyed Masoud Hosseini · Overview · Study log · Weekly summaries · Ideas · Search · Transcript · RSS feed

Neural Networks: Zero to Hero · Lecture 2 of 10 · 1:57:45

Lecture 2: Building a Bigram Language Model (makemore, Part 1)

The spelled-out intro to language modeling: building makemore on YouTube

Study guide

What this lecture covers

This lecture starts the makemore project, a character-level language model that generates new, name-like words by learning which characters tend to follow which in a dataset of about 32,000 real names. It answers a focused question: what is the simplest possible language model, and how do you train, sample from, and evaluate one? The lecture builds a bigram model, which only ever looks at the single preceding character to predict the next one, first by directly counting character pairs and then by training an equivalent single-layer neural network with gradient descent.

This is the second lecture in the Neural Networks: Zero to Hero series, following the backpropagation engine (micrograd) built in the first. It reuses that same gradient-based machinery, now implemented with PyTorch tensors instead of scalar Value objects, and introduces the tools (tensor indexing, broadcasting, one-hot encoding, softmax, negative log likelihood) that later lectures reuse when building multi-layer perceptrons and transformers. After watching, you should be able to count bigram statistics into a table, sample from a probability distribution with torch.multinomial, and explain why a linear layer followed by a softmax and negative log likelihood loss ends up learning the exact same thing as counting.

Key ideas

  • Character-level language model: predicts the next character in a sequence given the characters seen so far; makemore treats each name as a sequence with implicit start and end markers.
  • Bigram model: the simplest such model, which predicts the next character using only the single previous character, ignoring everything earlier in the word.
  • Counting as training: building a 27-by-27 table of how often each character follows each other character, then normalizing each row into a probability distribution, is itself a way of training a model.
  • Broadcasting: PyTorch's rule for combining tensors of different shapes by aligning dimensions from the right; using it incorrectly (for example, forgetting keepdim=True when summing) silently normalizes the wrong axis and produces a working-looking but wrong result.
  • Negative log likelihood: the loss function used to score the model, computed as the negative average of the log probabilities the model assigns to the actual next characters; lower is better, and zero is the theoretical minimum.
  • One-hot encoding: turning a character index into a vector of zeros with a single one, so it can be fed into a neural network layer instead of being multiplied directly as an integer.
  • Logits, softmax, and counts: a linear layer produces logits (interpreted as log counts); exponentiating gives count-like positive numbers, and normalizing those gives probabilities, exactly mirroring the counting approach.
  • Model smoothing and regularization: adding fake counts to every bigram (or, in the neural network version, penalizing large weights) prevents the model from assigning zero probability to unseen combinations, which would otherwise produce infinite loss.

Walkthrough

Reading and exploring the dataset (3:03)

The names dataset is loaded as a list of about 32,000 lowercase words. The lecture checks basic statistics, such as the shortest and longest names, and explains that every word encodes many bigram examples, including an implicit signal for which character starts a word and which one ends it.

Counting bigrams into a 2D tensor (12:45)

After first counting bigrams in a plain Python dictionary, the lecture switches to a 27-by-27 PyTorch integer tensor, one row and column per character plus a single special token for start and end. A character-to-integer lookup table is built, and the counts are visualized with matplotlib so each cell shows a bigram and how often it occurs.

Sampling from the counting-based model (24:02)

Each row of counts is normalized into a probability distribution, and torch.multinomial is used with a seeded generator to draw a character index according to those probabilities. Repeating this, always feeding the newly sampled character back in, generates full names. The results are name-like but noticeably weak, because a bigram model has no memory beyond the single previous character.

Efficient normalization and broadcasting (36:17)

Instead of re-normalizing a row every time it's needed, the whole counts matrix is converted to probabilities at once by dividing by row sums. This requires torch.sum with the correct dim and keepdim=True; the lecture deliberately introduces a bug by dropping keepdim to show how broadcasting can silently normalize columns instead of rows, and works through PyTorch's broadcasting rules to explain why.

The loss function: negative log likelihood (50:14)

To turn the model's per-bigram probabilities into a single quality score, the lecture multiplies all the assigned probabilities together to get a likelihood, converts that product into a sum of logs for numerical convenience, and negates and averages it to get the negative log likelihood loss. It shows that an unseen bigram produces zero probability and infinite loss, which motivates adding fake counts as smoothing.

Part 2: the neural network approach (1:02:57)

The same bigram problem is reframed as a neural network task: a character index is one-hot encoded, fed through a single linear layer of 27 neurons (no bias, no nonlinearity) to produce logits, which are exponentiated and normalized (a softmax) into probabilities, exactly mirroring the counting pipeline but now built from differentiable operations.

Putting it together: training, regularization, and sampling (1:42:55)

Gradients of the loss with respect to the weight matrix are computed with loss.backward() and used to update the weights, first on a single word and then on the full 228,000-bigram dataset, converging to roughly the same loss as the counting-based model. The lecture shows that the weight matrix is mathematically the log-count table in disguise, and that penalizing large weights (an L2 regularization term added to the loss) has the same smoothing effect as adding fake counts, before sampling names from the trained network exactly as before.

Before you watch

  • Complete or review Lecture 1 (micrograd), since this lecture reuses Value-style gradient descent, now expressed with PyTorch tensors.
  • Be comfortable with basic Python and simple array indexing.
  • A rough sense of probability (what a probability distribution is) is helpful before the counting section.

Check your understanding

  1. Why does normalizing counts row by row produce a valid probability distribution for the next character?
  2. What goes wrong if you sum a tensor along the wrong dimension or forget keepdim=True, and why does the result still "run" without an error?
  3. Why is an unseen bigram assigned a probability of zero, and how does smoothing fix that?
  4. In what sense are the trained neural network's weights equivalent to the table of bigram counts?
  5. Why is negative log likelihood used as the loss instead of, say, mean squared error, for this classification-style problem?

Vocabulary

language model (noun)
A system that learns to predict which word or character comes next in text.
makemore trains a small language model on a list of names.
character-level (adjective)
Working one letter at a time, instead of one whole word at a time.
This is a character-level model, so it predicts single letters.
sequence (noun)
A list of items placed one after another in order.
Each name is treated as a sequence of characters.
bigram (noun)
A pair of two characters (or words) that appear next to each other.
The model counts every bigram in the names dataset.
sample (verb)
To randomly pick an outcome according to given probabilities.
We sample a new character from the probability row.
count (verb)
To record how many times something happens.
We count how often each character follows another.
table (noun)
A grid of numbers arranged in rows and columns.
The bigram counts are stored in a 27-by-27 table.
normalize (verb)
To rescale numbers so they add up to one, like a probability.
We normalize each row so it becomes a probability distribution.
probability distribution (noun)
A set of numbers, adding up to one, that show how likely each outcome is.
Each row of the table becomes a probability distribution over the next character.
tensor (noun)
A grid of numbers of any number of dimensions, used by PyTorch to store data.
The counts are stored in a PyTorch tensor.
indexing (noun)
Picking out specific elements from a tensor or list by position.
Tensor indexing lets us pull out one row of the counts table.
broadcasting (noun)
PyTorch's rule for combining tensors of different shapes by matching their dimensions.
Broadcasting divides each row by its own sum without a loop.
dimension (noun)
One axis of a tensor, such as rows or columns.
We sum along the row dimension of the counts tensor.
row sum (noun)
The total you get by adding up all numbers in one row.
Dividing each count by its row sum gives probabilities.
one-hot encoding (noun)
Turning a category into a list of zeros with a single one marking that category.
One-hot encoding turns a character index into a vector for the network.
linear layer (noun)
A network layer that multiplies inputs by weights and adds them up, with no extra transformation.
A single linear layer maps the input to 27 output numbers.
logits (noun)
The raw, unscaled numbers a network outputs before they are turned into probabilities.
The linear layer produces logits that behave like log counts.
softmax (noun)
A function that turns a list of numbers into probabilities that add up to one.
Softmax turns the logits into a probability over the next character.
exponentiate (verb)
To raise a number to a power, especially using e as the base.
We exponentiate the logits to get positive, count-like numbers.
negative log likelihood (noun)
A loss that measures how unlikely the correct answers were, based on the model's probabilities.
We compute the negative log likelihood over all the bigrams.
likelihood (noun)
How probable the observed data is, according to the model.
Multiplying all the assigned probabilities gives the likelihood.
smoothing (noun)
A technique of adding small fake counts so no outcome gets zero probability.
Smoothing stops unseen bigrams from getting infinite loss.
regularization (noun)
A technique that discourages a model from fitting the data too exactly, to keep it more general.
L2 regularization penalizes large weights in the network.
differentiable (adjective)
Able to have a derivative calculated, so gradients can flow through it.
The whole pipeline is made of differentiable operations.
converge (verb)
To gradually settle toward a stable, final value.
Training converges to roughly the same loss as the counting method.
in disguise (phrase)
Being the same thing as something else, just presented differently.
The trained weight matrix is the log-count table in disguise.

Chapters

From the YouTube description

We implement a bigram character-level language model, which we will further complexify in followup videos into a modern Transformer language model, like GPT. In this video, the focus is on (1) introducing torch.Tensor and its subtleties and use in efficiently evaluating neural networks and (2) the overall framework of language modeling that includes model training, sampling, and the evaluation of a loss (e.g. the negative log likelihood for classification).

Links:
- makemore on github: https://github.com/karpathy/makemore
- jupyter notebook I built in this video: https://github.com/karpathy/nn-zero-to-hero/blob/master/lectures/makemore/makemore_part1_bigrams.ipynb
- my website: https://karpathy.ai
- my twitter: https://twitter.com/karpathy
- (new) Neural Networks: Zero to Hero series Discord channel: https://discord.gg/3zy8kqD9Cp , for people who'd like to chat more and go beyond youtube comments

Useful links for practice:
- Python + Numpy tutorial from CS231n https://cs231n.github.io/python-numpy-tutorial/ . We use torch.tensor instead of numpy.array in this video. Their design (e.g. broadcasting, data types, etc.) is so similar that practicing one is basically practicing the other, just be careful with some of the APIs - how various functions are named, what arguments they take, etc. - these details can vary.
- PyTorch tutorial on Tensor https://pytorch.org/tutorials/beginner/basics/tensorqs_tutorial.html
- Another PyTorch intro to Tensor https://pytorch.org/tutorials/beginner/nlp/pytorch_tutorial.html

Exercises:
E01: train a trigram language model, i.e. take two characters as an input to predict the 3rd one. Feel free to use either counting or a neural net. Evaluate the loss; Did it improve over a bigram model?
E02: split up the dataset randomly into 80% train set, 10% dev set, 10% test set. Train the bigram and trigram models only on the training set. Evaluate them on dev and test splits. What can you see?
E03: use the dev set to tune the strength of smoothing (or regularization) for the trigram model - i.e. try many possibilities and see which one works best based on the dev set loss. What patterns can you see in the train and dev set loss as you tune this strength? Take the best setting of the smoothing and evaluate on the test set once and at the end. How good of a loss do you achieve?
E04: we saw that our 1-hot vectors merely select a row of W, so producing these vectors explicitly feels wasteful. Can you delete our use of F.one_hot in favor of simply indexing into rows of W?
E05: look up and use F.cross_entropy instead. You should achieve the same result. Can you think of why we'd prefer to use F.cross_entropy instead?
E06: meta-exercise! Think of a fun/interesting exercise and complete it.

Chapters:
00:00:00 intro
00:03:03 reading and exploring the dataset
00:06:24 exploring the bigrams in the dataset
00:09:24 counting bigrams in a python dictionary
00:12:45 counting bigrams in a 2D torch tensor ("training the model")
00:18:19 visualizing the bigram tensor
00:20:54 deleting spurious (S) and (E) tokens in favor of a single . token
00:24:02 sampling from the model
00:36:17 efficiency! vectorized normalization of the rows, tensor broadcasting
00:50:14 loss function (the negative log likelihood of the data under our model)
01:00:50 model smoothing with fake counts
01:02:57 PART 2: the neural network approach: intro
01:05:26 creating the bigram dataset for the neural net
01:10:01 feeding integers into neural nets? one-hot encodings
01:13:53 the "neural net": one linear layer of neurons implemented with matrix multiplication
01:18:46 transforming neural net outputs into probabilities: the softmax
01:26:17 summary, preview to next steps, reference to micrograd
01:35:49 vectorized loss
01:38:36 backward and update, in PyTorch
01:42:55 putting everything together
01:47:49 note 1: one-hot encoding really just selects a row of the next Linear layer's weight matrix
01:50:18 note 2: model smoothing as regularization loss
01:54:31 sampling from the neural net
01:56:16 conclusion

← Lecture 1: The Spelled-Out Intro to Neural Networks and Backpropagation · Lecture 3: Building an MLP Character-Level Language Model →