Lecture 7: Let's Build GPT From Scratch

What this lecture covers

This lecture answers a concrete question: what neural network is actually running inside ChatGPT, and how do you build one yourself? Starting from an empty file, Andrej Karpathy writes a character-level GPT step by step, training it on a small text file (Tiny Shakespeare) until it generates plausible-looking Shakespeare.

This is lecture 7 of 10 in the Neural Networks: Zero to Hero series, and it assumes the tensor and language-modeling groundwork built in the earlier "makemore" videos. After watching, you'll be able to explain tokenization, self-attention, multi-head attention, residual connections, and layer normalization well enough to read the nanoGPT codebase and understand what each part of a Transformer block does.

Key ideas

• Tokenization: converting raw text into integers using a vocabulary; this lecture uses a simple character-level tokenizer instead of a subword scheme like GPT-2's byte pair encoding.
• Block size and batches: training samples fixed-length chunks (the context length) and stacks many of them into a batch so the GPU can process examples in parallel.
• Self-attention: each token produces a query (what it's looking for), a key (what it contains), and a value (what it shares); dot products between queries and keys become weights for aggregating values.
• Scaled attention: dividing the attention scores by sqrt(head_size) keeps their variance controlled so softmax doesn't collapse into near one-hot outputs at initialization.
• Multi-head attention: running several smaller self-attention heads in parallel and concatenating their outputs, so different heads can specialize in different relationships between tokens.
• Residual connections and layer norm: skip connections let gradients flow directly to early layers, and layer normalization stabilizes activations, both of which make deep Transformers trainable.
• Decoder-only vs. encoder-decoder: this lecture builds a decoder-only Transformer (like GPT) that masks future tokens; the original "Attention Is All You Need" paper also has an encoder and cross-attention for machine translation, which this build skips.

Walkthrough

Setup and the language-modeling task (0:00)

Karpathy introduces ChatGPT as a probabilistic language model and traces it back to the 2017 "Attention Is All You Need" paper, which introduced the Transformer. He explains that this lecture will build a much smaller character-level Transformer trained on the Tiny Shakespeare dataset, rather than reproducing ChatGPT itself, and previews the reference codebase, nanoGPT.

Reading data and tokenizing (7:52)

The Tiny Shakespeare file is loaded and its unique characters form a 65-character vocabulary. A simple encoder/decoder maps characters to integers and back. Karpathy contrasts this with subword tokenizers such as OpenAI's tiktoken, noting the trade-off between vocabulary size and sequence length.

Batching chunks of data (14:27)

Training never feeds the whole text at once. Instead, random fixed-length chunks (block_size) are sampled, and each chunk of length n contains n training examples, since the model must learn to predict the next character from every possible amount of preceding context. Multiple chunks are stacked into a batch dimension for parallel processing on the GPU.

The bigram baseline (22:11)

A minimal model looks up each token's embedding directly as logits for the next character, with no communication between tokens. Karpathy implements the forward pass, the cross-entropy loss, and a generate function, then trains this baseline with the Adam optimizer, watching the loss drop from around 4.7 toward 2.5.

The self-attention trick (42:13)

Before writing self-attention, Karpathy shows how to average a token's own features with the features of every earlier token, first with an explicit loop, then far more efficiently using matrix multiplication against a lower-triangular matrix. He then generalizes this from a simple average to a masked softmax over the raw dot-product scores.

Building self-attention (1:02:00)

This is the core of the lecture: each token produces a query, a key, and a value through linear layers. Attention weights come from query-key dot products, masked so future tokens can't be seen, then softmaxed and used to weight the values being aggregated. Karpathy explains attention as a data-dependent communication mechanism over a directed graph of nodes, distinguishes self-attention from cross-attention, and shows why scaling by sqrt(head_size) matters.

From one head to a full Transformer block (1:21:59)

Multiple attention heads run in parallel and concatenate their outputs. A feedforward layer is added so tokens can process what they gathered from attention. Residual connections and layer normalization are then introduced to keep the network optimizable as it gets deeper, and these pieces are grouped into a repeatable block, following the structure in the paper.

Scaling up and comparing to real GPTs (1:39:15)

With dropout added for regularization and hyperparameters scaled up (larger batch size, longer context, more layers and heads), the validation loss drops to about 1.48, producing noticeably more Shakespeare-like text. Karpathy closes by comparing this roughly 10-million-parameter model to GPT-3's 175 billion parameters, walking through nanoGPT's model.py, and outlining how pretraining differs from the supervised fine-tuning and RLHF stages used to turn a document completer into an assistant like ChatGPT.

Before you watch

• Watch the earlier "makemore" videos in this series first; this lecture assumes comfort with PyTorch tensors, nn.Module, and the autoregressive language-modeling framework.
• Basic calculus and statistics help with following the loss function and gradient discussion.
• Access to a GPU (the instructor uses an A100) is useful for reproducing the later, larger training runs, though the earlier examples run fine on a CPU.

Check your understanding

1. Why does sampling training chunks of a fixed block size produce multiple training examples from a single chunk?
2. What roles do the query, key, and value vectors play in self-attention, and how do they combine to produce an output?
3. Why is scaling the attention scores by 1/sqrt(head_size) important at initialization?
4. What problem do residual connections and layer normalization solve as Transformers get deeper?
5. What distinguishes the decoder-only Transformer built in this lecture from the encoder-decoder Transformer in the original paper?