Lecture 9: Building the GPT Tokenizer

What this lecture covers

The lecture answers a question left open in the earlier "Let's build GPT" video: how do real language models turn text into the integer tokens a Transformer actually consumes? Instead of the toy character-level scheme used before, Andrej Karpathy builds a working byte pair encoding (BPE) tokenizer close to what GPT-2 and GPT-4 use, entirely from scratch in Python.

By the end, you can explain why tokenization is a separate pipeline stage from the language model itself, implement the training, encoding and decoding steps of BPE, and recognize why so many odd LLM behaviors (bad spelling, weak arithmetic, worse performance on non-English text) trace back to how the tokenizer chops up text.

Key ideas

• Tokenization: the process of converting strings into a sequence of integers (tokens) from a fixed vocabulary, and back again; it is a preprocessing stage trained separately from the language model.
• Byte pair encoding (BPE): an algorithm that repeatedly finds the most frequent adjacent pair of tokens in a corpus and merges it into a new token, growing the vocabulary from raw bytes up to a chosen size.
• UTF-8 byte encoding: the lecture starts from Unicode code points, then encodes them as UTF-8 bytes (0-255) so the tokenizer works on a small, stable base vocabulary before merging.
• Regex-based pre-splitting: GPT-2 uses a regular expression to split text into chunks (letters, numbers, punctuation, whitespace handled separately) before BPE, so merges never cross those category boundaries.
• Special tokens: reserved IDs such as <|endoftext|> that delimit documents or conversation turns and are handled outside the normal BPE merge logic.
• SentencePiece: an alternative library (used by LLaMA 2 and Mistral) that runs BPE on Unicode code points directly rather than bytes, with a byte-fallback for rare characters.
• Vocabulary size: a hyperparameter trading off embedding table and output layer cost against how much text each token can compress; state-of-the-art models sit roughly in the tens of thousands to about 100,000 tokens.
• Tokenization artifacts: untrained or rarely-seen tokens can produce broken model output, as shown by the "SolidGoldMagikarp" example.

Walkthrough

Why tokenization matters (0:00)

The lecture opens by contrasting the character-level tokenizer from the earlier GPT video with what production models actually use. It walks through the GPT-2 paper's input representation section, noting a vocabulary of 50,257 tokens and a context size of 1,024 tokens, and previews a list of tokenization-caused problems: bad spelling, weak arithmetic, worse non-English performance, and strange warnings that will be explained later.

Exploring tokenization live (5:50)

Using the Tiktokenizer web app, the lecture shows tokenization happening interactively: the same word tokenizes differently depending on capitalization, leading whitespace, or position in a sentence, numbers split arbitrarily across tokens, non-English text (Korean is used as an example) produces far more tokens than English for the same meaning, and Python code with heavy indentation wastes tokens on individual space characters under the GPT-2 tokenizer.

Byte pair encoding from scratch (23:50)

Before coding, the lecture explains why raw UTF-8 bytes alone are not used directly (a 256-token vocabulary would make sequences far too long for the Transformer's context window). It then walks through BPE conceptually on a short toy sequence over a four-symbol vocabulary, repeatedly finding the most frequent adjacent pair and replacing it with a new symbol, and shows how this simultaneously shrinks the sequence and grows the vocabulary.

Training the tokenizer, encoding and decoding (34:58)

The lecture implements a get_stats function to count adjacent pairs and a merge function to replace a chosen pair with a new token ID, then wraps them in a loop that performs a fixed number of merges (for example, growing from 256 to 276 tokens with 20 merges) while tracking a merges dictionary. It reports a compression ratio for the trained tokenizer, then builds decode (using a vocab lookup table plus UTF-8 decoding, with errors="replace" for invalid byte sequences) and encode (finding the earliest-trained mergeable pair at each step until no pair can be merged).

GPT-2's regex splitting rules and the tiktoken library (57:36)

Reading GPT-2's released encoder.py, the lecture explains a regex pattern that pre-splits text into chunks by category (letters, numbers, punctuation, contractions like 's or 't, and whitespace) so that BPE merges never happen across a chunk boundary; this stops nonsensical merges like dog! and dog? collapsing into near-duplicate tokens. It shows how GPT-4's tokenizer changed this pattern (case-insensitive matching, different whitespace handling, and a cap on how many digits can merge together) and introduces OpenAI's tiktoken library for fast inference-only encoding and decoding, plus how special tokens like <|endoftext|> are handled outside the regular merge logic.

SentencePiece and vocabulary size choices (1:28:42)

The lecture introduces SentencePiece, the library behind LLaMA 2 and Mistral's tokenizers, contrasting its approach (running BPE directly on Unicode code points, with byte fallback for rare characters and options like add_dummy_prefix) against tiktoken's byte-first approach, and flags SentencePiece's many legacy configuration options as a source of confusion. It then returns to the GPT architecture code to show exactly where vocabulary size affects the model: the token embedding table and the final linear ("LM head") layer, and discusses the tradeoffs of setting vocabulary size too small or too large, plus how new tokens can be added to a pretrained model by resizing these two layers.

Why weird tokenizer behavior breaks LLMs (1:51:41)

The lecture closes by revisiting the opening list of oddities and explaining each through tokenization: spelling and character-reversal failures come from long tokens bundling many characters together; arithmetic errors come from digits merging inconsistently across numbers; the "SolidGoldMagikarp" phenomenon is explained as a token that was frequent in the tokenizer's training data (from Reddit) but never appeared in the language model's own training data, leaving its embedding row untrained and producing erratic output when triggered. It ends with practical recommendations: reuse an existing tokenizer such as GPT-4's when possible, and be cautious with SentencePiece's configuration if training a custom one.

Before you watch

• Watch the earlier "Let's build GPT from scratch" video first, since this lecture builds directly on its character-level tokenizer and Transformer code.
• Basic familiarity with Python dictionaries, lists, and string encoding/decoding will make the code sections easier to follow.
• A basic sense of what Unicode code points and UTF-8 are is helpful, though the lecture explains both from the ground up.

Check your understanding

1. Why does byte pair encoding start from UTF-8 bytes instead of working directly on raw Unicode code points or on individual characters?
2. How does GPT-2's regex pre-splitting step change which token merges are possible, compared to running BPE on raw byte sequences with no splitting?
3. What causes the "SolidGoldMagikarp" behavior, and why does it only affect certain tokens?
4. Why does increasing tokenizer vocabulary size have diminishing or even negative returns beyond some point, in terms of both compute and training?
5. In what specific way does SentencePiece's handling of code points differ from tiktoken's byte-level approach, and what problem does byte fallback solve?