Lecture 1: Overview and Tokenization

What this lecture covers

This opening lecture explains why the course insists on building every part of a language model pipeline by hand, rather than treating language models as an API you prompt. Instructors Percy Liang and Tatsunori Hashimoto argue that as more researchers stop training or even fine-tuning models themselves, understanding of the technology is eroding, and that fundamental research still requires the ability to reason about data, systems and modeling together.

The lecture then walks through the course's structure, grading, and the five units it is organized around, before diving into the first technical topic: tokenization. It ends with a full explanation of byte pair encoding (BPE), the algorithm students implement in the first assignment. After watching, you should understand the course's guiding principle of efficiency under a compute and data budget, and know why BPE tokenization is used instead of character-, byte- or word-based schemes.

Key ideas

• The "understand it, build it" philosophy: the course teaches mechanics (how transformers and parallelism work), mindset (squeezing the most out of hardware and taking scaling seriously), and partial intuitions (what data and modeling choices help), because frontier-scale intuitions may not transfer down to smaller scale.
• The bitter lesson, reinterpreted: the right reading isn't "scale is all that matters" but "algorithms at scale is what matters" — efficiency gains compound and matter more, not less, as budgets grow.
• Levels of openness: models range from fully closed (API only) to open-weight (weights released, no data details) to open-source (weights, data and methodology all documented).
• Efficiency given a budget: the course's central question is what is the best model you can train given a fixed compute and data budget, which is meaningful at any scale.
• Tokenizer: a procedure that encodes strings into sequences of integers (tokens) and decodes them back, so text can feed into a fixed-dimension model.
• Byte pair encoding (BPE): a tokenizer trained on the corpus itself, which repeatedly merges the most frequent adjacent pair of tokens so common sequences become single tokens and rare sequences stay as several.
• Compression ratio: bytes represented per token; a good tokenizer keeps this well above 1 so sequences stay short and attention (which is quadratic in sequence length) stays cheap.

Walkthrough

Why teach language models from scratch (2:08)

Liang argues the field is in a kind of crisis: eight years ago researchers trained their own models, a few years ago they at least fine-tuned open checkpoints like BERT, and now many people only prompt proprietary APIs. That's not inherently bad — abstraction unlocks new kinds of research — but the abstraction is leaky, and fundamental research still requires tearing up the stack. Because frontier models cost hundreds of millions of dollars and use secrecy for competitive reasons, the course instead trains small models, while being explicit that small-scale behavior (such as the ratio of attention to MLP flops, or emergent abilities that only appear past a scale threshold) doesn't always predict large-scale behavior.

Efficiency, scale, and the current landscape (9:15)

The lecture reframes the "bitter lesson" as being about algorithmic efficiency at scale, not brute-force compute alone, citing an OpenAI analysis that found a roughly 44x improvement in the compute needed to reach a given ImageNet accuracy between 2012 and 2019. It then traces a brief history of language modeling, from Shannon's entropy estimates and Google's pre-2010 n-gram models, through the deep learning ingredients of the 2010s (seq2seq, Adam, attention, the 2017 transformer paper, early mixture-of-experts and model-parallel training), to the foundation-model era (ELMo, BERT, T5) and the emergence of both closed frontier APIs and open-weight and open-source alternatives.

Course logistics and workload (20:28)

The course is a five-unit class with no scaffolding code: each assignment starts from a blank file, though unit tests and adapter interfaces check correctness. Students are encouraged to prototype cheaply on a laptop before running larger jobs on a shared cluster of H100 GPUs donated by Together AI, and some assignments include a leaderboard (for example, minimizing perplexity under a fixed training budget). Grading combines passing unit tests with achieving a target loss or efficiency level, broken down by point value per sub-problem.

The five units of the course (26:36)

The lecture previews the five pillars: basics (tokenizer, transformer architecture, training loop — assignment 1); systems (GPU kernels with Triton, parallelism such as data and FSDP-style sharding, and inference with prefill/decode and speculative decoding — assignment 2); scaling laws (fitting compute-optimal curves, in the spirit of "Chinchilla optimal," to predict hyperparameters at larger scale — assignment 3); data (evaluation such as perplexity and MMLU, then curation: converting HTML to text, filtering, and deduplication — assignment 4); and alignment (supervised fine-tuning followed by learning from feedback via preference data or verifiers, using algorithms like DPO and GRPO — assignment 5). Throughout, the lecture ties each design decision back to efficiency: aggressive filtering to avoid wasting compute on bad data, tokenization instead of raw bytes to keep sequences short, and single-epoch training because the course is compute-constrained rather than data-constrained.

From strings to tokens (1:00:15)

Turning to tokenization itself, the lecture defines a tokenizer as a procedure mapping Unicode strings to sequences of integers (tokens) and back, with vocabulary size being the number of distinct token values. Using an interactive tokenizer visualizer, it shows that spaces are typically attached to the following word as part of a token, and that "hello" and " hello" are different tokens entirely.

Three failed approaches before BPE (1:04:18)

The lecture walks through why simpler schemes fall short: character-based tokenization (mapping Unicode code points directly) wastes vocabulary space because some characters are far rarer than others; byte-based tokenization keeps the vocabulary tiny (0-255) but produces very long sequences with a compression ratio of 1, which is costly given attention's quadratic cost in sequence length; and word-based tokenization (splitting on regular expressions) captures the right intuition of variable-length units but has an unbounded vocabulary, since new or rare words force an out-of-vocabulary token.

Byte pair encoding, step by step (1:11:24)

BPE, a 1994 data-compression algorithm by Philip Gage later adapted for neural machine translation and then GPT-2, resolves this by training the tokenizer on the corpus: starting from bytes, it repeatedly finds the most frequent adjacent pair of tokens and merges it into a new vocabulary entry, recording each merge. The lecture works through a toy example ("the cat in the hat") by hand, showing counts, the choice of the most frequent pair, and how the sequence shrinks as merges accumulate. Encoding new text means replaying the learned merges in order; decoding simply looks up each token's underlying bytes. GPT-2 pre-tokenizes with a word-splitting regular expression before running BPE on each piece, which is the approach used in the assignment.

Before you watch

• No prior lecture in this course to review — this is the first one.
• Basic familiarity with what a transformer and GPU training loop are will help the overview sections land, though the lecture defines terms as it goes.
• Some comfort with Python and PyTorch is assumed, since the course's assignments build a training pipeline without scaffolding.

Check your understanding

1. Why does the lecture argue that observations at small model scale (such as attention vs. MLP flop share) don't always transfer to frontier scale?
2. What is the difference between the "mechanics," "mindset," and "intuitions" the course claims it can teach, and why can only two of the three be fully taught?
3. Why does byte-based tokenization keep the vocabulary small but produce a poor compression ratio, and why does that matter for attention cost?
4. Walk through how BPE decides which pair of tokens to merge, and how that same list of merges is used during encoding of new text.
5. How does the course's "compute-constrained" framing explain specific design choices like single-epoch training and aggressive data filtering?