Lecture 9: Scaling laws 1

What this lecture covers

This lecture asks why you can predict how a language model will perform before training it at full scale, and how to use that predictability to make engineering decisions cheaply. It starts the scaling laws unit, following the systems lectures on parallelism, and moves the discussion from 'how do I train efficiently' to 'how do I decide what to train.'

The lecture covers the history of scaling laws in statistical learning theory, walks through why power-law data scaling is a natural mathematical consequence of estimation problems, surveys empirical scaling results for architecture and optimizer choices, and works through the Chinchilla analysis for trading off model size against training data under a fixed compute budget. After watching, you should be able to explain why scaling curves are log-log linear, use small-scale experiments to make hyperparameter and architecture decisions, and describe the three methods Chinchilla used to estimate the compute-optimal token-to-parameter ratio.

Key ideas

• Scaling laws as extrapolation: train many small, cheap models across orders of magnitude of compute, fit a predictable curve, and extrapolate to decide what to do at large scale, instead of tuning directly on expensive large models.
• Power-law data scaling: on a log-log plot, test loss decreases linearly with data set size; this is explained by simple estimation problems (like estimating a Gaussian mean or a nonparametric regression function) where the error naturally decays as a power of the sample size, with the exponent tied to the problem's effective dimensionality.
• Model and architecture comparisons via scaling curves: rather than training two architectures up to full scale, you train both across a compute sweep and compare curves; a consistent constant-factor gap (e.g. transformers over LSTMs) indicates a real advantage that should persist at scale.
• Non-embedding parameters: scaling laws are cleaner when only non-embedding parameters are counted, since embedding parameters behave differently.
• Critical batch size: below a threshold, doubling batch size is as good as taking more gradient steps; beyond it, returns diminish sharply, and this critical point shrinks as the loss target gets smaller, meaning batch size can grow over training.
• muP-style reparameterization: scaling initialization and learning rates with model width can make the optimal learning rate transfer from small to large models without retuning, as an alternative to fitting a scaling law for the learning rate itself.
• Chinchilla compute-optimal scaling: for a fixed compute budget, there is a joint optimum of model size and token count (roughly 20 tokens per parameter in the original result), estimated by three independent methods that largely agreed once fit correctly.
• Perplexity scaling versus downstream task scaling: scaling laws are well-behaved for cross-entropy loss but far less predictable for downstream benchmark accuracy, so results should not be assumed to transfer directly.

Walkthrough

The motivation for scaling laws (0:04)

The lecture frames scaling laws as the alternative to expensively tuning large models directly: train small models cheaply, learn what to do, and extrapolate to the large run you actually care about.

History and theoretical grounding (2:04)

It traces scaling laws back to classical statistical learning theory (generalization bounds, nonparametric rates) and early empirical papers from the 1990s through 2017 (including Hestness et al.), showing that predictable power-law behavior in model performance was observed long before recent large language models.

Why power laws are natural: two worked examples (15:13)

The lecture derives the scaling exponent for estimating a Gaussian mean (error proportional to 1/n) and for nonparametric regression in d dimensions (error proportional to n^(-1/d)), showing that the slope of a scaling law on a log-log plot reflects the effective dimensionality of the learning problem, which explains why empirical exponents are much smaller than naive 1/n rates.

Using data scaling laws for engineering decisions (20:16)

Examples include using scaling laws to evaluate data mixtures at small scale (since composition mainly shifts the offset, not the slope), studying diminishing returns from repeating data across multiple epochs, and trading off repeating high-quality data against including new, lower-quality data.

Model and hyperparameter scaling (25:18)

The lecture compares transformers against LSTMs, alternative architectures against a transformer baseline, and Adam against SGD, all using compute-scaling curves rather than single large runs. It also covers depth-to-width aspect ratio scaling and the importance of excluding embedding parameters from parameter counts for clean scaling behavior.

Batch size, learning rate, and muP (34:25)

It introduces critical batch size as the point where increasing batch size stops giving proportional optimization benefit, notes that this threshold shrinks as the loss target improves, and contrasts standard practice of retuning learning rate per model width with muP-style reparameterization that aims to keep the optimal learning rate stable across scale.

Perplexity versus downstream scaling (43:28)

The lecture cautions that while training loss (perplexity) scales predictably with compute regardless of architecture details, downstream benchmark performance often does not follow the same clean relationship, so scaling-law conclusions from loss should be checked against actual downstream behavior.

Chinchilla: compute-optimal data and model size (46:29)

The lecture explains the joint data-model scaling problem, introduces functional forms from Rosenfeld and Kaplan, and walks through Chinchilla's three estimation methods (minimum envelope of training curves, isoFLOP curves, and parametric curve fitting), which converge on similar compute-optimal ratios once corrected for a curve-fitting error later identified by a replication from Epoch AI.

Inference cost and the shift beyond Chinchilla (1:00:39)

The lecture closes by noting that as language models became deployed products, the token-to-parameter ratio has grown far beyond the original Chinchilla recommendation, because minimizing inference cost now matters alongside minimizing training compute, and gives a brief example of the same scaling methodology applied successfully to diffusion language models.

Before you watch

• Familiarity with the course's earlier architecture and hyperparameter lectures is helpful, since this lecture revisits aspect ratio and hyperparameter choices through a scaling-law lens.
• Basic statistics (variance, estimation error) is assumed for the derivation of power-law exponents.
• Awareness of cosine learning rate schedules is useful for understanding why Chinchilla's estimates differed from Kaplan's.

Check your understanding

1. Why does test loss appear linear when plotted against data set size on a log-log plot?
2. What does the exponent of a data scaling law tell you about the difficulty or dimensionality of a learning problem?
3. How does the critical batch size change as a model's target loss decreases, and why does that matter for training schedules?
4. What are the three methods Chinchilla used to estimate the compute-optimal ratio of tokens to parameters, and why did method three initially disagree with the others?
5. Why has the tokens-per-parameter ratio grown well beyond Chinchilla's original recommendation in recent models?