Lecture 11: Scaling laws 2

What this lecture covers

This second scaling-laws lecture moves from theory to practice: how do real labs actually decide on hyperparameters, batch sizes, and learning rates when training a large model? The lecture works through three case studies (Cerebras-GPT, MiniCPM, and DeepSeek LLM) that each published unusually detailed scaling methodology, plus briefer notes on Llama 3, Hunyuan, and MiniMax-01. The second half is a from-scratch derivation of maximal update parameterization (MuP), the technique that lets a model keep the same optimal learning rate as it gets wider.

This is lecture 11 of Stanford's CS336, following directly from the first scaling-laws lecture. After watching, you should be able to describe the warm-up-stable-decay (WSD) learning rate schedule and why it enables near-free Chinchilla-style analysis, explain what MuP changes about initialization and per-layer learning rates, and recognize the common ingredients (isoFLOP fits, small-scale hyperparameter sweeps, learning rate stability) that recur across published scaling recipes.

Key ideas

• Chinchilla ratio drift: the "20 tokens per parameter" rule from Chinchilla is not a strict constraint; MiniCPM, Llama 3, and Hunyuan all fit substantially higher token-to-parameter ratios, suggesting the ratio depends on architecture and data quality.
• MuP (maximal update parameterization): a way of scaling initialization variance and per-layer learning rates with model width so that the optimal learning rate stays roughly constant as models get bigger.
• Warm-up-stable-decay (WSD) schedule: a trapezoid-shaped learning rate schedule (warm-up, flat stable phase, rapid decay) that lets one training run be rewound and re-decayed at different data budgets, replacing the need for many separate runs when fitting Chinchilla-style data-scaling laws.
• Critical batch size: the point of diminishing returns for batch size at a given model scale and loss target, itself predictable via a scaling law, as shown in MiniCPM's replication of the Kaplan-style batch size analysis.
• DeepSeek's direct approach: rather than using MuP, DeepSeek LLM fits scaling laws directly to the optimal batch size and learning rate across compute scales, betting on the stability of those fits.
• Spectral conditions behind MuP: at initialization, per-coordinate activations should stay order-one as width grows, and after one gradient step, the change in activations should also stay order-one; these two conditions determine the initialization scale and the per-layer learning rate scale.
• MuP's empirical robustness: a later ablation study found MuP's learning-rate transfer holds across nonlinearities, batch sizes, and most initialization choices, but breaks under learnable normalization gains, unusual optimizers like Lion, and strong weight decay.

Walkthrough

Why scaling recipes matter and what changed after ChatGPT (0:05)

The lecture opens by noting that after ChatGPT's release, frontier labs became much more secretive about their scaling methodology, so the class has to rely on a small number of papers that were unusually candid: Cerebras-GPT, MiniCPM, and DeepSeek LLM, with lighter coverage of Llama 3, Hunyuan, and MiniMax-01. The motivating question throughout is how to choose hyperparameters, especially learning rate and batch size, that remain sensible as a model scales up, since retuning them at full scale is prohibitively expensive.

Cerebras-GPT and the case for MuP (5:10)

Cerebras-GPT trains a family of 0.1B to 13B parameter models using the Chinchilla token-to-parameter recipe, and its key contribution is validating MuP at scale: models trained with MuP show smoother, more predictable scaling curves than standard parameterization, which oscillates around the predicted scaling line. The lecture shows Cerebras's reference table for MuP, where non-embedding parameters are initialized proportional to 1/width and per-layer learning rates are scaled down by 1/width, and describes how Cerebras tunes hyperparameters aggressively at a small proxy scale (down to 40 million parameters) before scaling up.

MiniCPM's careful scaling recipe (11:10)

MiniCPM aimed to train small but very well-trained language models, and used MuP-style scaling for initialization and per-layer learning rates similarly to Cerebras. The lecture walks through MiniCPM's replication of the Kaplan-style critical batch size analysis, fitting a predictable relationship between target loss and optimal batch size, and shows that the optimal learning rate stays stable across model sizes once MuP is applied, evidenced by the minimum of the loss-versus-learning-rate curve landing at the same point across small and large models.

The WSD schedule and cheap data-scaling analysis (20:11)

A central problem in Chinchilla-style analysis is that a cosine learning rate schedule is shaped differently depending on the training target, so early checkpoints from one run cannot be reused to study a different data budget. MiniCPM popularized the warm-up-stable-decay (WSD) schedule to fix this: a long flat "stable" phase can be reused, and rewound checkpoints can each be given their own rapid decay phase to approximate what a full cosine run to that token count would have produced, turning what would need many separate runs into close to one run. Using this, MiniCPM derives a token-to-parameter ratio much higher than Chinchilla's 20:1, though the lecture flags this particular ratio as an outlier compared to other replications.

DeepSeek LLM's direct scaling-law fits (31:17)

DeepSeek LLM takes a more direct approach than MuP: rather than reparameterizing the model, it fits scaling laws directly to the optimal batch size and learning rate across a grid of compute scales, then extrapolates those fits to the target model sizes (7B and 67B). It also uses a WSD-like schedule with two decay phases and replicates Chinchilla-style isoFLOP analysis to pick model and data sizes, successfully predicting the loss of its larger models from the fitted scaling law before training them.

Newer models: Llama 3, Hunyuan, and MiniMax-01 (40:24)

More recent releases publish less scaling detail, but the lecture notes a few results: Llama 3 refits the Chinchilla ratio to roughly 39-40 tokens per parameter and separately correlates training loss with downstream benchmark accuracy via sigmoid fits; Hunyuan-1 fits an even higher ratio of about 96:1; and MiniMax-01 uses isoFLOP-style analysis to argue that its linear and hybrid attention variants scale comparably to standard softmax attention, justifying the architecture choice for a long-context model.

Deriving MuP from first principles (49:29)

The lecture derives MuP from two conditions: activations at initialization should stay order-one per coordinate as width grows, and the change in activations after one gradient step should also stay order-one. Working through a simplified deep linear network, it shows that the first condition implies initializing weights with standard deviation proportional to 1/sqrt(fan_in), and the second condition (combined with an assumption that loss changes stay order-one) implies a learning rate proportional to fan_out/fan_in for SGD, or 1/fan_in for Adam. Applied back to Cerebras-GPT's table, this explains why MuP scales per-layer learning rates down by 1/width under Adam-style optimizers.

Testing how robust MuP really is (1:09:35)

The lecture closes with a large-scale ablation study on MuP-style learning-rate transfer. Learning rate transfer holds reliably across changes in activation function, batch size, and most initialization choices, and a full run up to a 10B-parameter model confirms the transferred learning rate remains near-optimal. However, transfer breaks down when learnable normalization gains are added, when using unconventional optimizers like Lion, or under strong weight decay, showing that MuP is useful but not universally guaranteed.

Before you watch

• Review the first scaling-laws lecture, especially the Chinchilla isoFLOP method and the Kaplan critical batch size analysis, since this lecture builds directly on both.
• Basic familiarity with random matrix concentration (the operator norm of a Gaussian matrix) helps the MuP derivation go smoothly, though it is not a strict prerequisite.
• Recall standard transformer initialization (1/sqrt(fan_in)-style Kaiming initialization) to appreciate what MuP changes and what it leaves the same.

Check your understanding

1. Why can't a single cosine-schedule training run be used to study how loss changes with data budget, and how does WSD fix this?
2. What are the two "spectral conditions" that MuP is designed to satisfy, and what do they say about activations?
3. How does MuP's learning rate rule differ between SGD and Adam, and why?
4. Why does DeepSeek LLM avoid using MuP, and what does it do instead to pick hyperparameters?
5. Name two conditions under which MuP's learning-rate transfer breaks down according to the empirical ablation study.