Lecture 8: State of GPT

What this lecture covers

This talk, given at Microsoft Build, answers two questions: how are models like ChatGPT actually trained, and how should you use them once they exist? It's a conference presentation rather than a coded walkthrough, and it pairs well with the "Let's build GPT" lecture earlier in this course by zooming out from the Transformer architecture to the full training pipeline and to practical prompting technique.

The talk is split into two halves. The first covers the four-stage recipe for producing a GPT assistant: pretraining, supervised fine-tuning, reward modeling, and reinforcement learning from human feedback. The second half covers how to think about prompting these models effectively, including techniques like chain-of-thought prompting, retrieval augmentation, and tool use. After watching, you should be able to describe what happens at each training stage and apply several concrete prompting strategies to your own use of LLMs.

Key ideas

• Pretraining dominates the compute budget: it uses internet-scale data, thousands of GPUs, and months of training, while the later fine-tuning stages use much smaller datasets and far less compute.
• Tokenization: raw text is losslessly translated into integer tokens (for example via byte pair encoding) before it can feed into a Transformer.
• Base models complete documents, they don't answer questions: a freshly pretrained model imitates whatever text pattern it's given rather than acting as a helpful assistant.
• Supervised fine-tuning (SFT): a small, high-quality dataset of prompt-and-ideal-response pairs, written by human contractors, is used to continue language-model training and turn a base model into a basic assistant.
• Reward modeling and RLHF: humans rank multiple model completions for the same prompt; a reward model learns to predict these rankings, and reinforcement learning then tunes the assistant to produce completions the reward model scores highly.
• Transformers are token simulators: they spend roughly equal computation per token, don't know what they don't know, and can't revise earlier tokens, so techniques like "think step by step" work by spreading reasoning across more tokens.
• Retrieval augmentation and tool use: loading relevant documents or calculator/code-interpreter access into the context window compensates for the model's imperfect recall and weak arithmetic.

Walkthrough

The four-stage training recipe (0:00)

Karpathy introduces pretraining, SFT, reward modeling, and RLHF as four stages that each have their own dataset, training objective, and resulting model, with pretraining set apart as the most compute-intensive by a wide margin.

Pretraining: data, tokenization, and scale (2:03)

Using LLaMA's published data mixture as an example (CommonCrawl, GitHub, Wikipedia, books, and more), he walks through tokenization, typical hyperparameters like vocabulary size and context length, and why LLaMA outperforms the larger GPT-3 despite fewer parameters: it was trained on far more tokens. He also shows what pretraining looks like in practice, with a small GPT trained on Shakespeare gradually producing more coherent text as loss decreases.

From base model to assistant: SFT (9:09)

Base models can be prompted cleverly (few-shot prompting, or framing a fake Q&A document) to behave somewhat like assistants, but this is unreliable. Supervised fine-tuning instead trains directly on a small set of high-quality prompt-and-ideal-response demonstrations written by contractors, producing a model that behaves like a real, if limited, assistant.

Reward modeling and reinforcement learning (13:11)

Rather than asking contractors to write ideal answers directly, this stage asks them to rank multiple completions from the SFT model, since comparing outputs is easier than generating them. A reward model is trained on these rankings, and reinforcement learning then optimizes the assistant's policy to produce completions the reward model scores highly, which is the process behind models like ChatGPT.

Why RLHF helps, and where it falls short (17:14)

RLHF models are generally preferred by human evaluators over SFT models, partly because comparing outputs is computationally and cognitively easier than generating them well. But RLHF models lose some output diversity compared to base models, which is why Karpathy still prefers base models for tasks like generating many varied examples from a small seed set.

Thinking like the model: prompting technique (20:17)

Using the example of writing a sentence comparing two states' populations, Karpathy contrasts the rich internal reasoning a human does with a Transformer's fixed compute-per-token budget. This motivates techniques such as "let's think step by step," self-consistency (sampling multiple times and picking the best), and asking the model to check its own work, since models don't automatically know when they've made a mistake.

Tools, retrieval, and constrained output (31:27)

Because models don't reliably know their own weaknesses, prompts can explicitly tell them to use a calculator or other tools. Retrieval-augmented generation, illustrated with LlamaIndex, loads relevant document chunks into the context window so the model can reference primary sources instead of relying on imperfect memory. Constrained prompting libraries like Microsoft's Guidance can force outputs into a specific format, such as valid JSON.

Fine-tuning and closing recommendations (34:30)

Parameter-efficient fine-tuning techniques such as LoRA make it cheaper to adapt a model by training only small pieces of it. Karpathy recommends defaulting to prompt engineering with GPT-4 first, only considering fine-tuning afterward, and treating RLHF as largely research territory. He closes with a caution about current LLM limitations, including bias, hallucination, and susceptibility to prompt injection, recommending human oversight for any high-stakes use.

Before you watch

• Watching "Let's build GPT" earlier in this series first will make the architecture references (tokens, context length, Transformer layers) much more concrete.
• Basic familiarity with what a language model does and what a probability distribution over tokens means is assumed.

Check your understanding

1. Why does pretraining use the vast majority of the total compute in training a GPT assistant?
2. What is the practical difference between the datasets used for supervised fine-tuning and reward modeling?
3. Why might comparing two completions be an easier task for a human than writing one from scratch, and how does this asymmetry help RLHF?
4. Give two concrete prompting techniques the lecture describes for getting a model to reason more effectively, and explain why each works.
5. Why does the lecture recommend keeping humans in the loop rather than treating current LLMs as fully autonomous agents?