Lecture 19: Model Interpretability and Editing, with Been Kim

What this lecture covers

Been Kim, a research scientist at Google Brain, gives a guest lecture on interpretability: how far we actually are from understanding what deep models know, and what her lab has learned trying to close that gap. The lecture sits near the end of the course, after students have built and trained their own language models, and now turns to the harder question of explaining what those models represent internally.

The talk moves through three strands of research: theoretical and empirical evidence that popular attribution methods like saliency maps, SHAP and Integrated Gradients do not reliably reveal model behavior; a case study showing that locating a fact inside a language model (as in the ROME method) does not predict how well that fact can be edited; and work that treats reinforcement learning agents as an unfamiliar species to be studied through observation and controlled intervention. After watching, you should be able to explain why attribution scores are not trustworthy evidence of feature importance, describe the mismatch between localization and editing found in ROME-style methods, and describe how concept bottlenecks let researchers test causal claims about multi-agent coordination.

Key ideas

• Representation gap: there is a persistent gap between what a model actually computes and what humans believe it computes, and closing that gap is a prerequisite for understanding surprising model behavior such as AlphaGo's "move 37".
• Saliency maps: these attribution methods assign an importance score to each input feature, but Kim's group found that trained and untrained (random) networks can produce nearly identical saliency maps.
• Hypothesis-testing proof: reframing attribution as a hypothesis test (is this feature actually important?) lets you apply statistical tools; the proof shows that popular attribution methods do no better than random guessing at confirming or denying such a hypothesis.
• ROME and causal tracing: ROME locates where a fact is stored in a language model by corrupting and restoring activations layer by layer, then edits that location to change the fact.
• Localization is not correlated with editing success: across layers, the choice of layer explains most of the variance in whether an edit succeeds, while the causal-tracing localization score explains almost none of it.
• Concept bottleneck models: inserting a layer of human-labeled concepts (such as an agent's position or orientation) into a network lets researchers intervene on one concept at a time and measure its causal effect on behavior.
• Observational study of agents: clustering the states and actions of reinforcement-learning agents, without using labels, can recover human-interpretable behaviors such as "running and chasing" or "fort building".

Walkthrough

Why we should doubt what we think machines know (9:12)

Kim frames the talk around wanting to understand emergent, surprising behavior such as AlphaGo's move 37, and argues this requires first knowing whether existing interpretability tools accurately describe what a model knows. She introduces saliency maps as the standard tool: each input pixel gets a score meant to reflect how sensitive the model's output is to that pixel.

Saliency maps and attribution methods do not reveal model behavior (12:14)

Kim describes a series of experiments where trained and untrained networks produced indistinguishable saliency maps, and where inserted bugs (labeling errors, spurious correlations, out-of-distribution inputs) were not detectable from the explanations. She then walks through a theoretical proof, framed as hypothesis testing, showing that methods like SHAP and Integrated Gradients cannot do better than random guessing at validating a claim like "this feature is important". The result holds both in theory and in follow-up experiments on tasks such as loan recourse.

Editing language models: the ROME case study (22:22)

Kim describes an attempt to locate and edit ethical knowledge in a language model, starting from the ROME method, which uses causal tracing to find the layer where a fact (such as "the Space Needle is in Seattle") is stored, then edits that layer. Testing this on GPT-J, her group found that the layer ROME identifies as the source of a fact is often not the layer where editing works best, and across many facts the correlation between localization strength and edit success was close to zero, even slightly negative. A regression analysis showed the choice of layer explains about 94% of edit-success variance, versus about 1.6% for the tracing signal. Trying several alternative definitions of "editing" did not change this conclusion, including for MEMIT and fine-tuning approaches. The finding does not invalidate causal tracing as a way to see where a fact appears in a forward pass, but it does mean localization should not be used to justify or motivate editing methods.

Discovering emergent multi-agent behavior by observation (35:36)

Kim reframes trained agents as an unfamiliar species and describes an observational study of multi-agent reinforcement learning systems, including OpenAI's hide-and-seek agents. Without using the human-assigned behavior labels, a generative model clusters agents' states and actions into a shared latent embedding space. The resulting clusters recover recognizable behaviors, such as clearly separated "running and chasing" embeddings, and reveal that hiders develop a more distinct "fort building" representation than seekers do, consistent with hiders being the ones who build forts.

Intervening on agent concepts to test coordination (43:41)

To go beyond observation, Kim's group built reinforcement-learning agents with a concept bottleneck: a layer where each neuron corresponds to a human-labeled concept, such as an agent's position, orientation, or whether it is holding an item. Zeroing out a concept at inference time shows its causal effect on the agents' reward. In a cooking-game domain, intervening on orientation degraded coordinated performance far more than intervening on position, because orientation signals where an agent is about to move. In a "cleanup" domain with four agents, building a simple regression-based graph over intervention outcomes revealed that two agents were only coincidentally correlated, due to getting physically stuck together, rather than genuinely coordinating.

Chasing superhuman chess concepts with AlphaZero (55:51)

Kim closes with ongoing work using AlphaZero, the self-trained chess engine that exceeds human and Stockfish performance. Earlier work located known human chess concepts, such as material imbalance, inside AlphaZero's network and compared its opening moves to human openings, finding it masters a much wider range of opening styles. The current project tries to discover genuinely new chess concepts by deliberately excluding known human strategies from the analysis, then testing whether world champion Magnus Carlsen can learn and apply them through a set of puzzles generated from AlphaZero's own board positions.

Before you watch

• Familiarity with saliency maps, SHAP or Integrated Gradients is helpful, though Kim explains them from scratch.
• Some exposure to reinforcement learning basics (policies, states, actions, reward) makes the multi-agent sections easier to follow.
• Knowing roughly what a language model layer and its activations are helps with the ROME/editing discussion.

Check your understanding

1. Why does a saliency map look similar for a trained network and an untrained (random) network, and what does this imply about using such maps as evidence of feature importance?
2. In Been Kim's hypothesis-testing framing, what does it mean for an attribution method to be "no better than random guessing"?
3. What did the ROME case study find about the relationship between where a fact is localized in a model and how successfully it can be edited there?
4. How does a concept bottleneck model let researchers test causal claims about which features drive coordination between agents, compared to just observing correlations?
5. What is the goal of the ongoing AlphaZero chess project, and how is its success meant to be evaluated?