Open Source and Science with Foundation Models

What this lecture covers

Percy Liang argues that as foundation models have gotten more capable, access to them has shrunk from full weights, code and data down to just an API — and that this loss of access changes what kind of research is even possible. The talk walks through three levels of access (API, open weights, open source) and shows concrete research projects enabled at each level.

Situated as a guest lecture in the LLM Agents course, the talk connects agent-building directly to the access question: it shows what kinds of agents you can build with only API access, then argues why deeper access is needed to truly understand and improve these systems. After watching, you should be able to describe the difference between API, open-weight and open-source access, explain the memory-retrieve-reflect-plan architecture used across several agent projects, and describe why open weights enabled research such as neuron interpretability and model-provenance testing.

Key ideas

• Access shapes research: Liang argues that historically, what data or compute researchers can access (internet text, crowdsourced datasets, GPUs) has determined which research areas become possible, and today's shift to API-only access is similarly constraining.
• Three levels of access: API access (behavior only, like a cognitive scientist), open weights (internals inspectable, like a neuroscientist), and open source (full control over architecture, training and data, like a systems builder).
• Agent architecture: agents perceive observations into a memory stream, retrieve relevant memories (by recency, importance, relevance), and then act, reflect, or plan — this same pattern is reused across the agent examples in the talk.
• Problem-solving agents: MLAgentBench (writing and iterating on machine learning code) and a cybersecurity capture-the-flag agent both use the same reflect-plan-act loop; both showed large gaps between current model performance and human experts.
• Simulation agents: Smallville-style generative agents model behavior for a population of LLM-driven characters using memory, retrieval and reflection, and a follow-up study seeded agents with two-hour interviews of 1,000 real people to see how closely agent responses matched the real individuals' survey answers.
• What open weights enable: neuron-level interpretability (e.g. diagnosing why a model claims 9.11 is bigger than 9.8), model pruning/distillation to smaller sizes, and adversarial prompt attacks that sometimes transfer from open to closed models.
• Model independence testing: a statistical method Liang's group built to test, using only model weights, whether two published models were trained independently or one is a fine-tune of the other — verified on real cases like the Miqu leak and Llama 3.2's smaller models.
• Open source requires more than weights: the Open Source Initiative's definition requires available code (including training and data-processing code) and data information, not just downloadable parameters; compute remains the main practical bottleneck.

Walkthrough

Why access matters and the three-tier framework (0:00)

Liang opens by noting that as model capability has risen, openness has fallen almost symmetrically, leaving mostly API access. He argues from history — internet-scale text enabling NLP, crowdsourced datasets like ImageNet and SQuAD, and GPUs enabling deep learning — that access determines what research becomes possible, then introduces the three levels he'll cover: API access, open weights, and open source.

Building agents with only API access (4:04)

Treating an API model as a general-purpose function, Liang describes a common agent architecture: observations go into a memory stream, relevant memories are retrieved, and the agent acts, reflects, or plans. He presents MLAgentBench, where an agent iteratively writes and revises machine learning code based on execution logs, and a cybersecurity capture-the-flag agent that reads server code, finds vulnerabilities, and executes exploits via a bash shell. In both, Claude 3(.5) performed best among tested models, but a large gap remained versus expert humans, especially on harder tasks.

Simulation agents and modeling real people (14:09)

Liang describes the "Smallville" generative-agents project, where 25 LLM-driven characters use memory, retrieval (by recency, importance, relevance) and reflection to behave believably and even spread information like a party invitation between each other. A newer project seeded 1,000 real people's two-hour interview transcripts as agent memories, then compared agents' answers to survey and experiment questions against the actual humans' answers, finding agents captured a large share of individual variation, though less than the humans' own test-retest consistency.

What open-weight access adds (25:25)

Moving down a level, Liang defines open-weight models (like Llama) as widely but not unconditionally available, and highlights reproducibility as a key benefit once an API model is deprecated. He shows interpretability work that traced a wrong answer ("9.11 is bigger than 9.8") to a neuron associated with dates, a pruning example producing smaller models from a larger one with little performance loss, and an adversarial attack optimized against Llama's gradients that also transferred to jailbreak GPT-4.

Testing whether models are independently trained (31:29)

Liang presents his group's method for testing, from weights alone, whether two published models were trained independently. The approach compares an observed similarity statistic against a distribution generated by permuting the hidden units of a model (a stand-in for retraining from a different random initialization), producing a formal statistical test. Applied to real models, it confirmed that the Miqu leak was a Mistral fine-tune of Llama 2, that Striped Hyena was derived from Mistral, and that Llama 3.2's 3B model reused specific layers from the 8B model.

Open source: definitions and the compute bottleneck (40:33)

Liang traces open source's roots to open-source software and academic open science, then presents the Open Source Initiative's "four freedoms" (use, study, modify, share) applied to AI, which require not just weights but training/data-processing code and data information. He describes three of his own projects that needed full training access: an automated data-mixture optimization method, a second-order optimizer giving a 2x speedup over Adam, and the Backpack language model, an interpretable alternative architecture supporting precise editing. He closes by proposing three ways to address the compute bottleneck limiting open-source research: using scaling laws to work meaningfully at small scale, pooling idle decentralized compute despite slower interconnects, and advocating for public compute funding comparable to projects like the Human Genome Project.

Before you watch

• General familiarity with what an LLM agent is (perception, memory, action) is useful background from earlier lectures in this course.
• No specific prior lecture is required, though awareness of basic transformer/foundation-model terminology helps.

Check your understanding

1. Why does Percy Liang claim that "access shapes research," and what historical examples does he give?
2. Describe the memory stream, retrieval and reflection components shared across the MLAgentBench, cybersecurity, and Smallville agent examples.
3. How does the model independence test work, and what real-world cases did it help verify?
4. What does the Open Source Initiative's definition require beyond publishing model weights, and why does data information matter even when raw data cannot be shared?
5. What three strategies does Liang suggest for making progress on open-source foundation model research despite limited compute?