Measuring Agent Capabilities and Anthropic's RSP

What this lecture covers

Ben Mann, an Anthropic co-founder, explains why measuring what an AI model can actually do is difficult, and how Anthropic's Responsible Scaling Policy (RSP) tries to tie deployment decisions to measured capability levels rather than guesswork. He connects this framework to concrete agent demos, including code-editing and early "computer use" agents that control a mouse and keyboard directly.

This talk sits alongside other guest lectures on AI safety in the course, focusing specifically on industry practice at a frontier lab rather than academic research. After watching, you should be able to explain what "elicitation" means and why it complicates capability claims, describe Anthropic's AI Safety Level (ASL) framework and how it parallels biosafety levels, and identify the layered defenses proposed for securing computer-use agents against prompt injection.

Key ideas

• Elicitation: a model's raw capability and the capability you can actually get out of it via prompting can differ hugely (e.g. chain-of-thought prompting dramatically improves reasoning); this makes it hard to know a model's true ceiling at release time.
• Compute-capability scaling: training compute has grown roughly six orders of magnitude over about 14 years, and capabilities have tracked this growth, motivating concern about where near-future models will land.
• Responsible Scaling Policy (RSP): modeled on biosafety levels, it defines AI Safety Levels (ASL-1 through ASL-5) with required containment and deployment safeguards that must be in place before a model of that capability level is trained or released.
• ASL levels: ASL-1 covers models with no meaningful risk; ASL-2 (where Mann places current frontier models like Claude 3.5 Sonnet) shows early dangerous-capability signals that don't yet exceed existing resources like search engines; ASL-3 involves significantly increased misuse risk and early signs of possible autonomous replication; ASL-4 and ASL-5 involve progressively more advanced, undefined but potentially catastrophic capabilities.
• Benchmark saturation: newly defined benchmarks tend to be solved very quickly once introduced, making it hard to forecast when models will cross meaningful capability thresholds using benchmarks alone.
• Task time-horizon evals: an evaluation approach measuring how many minutes of human-equivalent task time a model can complete; scores were rising quickly across recent Claude models at the time of the talk.
• Agentic environment matters: a model with weaker raw reasoning but access to tools (running code, seeing errors, iterating) can outperform a stronger model without that feedback loop, because it can correct its own mistakes.
• Computer-use agents and prompt injection risk: agents that directly control a mouse, keyboard and screenshots introduce new attack surfaces, such as hidden text on a webpage instructing the model to exfiltrate credentials.
• Defense in depth: proposed protections include sandboxing (running agents in credential-free virtual machines), an instruction hierarchy that treats developer/system instructions as privileged over content the model merely observes, and scalable oversight using separate models to monitor for jailbreak attempts.

Walkthrough

Why measurement matters (1:01)

Mann introduces elicitation: the gap between what a model can do in principle and what you can get it to do through prompting. He notes this gap has always existed and argues that safe deployment requires confidence that a model's assessed capability level will hold even against future elicitation breakthroughs, which becomes more consequential once models can take real-world actions as agents.

Compute scaling and the path to more capable models (4:06)

Using a chart of training compute over roughly 14 years, Mann shows a steady multi-order-of-magnitude increase and a corresponding rise in capability, loosely comparing GPT-2 through GPT-4 to school-age analogies. He notes extrapolations (with heavy caveats) suggesting AI could approach expert-researcher-level capability by around 2028, framing this as the backdrop for why Anthropic invests heavily in safety work.

Coding agent demo and the RSP framework (11:44)

After a short demo of Claude 3.5 Sonnet autonomously finding and fixing a bug with tests, Mann introduces the Responsible Scaling Policy, modeled on biosafety levels used for handling dangerous pathogens. He walks through the ASL-1 to ASL-5 scale: ASL-2 models show early dangerous-capability signals (evaluated partly with hired biosecurity experts) without exceeding existing resources; ASL-3 involves meaningfully increased misuse risk and early autonomous-replication signs (illustrated with a short story about a fictional agent escaping a test sandbox); ASL-4 and ASL-5 describe progressively more advanced, still-undefined capability levels, with ASL-5 associated with potential catastrophic risk if misaligned. Mann states Claude 3.5 Sonnet was assessed near the edge of ASL-2 at the time of the talk.

Forecasting capability and evaluating agents (22:05)

Mann explains a key difference from biology: AI safety levels must be forecast for capabilities that don't yet exist, whereas biosafety levels address known pathogens. He discusses why benchmarks tend to saturate quickly once defined, making forecasting hard, and introduces an evaluation measuring how much human-equivalent task time a model can complete autonomously, which was rising across recent model generations. He also shows that a model with a good execution environment (running and testing its own code) can outperform a nominally stronger model without that feedback loop, illustrating that evaluation results depend heavily on the surrounding agent scaffold.

Computer-use agents and their safety risks (29:12)

Mann introduces Anthropic's computer-use reference implementation, where a model controls mouse and keyboard actions via screenshots, demonstrated with a coding task and a form-filling task pulling data from different applications. He explains that this capability was released only as a reference implementation, not a public product, because safety is not yet well understood. He walks through a concrete risk: a malicious webpage could contain near-invisible text instructing the model to exfiltrate credentials, and warns that even a "babysitter" classifier model could itself be manipulated by similar injected instructions.

Defense in depth and next steps (35:16)

Mann describes a layered "Swiss cheese" defense model: running agents in sandboxed, credential-free virtual machines; developing an instruction hierarchy so system/developer instructions are prioritized over content merely observed by the model; and building scalable oversight so monitoring models can reliably catch jailbreak attempts. He closes by naming benchmarks used to track capability (SWE-bench Verified for coding, OSWorld for general desktop tasks) and mentions Anthropic's Long-Term Benefit Trust, which is intended to shift governance control toward safety-focused, non-financially-interested board members as AI capability grows.

Before you watch

• Familiarity with the agent architecture concepts from earlier lectures in this course (tool use, agent loops) helps contextualize the computer-use demos.
• No deep technical background is required; the talk is largely conceptual and policy-oriented.

Check your understanding

1. What is elicitation, and why does it make it hard to know a model's true capability level at release time?
2. How is Anthropic's Responsible Scaling Policy modeled on biosafety levels, and where does the analogy break down?
3. What distinguishes ASL-2 from ASL-3 capabilities in Mann's description?
4. Why did a Claude model with weaker reasoning outperform a stronger reasoning model on some coding evaluations?
5. What specific attack does Mann describe against computer-use agents, and what three layers of defense does he propose against it?