Lecture 6: Capabilities

What this lecture covers

Following the previous lecture's treatment of Unix privilege separation, this lecture introduces capabilities as a fundamentally different way for a program to hold and exercise privileges. It starts from Norm Hardy's confused deputy problem, in which a Fortran compiler with extra file-writing privileges could be tricked by a malicious user into overwriting sensitive files, because the compiler implicitly used all the privileges available to it rather than explicitly choosing which ones to apply. That failure is used to define ambient authority, the idea that an operation's success depends on implicit context (like a process's uid) rather than something the caller explicitly presents.

The lecture then builds up capabilities as the alternative: an unforgeable token that, if you hold it, lets you act on an object, with no other implicit check involved. It uses Unix file descriptors as a concrete, already-familiar example of a capability, then walks through Capsicum, a FreeBSD extension that turns an entire process into a capability-only sandbox by eliminating access to global namespaces such as the filesystem root, arbitrary network addresses, and process IDs. After watching, you should be able to explain why ambient authority is dangerous, how file descriptors behave like capabilities, and how Capsicum's cap_enter, openat, process descriptors, and libcapsicum combine to sandbox real programs like tcpdump and gzip.

Key ideas

• Confused deputy: a privileged program (Norm Hardy's Fortran compiler) that implicitly combines multiple sources of privilege and can be tricked into misusing the more powerful one on the caller's behalf.
• Ambient authority: a security decision that depends on implicit context, such as a process's current uid, rather than an explicit token the caller presents; the analogous network example is firewall-based trust by IP address or network location.
• Discretionary vs. mandatory access control: in DAC (the Unix model), the object's owner sets its policy; in MAC, a separate authority (historically military classification systems) sets policy regardless of the owner's wishes.
• Capability: an unforgeable reference to an object that itself grants access, optionally restricted by a small set of permission bits (read-only, write-only, and so on); if you don't hold it, you simply cannot act on the object.
• File descriptors as capabilities: a file descriptor is just an integer that the kernel resolves through a per-process table; you can only reference table entries you legitimately obtained (by opening a file or receiving a descriptor), which is why descriptors can't be forged.
• Capsicum's cap_enter: a system call that puts a process into capability mode, after which it can no longer use global-namespace operations like open, arbitrary connect, or kill by PID, and must use capability-relative calls like openat.
• Process descriptors (pdfork): Capsicum's replacement for PIDs, representing a child process as a file descriptor so it can be passed around and managed without a global process namespace.
• libcapsicum and lch_start: a userspace library that names passed file descriptors and starts a sandboxed program from a clean, explicitly constructed set of arguments and capabilities, avoiding leftover sensitive state in memory or open descriptors.

Walkthrough

The confused deputy problem (1:02)

The lecture reconstructs Norm Hardy's original problem: a Fortran compiler held a "home files license" giving it write access to an entire system directory, so it could log compilation statistics. Because the compiler simply called open/write using whatever privileges it had, a user could pass an output path pointing at the billing file, and the compiler would overwrite it using its extra license rather than the user's own (insufficient) privileges. The class works through Unix-flavored fixes, splitting the compiler into an unprivileged front end and a privileged logger, and considers why setuid-based versions of that split are awkward, concluding that the root issue is the compiler's inability to say explicitly which privilege it intends to use for a given operation.

Ambient authority and access control models (10:08)

The lecture generalizes the compiler bug into the concept of ambient authority: an operation succeeds because of context the process silently carries (its uid), not because the caller explicitly proved it should be allowed. Firewalls are given as a non-OS example, where a request is trusted because of which side of the network it originates from. This is contrasted with discretionary access control, where a file's owner sets its own policy, versus mandatory access control, where a separate authority (historically classification levels in military systems) overrides the owner's wishes. The lecture also notes a second failure mode: replicating access-control logic inside an application (instead of relying on the kernel) is fragile because it's easy to omit checks or drift out of sync with kernel changes.

Capabilities and file descriptors (18:13)

Capabilities are introduced as objects that, if held, directly grant access, optionally restricted to specific operations like read or write, with no further ambient check. The lecture shows that Unix file descriptors already work this way: a descriptor is an integer resolved only through the calling process's file table, so it can't be forged out of thin air, only obtained by legitimately opening a file or receiving one from another process. Applying this to the Fortran compiler, the fix becomes passing already-open file descriptors instead of path names, so a front-end process converts filenames the user is allowed to access into descriptors before handing off to a privileged back end. The lecture notes a limitation of pure capability systems: they make it hard to name an object without also granting rights to it, which is sometimes exactly what you want to do.

Motivating sandboxing and comparing to Unix and VMs (31:20)

The discussion shifts to the Capsicum paper's own motivation: limiting the privileges of components handling untrusted, attacker-controlled input, such as network packet parsers like tcpdump or the gzip decompressor, since parsing in C is prone to memory-corruption bugs. The lecture compares this to sandboxing with plain Unix mechanisms (uid switching, chroot, file permissions), noting the drawbacks: you need root to change uids, world-readable or world-writable files can leak through regardless of uid, and chroot complicates sharing files across processes. Virtual machines are considered as a stronger but heavier alternative, good for full isolation but poor for fine-grained sharing of specific files or network connections.

How Capsicum works (41:27)

Capsicum eliminates access to "global namespaces": the filesystem path hierarchy, arbitrary network addresses, and PIDs. Once a process calls cap_enter, it can no longer make calls like open or connect with an absolute name; instead it must use capability-relative calls such as openat on an existing directory descriptor. Networking similarly becomes capability-based: all connections must exist as descriptors before entering capability mode. Process management uses pdfork and process descriptors instead of PIDs, letting a child process be referenced and handed to another process as a file descriptor. The lecture also covers why .. path components are disallowed in capability mode (they would require checking capabilities on a parent directory, which is hard to do safely against symlink races), and one remaining place where uid still matters: opening a file inside a directory you hold a capability to still goes through ordinary Unix permission checks, preserving compatibility.

The libcapsicum library and starting a sandbox safely (58:34)

Beyond the kernel mechanism, libcapsicum provides named "FD lists" so passed file descriptors don't have to be tracked by fragile integer conventions, and lch_start, a safer way to launch a sandboxed program. lch_start matters because merely calling cap_enter doesn't clear out whatever a process already holds: the lecture describes how the Capsicum authors initially sandboxed tcpdump by calling cap_enter mid-program, only to find it still held an open descriptor to the user's terminal, letting a compromised process read keystrokes. lch_start instead execs a fresh process with an explicitly chosen set of arguments and capabilities, avoiding leftover sensitive memory or descriptors, and restricts setuid binaries from running inside capability mode.

Applying Capsicum: tcpdump, gzip, and OKWS (1:07:41)

The lecture walks through why sandboxing gzip required more restructuring than tcpdump: gzip's driver logic (handling multiple files, creating outputs) had to be separated from its core compression/decompression logic, which moves into a nearly privilege-free helper process reached over what amounts to an RPC call. Returning to OKWS from the prior lecture, the class discusses how Capsicum could improve it: replacing chroot with precise per-file capabilities, letting okld drop privileges after binding port 80 instead of staying root, giving process descriptors to per-service monitors, and using capability masks to make the log file genuinely append-only. The awkward case is that a launcher process would need to know in advance which network connections each service needs; the lecture describes FreeBSD's later addition of a helper daemon, Casper, that mediates requests like DNS lookups for sandboxed processes without granting general namespace access.

Trusting and attacking Capsicum (1:16:46)

The lecture closes by examining what could go wrong: a developer could hand a sandboxed process overly broad capabilities or overly permissive masks, code that runs before cap_enter or lch_start is unprotected, and Capsicum only helps for the portions of an application that can actually run inside capability mode, since most nontrivial Unix programs need some global-namespace operations somewhere. The deepest risk is coverage: every system call has to be audited to ensure it can't reach an object without a corresponding capability, and the kernel itself must be free of memory-corruption bugs, since a kernel exploit defeats the whole model. Linux's seccomp, which restricts by system call rather than by object, is mentioned as a coarser alternative available on Linux.

Before you watch

• Watch the earlier lecture on Unix privilege separation and OKWS, since this lecture repeatedly contrasts capability-based design with that Unix model.
• Be familiar with basic Unix file descriptors, fork, and exec.
• Recall how buffer overflows and memory-corruption bugs arise in C, since they motivate why parsers like tcpdump and gzip need sandboxing in the first place.

Check your understanding

1. In Norm Hardy's confused deputy problem, why did giving the Fortran compiler its own privileges create a vulnerability, and how does passing file descriptors instead of path names fix it?
2. What makes a Unix file descriptor behave like a capability, and why can't a process forge one?
3. Explain what "global namespace" means in Capsicum's design, and give two examples of global namespaces it eliminates access to after cap_enter.
4. Why does lch_start execute a fresh process instead of simply calling cap_enter partway through an existing one?
5. Describe one concrete way Capsicum could have improved OKWS's design compared to relying on chroot and setuid alone.