Lecture 3: Buffer Overflow Exploits and Defenses

What this lecture covers

This lecture wraps up the discussion of the baggy bounds bounds-checking system, covers two more widely deployed defenses against buffer overflows (non-executable memory and address space randomization), and then builds up the blind return-oriented programming (BROP) attack step by step, showing how an attacker with no source code or binary can still defeat stack canaries, non-executable stacks, and randomization together over a network connection.

The lecture sits directly after the previous one on control hijacking and baggy bounds, and prepares you for the assigned BROP paper. After watching, you should be able to trace through a baggy-bounds example by hand, explain why non-executable memory and ASLR are popular defenses despite not being complete, describe how return-oriented programming chains together existing code "gadgets" instead of injecting new code, and outline the stages of a blind ROP attack against a server whose only observable behavior is whether a connection stays open or closes.

Key ideas

• Baggy bounds slack: pointers can drift outside their exact allocation as long as they stay within half a slot of the rounded-up power-of-two bound; going further sets a high-order "out of bounds" bit, and dereferencing that flagged pointer causes a hard fault.
• Uninstrumented code interoperability: baggy bounds initializes all table entries to the maximum bound (2^31) by default, so pointers from unmodified legacy libraries never trigger false out-of-bounds errors, at the cost of providing no real protection for that code.
• Non-executable memory (W^X): hardware page permissions let the OS mark stack (and other data) pages as non-executable, so injected shell code on the stack can no longer be jumped to and run directly.
• Address space layout randomization (ASLR): randomizing where the stack, heap, and code load each run makes hardcoded attacker addresses unreliable, but can be defeated by leaking addresses, by heap-spraying with NOP sleds, or by any channel that reveals the randomness.
• Return-oriented programming (ROP): instead of injecting new executable code, an attacker chains together existing code fragments ("gadgets," often ending in ret) by controlling stack contents, effectively using the stack pointer as a second instruction pointer.
• Canary guessing: if a server keeps running (via fork, not exec) with the same canary and layout after a crash, an attacker can guess a canary one byte at a time, using "does the connection stay open" as a correctness signal.
• Blind ROP (BROP): without source, binary, or symbols, an attacker can still find a "stop gadget" that pauses rather than crashes, then use stop/crash signals over the network to locate stack-popping gadgets, identify which registers they load, and eventually invoke write to exfiltrate the running binary itself.
• Exec over fork: because fork preserves the parent's address layout and canary in the child, restarting the crashed worker with exec (which re-randomizes on Linux with position-independent executables) is a practical defense against these probing attacks.

Walkthrough

Finishing baggy bounds: worked examples and slack (1:01)

Using a 44-byte allocation rounded up to 64 bytes with a 16-byte slot size, the lecture walks through pointer arithmetic that stays within the "baggy" slack (allowed), crosses more than half a slot past the bound (hard error), and drifts out by a small amount before coming back in bounds (allowed, with the high-order bit set only while out of bounds). It clarifies that baggy bounds only detects trouble at the moment a pointer operation is instrumented and compared against its recorded bounds, not by tracking semantic correctness.

Interoperating with uninstrumented code and the 64-bit encoding (8:08)

The lecture explains that baggy bounds initializes all bounds-table entries to the maximum size (2^31) so pointers originating in unmodified legacy libraries never falsely trigger out-of-bounds detection, though this also means no real memory safety is provided for that uninstrumented code, and a dangerous pointer can still be silently passed across the instrumented/uninstrumented boundary. On 64-bit systems, the separate bounds table can be replaced by packing size and offset information directly into unused high bits of the pointer itself, since applications rarely use the full 64-bit address range; this keeps pointers a single machine word, unlike traditional fat pointers, which matters for atomic updates and struct layout compatibility.

Costs of bounds checking and other defenses in practice (18:22)

The lecture lists baggy bounds' costs: extra memory for the bounds table (or reduced address space on 64-bit), CPU overhead for instrumented pointer checks, false alarms that erode developer trust in security tools, and the need for compiler support. It notes that GCC and Visual Studio enable stack canaries by default and that Linux and Windows both support non-executable memory and ASLR, while baggy bounds remains rare in production because of these costs.

Non-executable memory and address space randomization (20:27)

Non-executable memory uses a hardware execute permission bit so pages can be write-or-execute but not both, blocking the classic attack of jumping directly into injected shell code on the stack; it requires no application changes but complicates legitimate dynamic code generation such as JIT compilers, which must toggle write and execute permissions carefully. ASLR randomizes the starting locations of the stack, heap, and code so hardcoded addresses found via a local GDB session don't transfer to the running server. The lecture notes ASLR can be undermined by leaking randomness or, cleverly, by heap-spraying large amounts of shell code preceded by NOP sleds so an imprecise jump still lands on executable attacker code, plus more exotic ideas like randomizing syscall numbers per run or XOR-encrypting instructions with a hardware-held key.

Introducing return-oriented programming with existing functions (31:41)

The lecture motivates ROP as a way to defeat non-executable memory: instead of injecting code, the attacker strings together code that already exists in the program. A first simple case overwrites a return address with the address of a convenient existing function (for example one that directly runs a shell). A more realistic case fakes a calling frame for an existing system call, pushing the address of a string already in memory (such as a shell path) as its argument, so that returning into system effectively calls it with attacker-chosen arguments.

Gadgets and chaining multiple calls (44:00)

The lecture introduces "gadgets," small instruction sequences ending in ret (such as pop eax; ret), found by scanning the binary. By alternating the address of system, a pop; ret gadget, and an argument string on the stack, an attacker can call system repeatedly in sequence, effectively using the stack pointer as a second instruction pointer that drives execution from one gadget to the next without ever running injected code, which is what allows this technique to bypass non-executable memory protections.

Defeating stack canaries by probing (51:12)

Assuming a server has a buffer overflow, crashes and restarts on a bad canary guess, and does not re-randomize the canary or memory layout after restart (a common consequence of using fork instead of exec), an attacker can guess a multi-byte canary one byte at a time: for each byte, try values until the connection stays open instead of crashing, then move to the next byte. The lecture also mentions that a defender could mask this signal by catching the segfault and keeping the connection artificially alive.

The blind ROP (BROP) attack, step by step (58:25)

Switching to 64-bit systems, where function arguments pass through registers rather than the stack, the lecture builds the BROP attack in stages. First, the attacker finds a "stop gadget," a return address that pauses the program (for example via sleep or an infinite loop) rather than crashing it, giving a second useful signal beyond "crash." Second, using sequences of a probe address, the stop gadget, and a guaranteed-crash address (such as address zero), the attacker identifies gadgets that pop exactly one stack entry. Third, by pushing the syscall number for pause (which ignores all arguments) after each candidate pop gadget and checking which combination causes the program to pause, the attacker identifies which gadget pops into the RAX register specifically and locates the address of the syscall instruction itself. Fourth, applying the same technique with different system calls, the attacker finds gadgets that pop into the other argument registers (RDI, RSI, RDX). Finally, with control over these registers and knowledge of syscall, the attacker invokes write using a guessed low file descriptor and a pointer into the program's own code segment, causing the server to send its own binary back over the connection. From there, the attacker can analyze the exfiltrated binary offline to locate any remaining gadgets and complete a full exploit.

Defending against BROP (1:16:49)

The lecture closes with defenses: spawning worker processes with exec rather than fork so each restart gets a freshly randomized layout (Linux does this with position-independent executables; Windows has no direct fork equivalent), and keeping crashed connections open briefly to remove the crash/no-crash signal, though this risks turning into its own denial-of-service problem from accumulated zombie processes. It also addresses a homework-style question about seeding a canary with a hash of the current time, explaining that hashing a guessable input does not add real entropy, since an attacker can narrow down the likely time range and still succeed.

Before you watch

• Watch the previous two lectures in this course first: the introduction to control hijacking attacks and the initial baggy bounds discussion are assumed background here.
• Comfort with x86/x86-64 calling conventions, the stack layout (return address, saved base pointer), and basic GDB usage from the labs will make the gadget-chaining sections much easier to follow.
• Familiarity with how fork and exec differ in process creation helps with the defense discussion near the end.

Check your understanding

1. In the baggy bounds system, what distinguishes a pointer operation that sets the high-order "out of bounds" bit from one that causes an immediate hard fault?
2. Why does initializing all bounds-table entries to 2^31 make baggy bounds compatible with uninstrumented legacy code, and what protection does that sacrifice?
3. Explain how faking a calling frame for an existing system function lets an attacker run arbitrary shell commands without injecting any new code.
4. What two outcomes (besides success) does an attacker use as signals while probing for gadgets in the BROP attack, and what does each one indicate?
5. Why does restarting a crashed server with exec rather than fork defeat the canary-guessing and BROP techniques described in this lecture?