Lecture 2: Control Hijacking Attacks

What this lecture covers

This lecture continues directly from the previous one, going deeper into buffer overflow attacks and the defenses built against them. It starts by explaining why so much critical infrastructure software is written in C, why C's raw, unchecked pointers make control-hijacking attacks possible, and what an attacker can do once they control the instruction pointer. It then works through several mitigation strategies in increasing detail: avoiding bugs, static analysis, program fuzzing, using memory-safe languages, stack canaries, malloc/free corruption attacks, and bounds-checking approaches including electric fences, fat pointers, and the baggy bounds scheme (with its supporting buddy-allocator data structure).

After watching, you should be able to explain how a stack canary catches a buffer overflow before it can be exploited, why canaries fail against attacks that don't touch the return address, and how baggy bounds encodes allocation sizes as powers of two to make bounds checks fast.

Key ideas

• Instruction pointer hijacking: a buffer overflow attack needs two things to succeed: control over where execution jumps, and a way to get useful (often attacker-supplied) code at that location.
• Stack canary: a value placed between a buffer and the saved return address; if an overflow overwrites the canary, the function's return sequence detects the change and aborts before jumping to a corrupted address.
• Canary weaknesses: canaries only protect the return address; overwriting an unrelated pointer or variable elsewhere on the stack (for example, one declared before the buffer) bypasses the check entirely, and a deterministic canary value can potentially be guessed or leaked.
• Malloc metadata corruption: heap allocator bookkeeping (size and free-list pointers stored alongside allocated blocks) can be corrupted by a buffer overflow, letting an attacker control what free writes to and where.
• Bounds checking: since it's often ambiguous in C what a valid pointer even is, practical systems enforce a weaker rule: a pointer derived from a base pointer may only reference memory that belongs to that same allocation.
• Electric fences: guard pages placed next to heap allocations trigger a hardware fault immediately on out-of-bounds access, which is useful for debugging but wastes an entire page per allocation.
• Fat pointers: widen a pointer to carry its allocation's base, end, and current address so every access can be checked, at the cost of much larger pointers, broken struct layouts, and lost atomicity.
• Baggy bounds: rounds every allocation up to a power of two (via a buddy allocator) and stores just the log2 of the size in a compact per-slot table, making bounds lookups fast and cheap.

Walkthrough

Why C enables buffer overflow attacks (1:06)

The lecture explains that system software such as databases, compilers, and network servers is typically written in C because it behaves like high-level assembly and offers speed. The tradeoff is that C exposes raw memory addresses with no automatic bounds checking, partly because checking is expensive and partly because it can be genuinely hard to define what "in bounds" even means for a given pointer. Exploiting this also requires knowledge of the underlying architecture, such as which direction the stack grows and how function calling conventions lay out the stack.

From overflow to hijacked control flow (4:07)

Using the same kind of vulnerable gets-based function seen in the previous lecture, the lecture reviews the stack layout (buffer, saved base pointer, return address, caller's frame) and shows how an overflow that grows toward the return address lets an attacker choose where the function jumps on return. It stresses that the operating system does not inspect every memory access during normal execution; only the hardware and, at specific moments like system calls, the OS get involved, so nothing stops a process from overwriting its own address space. Once an attacker controls execution, they inherit the privileges of the compromised process, which can mean reading files, sending spam, or bypassing firewall trust boundaries from inside a trusted network.

Three defense strategies: avoiding bugs, static analysis, and fuzzing (9:16)

The lecture surveys non-runtime approaches. Avoiding unsafe functions like gets (which modern compilers flag) is a simple first step, but many programs manipulate buffers with custom parsing code that never calls a flagged function. Static analysis examines source code before execution to catch problems such as uninitialized variables, and can propagate constraints from branch conditions (for example, knowing a variable must be greater than eight after a specific if check) into the analysis of called functions. Program fuzzing feeds large volumes of automatically generated, often random, inputs to code to maximize branch coverage, and can use the same constraint information from static analysis to generate inputs that deliberately exercise each branch.

Memory-safe languages and the performance argument (16:25)

Switching to a memory-safe language such as Python, Java, or C# avoids these bugs by construction, but the lecture explains why this isn't always practical: huge amounts of legacy C code exist and can't simply be rewritten, and some tasks (device drivers, low-level hardware access) genuinely need C's direct access. On performance, the lecture describes how just-in-time (JIT) compilation narrowed the speed gap between interpreted high-level languages and native code by generating machine instructions from bytecode at runtime instead of interpreting it in a loop, and notes that programs bound by I/O (network, disk, user input) often don't need raw compute speed at all.

Stack canaries in detail (27:33)

The lecture introduces stack canaries as a value placed directly before the return address so an overflow reaching the return address must first overwrite the canary. Compiler-inserted code checks the canary just before a function returns and aborts if it has changed. Several canary designs are discussed: a fixed value built from bytes like null, carriage return, line feed, and negative one that many unsafe string functions stop on, and a randomized value whose strength depends entirely on how many bits of real entropy back it. Class discussion covers why naive sources of "randomness," such as timestamps, can carry far fewer real entropy bits than expected, and why rolling your own randomness or crypto is generally a bad idea.

Where canaries fail (37:49)

Canaries only protect the return address. An overflow that corrupts a function pointer or other stack variable declared before the buffer, without ever reaching the return address, bypasses the canary completely, since the attack never triggers the check. The lecture also walks through a malloc/free corruption attack: an overflow in one heap-allocated buffer can corrupt the size and linked-list pointer metadata of an adjacent allocation, and because free's internal pointer-merging logic trusts that metadata, an attacker who controls the corrupted size field can effectively make free write an attacker-chosen value to an attacker-chosen address.

Bounds checking: electric fences and fat pointers (57:09)

The lecture defines bounds checking's practical goal in C: a pointer derived from a base pointer should only be used to access memory belonging to that same allocation, which is weaker than fully correct pointer semantics but catches memory corruption. Electric fences allocate a hardware-protected guard page beside every heap object so any out-of-bounds touch faults immediately, which is excellent for debugging but far too space-inefficient for production, since even a two-byte allocation needs a full page (typically 4 KB) of protection. Fat pointers instead widen every pointer to carry its base address, end address, and current address, letting the compiler insert bounds checks on every dereference; the cost is much larger pointers, incompatibility with unmodified libraries and structs, and loss of atomic pointer updates.

Baggy bounds and buddy allocation (1:06:17)

The lecture builds up to the baggy bounds system used in the assigned paper. It first explains buddy allocation: memory starts as one large block and is recursively split into halves (powers of two) until a block just over half-filled by the request is found, and adjacent free blocks of the same size are merged back together on deallocation. Baggy bounds relies on this by rounding every allocation up to a power of two, storing each allocation's size as log2(size) in a compact table with one entry per fixed-size slot (16 bytes in the paper), and updating multiple table slots for allocations spanning more than one slot. Given a pointer, the size is recovered by looking up the table entry and left-shifting 1, the base is recovered by masking off the low bits using size - 1, and a derived pointer is checked to be in bounds with a simple comparison. As a final defense, baggy bounds can set the high-order bit of an out-of-bounds pointer so the virtual memory system faults if that pointer is ever dereferenced.

Before you watch

• Watch the previous lecture in this course (Introduction, Threat Models) first, since it introduces the basic stack layout and buffer overflow mechanics this lecture builds on.
• Familiarity with C pointers, malloc/free, and basic stack frame layout (return address, saved base pointer) will make the walkthrough much easier to follow.
• Some exposure to binary and hexadecimal representation helps with the baggy bounds arithmetic near the end.

Check your understanding

1. Why does controlling the instruction pointer alone not guarantee a successful attack, and what else does an attacker typically need?
2. Describe a buffer overflow scenario where a stack canary would fail to prevent exploitation.
3. How does corrupting an adjacent heap allocation's size metadata let an attacker influence what free writes to memory?
4. What tradeoff does baggy bounds make by rounding every allocation up to a power of two, and why does this make bounds checks fast?
5. Why are electric fences useful for debugging but impractical for production systems?