Buffer overflow vulnerabilities are the “ancestor” of modern exploits. They occur when a program tries to store more data in a buffer than it can hold, leading to memory corruption. For a red teamer, mastering the buffer overflow is a rite of passage. It requires a deep understanding of CPU architecture, memory management, and assembly language.

In this article, we’ll move beyond the basic “hello world” of overflows and explore how to bypass modern defenses to achieve reliable code execution.

1. The Anatomy of a Stack Overflow

A buffer is just a temporary storage area in memory. When you allocate a local variable in C (like char buffer[64]), it is placed on the Stack.

The Stack Frame (x86)

When a function is called, a “Stack Frame” is created. It grows downwards (from high memory to low memory).

  1. Function Arguments: Pushed by the caller.
  2. Return Address (RET): Pushed by the CALL instruction. This tells the CPU where to go after the function finishes.
  3. Saved Base Pointer (EBP): Pushed by the function prologue.
  4. Local Variables: Allocated space for your buffer.

The Overflow

If your buffer grows up (low to high addresses) and you write past the end of it, you overwrite:

  1. Other local variables.
  2. The Saved EBP.
  3. The Return Address.

When the function executes RET (Return), it pops that address off the stack into the Instruction Pointer (EIP/RIP). If you control that address, you control execution flow.

2. Modern Defenses: The Trinity of Protection

Operating systems have evolved. An exploit from 1999 won’t work today without modification.

  1. ASLR (Address Space Layout Randomization): Randomizes where the stack, heap, and libraries (DLLs) are loaded. Hardcoded jump addresses will fail.
  2. DEP/NX (Data Execution Prevention / No-Execute): Marks the stack as Read/Write but NOT Execute. If you jump to your shellcode on the stack, the CPU throws an exception.
  3. Stack Canaries (Cookies): A random value placed before the EBP. The function prologue writes it; the epilogue checks it. If it changed, the program crashes before returning.

3. The Counter-Attack: Bypassing the Shield

Bypassing DEP with ROP (Return-Oriented Programming)

Since we can’t execute our own code, we use the program’s existing code. We find small snippets of code ending in a ret instruction (called gadgets).

We chain these gadgets on the stack.

  • Gadget 1: pop eax; ret (Loads a value into EAX).
  • Gadget 2: call VirtualProtect (Changes memory permissions).

Tools like Ropper or ROPgadget automate finding these snippets.

Bypassing ASLR with Leaks

To beat ASLR, we need an Information Leak. If we can leak a single pointer (e.g., a return address on the stack pointing to kernel32.dll), we can calculate the offset to the base of that DLL. Since the relative offsets inside the DLL are constant, one leak breaks the randomization for that entire module.

Bypassing Canaries

  1. Information Leak: Read the canary value (via a format string bug), then write it back correctly during your overflow.
  2. Brute Force: In forking servers (like older Apache), the canary is inherited by the child. You can guess byte-by-byte without the parent process crashing.

4. Crafting the Payload: Shellcode

Shellcode is position-independent machine code.

The “Bad Char” Problem

Many functions (like strcpy) stop copying when they hit a Null Byte (0x00). If your shellcode contains 0x00, only part of it gets copied. Other bad chars might be 0x0A (New Line) or 0x0D (Carriage Return).

Generating Clean Shellcode:

1

msfvenom -p windows/x64/exec CMD=calc.exe -b "\x00\x0a\x0d" -f python

The -b flag tells msfvenom to encode the payload to avoid these bytes. This adds a decoder stub at the beginning of your shellcode.

5. Development Workflow

  1. Fuzzing: Send increasingly long strings until the application crashes (A * 1000).
  2. Offset: Find exactly where the EIP/RIP overwrite happens (using pattern_create / pattern_offset).
  3. Bad Chars: Identify which characters corrupt the payload.
  4. Gadget Hunting: Find a JMP ESP (if no DEP) or ROP chain (if DEP) in a non-ASLR module.
  5. Weaponize: Replace the placeholder with shellcode + NOP sled.

Conclusion

Memory corruption is a complex field. While simple stack overflows are rare in modern browsers, they remain common in IoT devices, routers, and legacy enterprise software. By understanding the underlying mechanics of memory, you gain a perspective on security that few other disciplines can provide.

Happy hacking!

References