An introduction to reverse engineering

Reverse engineering is something I keep reaching for in offensive work, even when I don’t expect to. Whether you’re trying to figure out what an unknown implant does on disk, see exactly how an EDR is hooking your loader, dig through a proprietary network protocol that doesn’t have public docs, or just unpack a sample for indicators — at some point you stop being able to ask the source and have to read the binary.

This post covers the parts of RE you need to function. Executable-file structure, enough assembly to follow what the disassembler is showing you, the difference between static and dynamic analysis and when each one is the right call, and the anti-debugging and anti-VM tricks that will get in your way.

What reverse engineering is

#

Reverse engineering is what you do when you have a compiled binary and need to know how it works without the source. For offensive security, that shows up in a handful of recurring places:

• Malware analysis — understanding a sample well enough to defend against it or reuse its techniques.
• Vulnerability research — finding exploitable bugs in closed-source software.
• EDR evasion — figuring out where the product hooks so you can avoid the hooks.
• Anti-cheat / DRM analysis — understanding the protection so you can test or circumvent it.
• Custom protocol analysis — making sense of proprietary wire formats the target uses.

Executable file formats

#

Three formats cover almost everything you’ll touch: PE on Windows, ELF on Linux and BSD, and Mach-O on macOS and iOS.

PE (Windows)

#

PE is what Windows uses for .exe, .dll, and .sys. The headers and sections you’ll see in every analysis:

• DOS header — legacy “MZ” magic plus a pointer to the PE header.
• PE header — PE\0\0 signature and metadata.
• Optional header — required despite the name. Holds the entry point (AddressOfEntryPoint), image base, and section alignment.
• Section table — virtual address, raw size, and permissions for each section.
• .text — executable code.
• .data — initialized globals and statics.
• .rdata — read-only data, including string literals.
• .idata — the Import Address Table (IAT); external functions the binary calls. Usually the fastest way to figure out what a binary does.
• .edata — the Export Address Table (EAT); functions a DLL exports.
• .rsrc — icons, dialogs, embedded files.
• .reloc — relocation info used for ASLR.

Tools:

# Windows
dumpbin /headers program.exe
dumpbin /imports program.exe
dumpbin /exports program.dll

# Cross-platform
pefile (Python library)
peframe

ELF (Linux, BSD, most embedded)

#

• ELF header — magic bytes \x7fELF, architecture, entry point, and offsets to the program and section header tables.
• Program headers — describe segments for loading the binary into memory.
• Section headers — describe sections for linking and debugging.
• .text — executable code.
• .data — initialized globals.
• .bss — uninitialized globals, zero-filled at load time.
• .rodata — read-only data (strings, constants).
• .plt / .got — Procedure Linkage Table and Global Offset Table; the dynamic-linking machinery.
• .symtab / .dynsym — symbol tables, may be stripped on a hostile binary.

Tools:

# Display headers
readelf -h binary
readelf -S binary  # Section headers
readelf -l binary  # Program headers

# Display symbols
nm binary
objdump -T binary  # Dynamic symbols

# Quick file type identification
file binary

Mach-O (macOS, iOS)

#

Apple’s format. The interesting wrinkle is “fat binaries” or “Universal Binaries” — single files that contain code for multiple architectures (Intel x86_64 and Apple Silicon ARM64 today; PowerPC back in the day). lipo lets you split them apart.

Tools:

# Display headers
otool -h binary
otool -l binary   # Load commands

# For fat binaries
lipo -info binary

x86/x64 assembly

#

You don’t need to write assembly to reverse engineer — but you do need to read it. Most Windows and Linux desktop binaries are still x86_64. ARM64 is the norm on Apple Silicon, on Android, and increasingly on Windows-on-ARM laptops. The instruction sets are different, but the concepts (registers, the stack, calling conventions) carry over.

Registers

#

Registers are the CPU’s local variables — small, fixed-size storage slots that everything else reads from and writes to.

General Purpose Registers (x64):

Special Registers:

• RIP/EIP: Instruction Pointer - points to the next instruction to execute.
• RFLAGS/EFLAGS: Flags register - contains status flags (Zero, Carry, Sign, Overflow).

Common instructions

#

; Data Movement
mov eax, 5          ; EAX = 5
mov eax, [rbx]      ; EAX = value at address in RBX
lea rax, [rbx+rcx]  ; RAX = RBX + RCX (address calculation, no memory access)

; Arithmetic
add eax, ebx        ; EAX = EAX + EBX
sub eax, 10         ; EAX = EAX - 10
imul eax, ebx       ; EAX = EAX * EBX (signed)
inc eax             ; EAX = EAX + 1
dec eax             ; EAX = EAX - 1

; Bitwise
and eax, 0xFF       ; Mask lower byte
or eax, 1           ; Set bit 0
xor eax, eax        ; EAX = 0 (common idiom)
shl eax, 4          ; Shift left by 4 bits (multiply by 16)
shr eax, 1          ; Shift right by 1 bit (divide by 2)

; Comparison and Flags
cmp eax, ebx        ; Sets flags based on EAX - EBX (result discarded)
test eax, eax       ; Sets flags based on EAX AND EAX (checks if zero)

; Branching
jmp label           ; Unconditional jump
je/jz label         ; Jump if Equal/Zero
jne/jnz label       ; Jump if Not Equal/Not Zero
jg/jge label        ; Jump if Greater/Greater or Equal (signed)
ja/jae label        ; Jump if Above/Above or Equal (unsigned)
jl/jle label        ; Jump if Less/Less or Equal (signed)
jb/jbe label        ; Jump if Below/Below or Equal (unsigned)

; Function Calls
call function_addr  ; Push return address, jump to function
ret                 ; Pop return address, jump to it

; Stack Operations
push rax            ; Push RAX onto stack
pop rbx             ; Pop top of stack into RBX

Calling conventions

#

You can’t follow function calls in disassembly without knowing where the arguments live. The three you’ll meet most often:

Static analysis

#

Static analysis means reading the binary without running it. Safer — you can’t accidentally execute malware — but a determined defender can obfuscate or pack the code enough that what you’re reading isn’t what runs.

Disassembly

#

A disassembler turns raw bytes back into assembly. The tools, roughly in order of how often I reach for them:

# Quick disassembly with objdump
objdump -d -M intel binary | less

# Disassemble specific function
objdump -d -M intel binary | grep -A 50 "<main>:"

Decompilation

#

A decompiler tries to reconstruct higher-level C or C++ from assembly. The output is hugely easier to read than disassembly — but you’ve lost variable names, comments, type information, and a chunk of the original control flow. Read it skeptically; the decompiler is guessing.

Ghidra’s workflow: create a project, import the binary, run Auto-Analyze, then navigate to whatever function you want to read. The decompiler window shows reconstructed C alongside the assembly. As you figure out what a function does, rename it (right-click → “Edit Function Signature” or “Rename Variable”). Define structs as you discover them. The renames stick and propagate to every call site, which makes the next pass through the binary much faster.

Where to start reading

#

The entry point is the lowest-friction starting point — AddressOfEntryPoint in PE, e_entry in ELF. But the runtime entry point isn’t the same as main(); the C runtime does init work first. Most disassemblers will find main for you.

For interesting strings, run strings binary | grep -i something and then find the cross-references in Ghidra. Strings are usually the fastest way to triangulate what a binary cares about.

For interesting behavior, check the imports. A binary calling CreateRemoteThread, VirtualAlloc, WriteProcessMemory, socket, and connect is up to something specific. The IAT is the cheap-and-fast answer to “what does this thing do.”

Dynamic analysis

#

Static analysis tells you what’s in the binary. Dynamic analysis tells you what it actually does — runtime values, decrypted strings, which branches actually execute. Run it in a VM you don’t mind throwing away.

Debuggers

#

A debugger lets you pause, inspect, and step.

The commands you’ll use most:

# GDB with GEF
gdb ./binary
> break main                 # Set breakpoint at main
> run                        # Start execution
> info registers             # View all registers
> x/10x $rsp                 # Examine 10 hex words at stack pointer
> stepi                      # Step one instruction
> nexti                      # Step over (don't follow calls)
> continue                   # Resume execution
> disassemble                # Disassemble current function

Behavioral Monitoring

#

Instead of debugging step-by-step, observe the program’s interactions with the system.

• Process Monitor (Windows): Logs file, registry, network, and process activity.
• API Monitor (Windows): Logs Windows API calls with arguments.
• strace (Linux): Logs system calls.
• ltrace (Linux): Logs library calls.
• Wireshark: Captures network traffic.

Instrumentation with Frida

#

Frida is a dynamic instrumentation toolkit that lets you inject JavaScript into running processes to hook functions, modify behavior, and trace execution.

# Install Frida
pip install frida-tools

# List running processes
frida-ps

# Attach to a process and run a script
frida -p <PID> -l my_script.js

Example Frida Script (Hooking a Function):

// my_script.js
Interceptor.attach(Module.findExportByName(null, "strcmp"), {
    onEnter: function(args) {
        console.log("strcmp called!");
        console.log("  arg0: " + Memory.readUtf8String(args[0]));
        console.log("  arg1: " + Memory.readUtf8String(args[1]));
    },
    onLeave: function(retval) {
        console.log("  return: " + retval);
    }
});

This is invaluable for bypassing SSL pinning, tracing encryption routines, or understanding complex logic without laboriously stepping through every instruction.

Anti-reverse-engineering

#

Malware authors, EDR vendors, and commercial software-protection schemes all use tricks to make analysis harder. The categories are well-established; the specific instances are a moving target.

Anti-debugging

#

The common Windows checks:

• IsDebuggerPresent() — reads the BeingDebugged flag from the PEB.
• CheckRemoteDebuggerPresent() — checks for an attached remote debugger.
• NtQueryInformationProcess(ProcessDebugPort) — queries the debug port directly.
• Timing checks — measures execution duration; debuggers slow things down.
• Hardware breakpoint detection — inspects debug registers DR0–DR7.
• Exception tricks — throws exceptions that debuggers handle differently than the runtime would.

Bypasses, roughly in order of effort:

• Patch the check (turn JZ into JMP or NOP NOP).
• Use x64dbg’s ScyllaHide plugin, which masks most of the common checks.
• Configure the debugger to pass exceptions through to the program.
• Manually clear BeingDebugged in the PEB.

Anti-VM

#

Malware often goes dormant in a VM to evade sandbox analysis. The checks fall into a few categories:

• VM artifacts — VMware or VirtualBox drivers, registry keys, MAC prefixes, process names like vmtoolsd.exe.
• Hardware shape — low RAM, few CPU cores, undersized disk.
• Timing — rdtsc to detect hypervisor overhead.
• Hypervisor detection — cpuid with specific leaf values.

Bypasses:

• Use a physical analysis machine if you can.
• Mask the artifacts — uninstall VMware Tools, randomize MACs, rename suspicious files.
• Use a stealth-VM config specifically designed to evade detection.
• Patch the checks in the binary.

Packing and obfuscation

#

Packers (UPX, Themida, VMProtect) compress or encrypt the real code and ship a small “stub” that decrypts it into memory at runtime. Static analysis on a packed binary only shows you the stub.

Detection tools include Detect It Easy (DIE), PEiD, and ExeInfoPE. High-entropy sections (binwalk -E) are a tell — encrypted data looks random.

Unpacking is either manual (run in a debugger, wait until the real code is in memory, dump) or automatic (upx -d packed.exe works for UPX). Commercial packers like VMProtect generally don’t have automatic unpackers worth using.

Obfuscation tactics you’ll see alongside packing:

• Control-flow flattening — replaces normal loops and conditionals with a giant dispatcher switch.
• Dead code insertion — adds instructions that have no effect.
• Instruction substitution — replaces simple ops with equivalent but more complex sequences.
• String encryption — strings are decrypted at the moment of use.

A practical workflow

#

The order I work through a new binary, more or less:

1. Initial recon

#

# Identify file type and architecture
file target.exe

# Check for packing/encryption
Detect-It-Easy target.exe

# Extract strings for quick wins (IPs, URLs, passwords, debug messages)
strings -n 8 target.exe | grep -iE "(http|password|key|secret|admin)"

# Calculate hashes for research
sha256sum target.exe
# Search hash on VirusTotal, Hybrid Analysis, etc.

2. Static analysis pass

#

1. Open in Ghidra (or whatever you use).
2. Run Auto-Analysis.
3. Check the Imports list — what APIs does it call? Networking? File I/O? Process manipulation?
4. Check the Strings window — anything interesting?
5. Navigate to main() or the entry point.
6. Identify the high-level structure: initialization, main loop, cleanup.
7. Rename functions and variables as you figure them out.

3. Dynamic verification

#

1. Set up a safe environment — VM snapshot, network isolation or FakeNet.
2. Open the binary in a debugger (x64dbg, GDB).
3. Set breakpoints at functions you identified as interesting (encryption routines, network calls, config parsing).
4. Run and observe.
5. Examine register/memory contents at breakpoints to understand data flow.

4. Patching

#

If a security check (anti-debug, license check, etc.) is blocking your analysis:

1. Find the check in the disassembly.
2. Identify the conditional jump (for example, JZ 0x401050 - Jump if Zero).
3. Patch it to always pass or always fail:
   • JZ -> JMP (always jump)
   • JZ -> NOP NOP (never jump, continue to next instruction)
4. Save the patched binary or apply the patch in the debugger.

Where to practice

#

Crackmes and RE-flavored CTFs:

• Crackmes.one — user-submitted RE challenges across difficulty levels.
• Microcorruption — embedded CTF with a custom debugger interface.
• PicoCTF — beginner-friendly CTF with a steady stream of RE challenges.
• OverTheWire Narnia — exploit development warm-ups.

Reading material:

• Practical Malware Analysis — still the book to read for malware RE.
• Reverse Engineering for Beginners — Dennis Yurichev’s free book; comprehensive x86/ARM coverage.
• Ghidra tutorials on YouTube — quality varies; pick a series and stick with it.

Closing

#

The thing nobody tells you about RE is that most of it is patience. Reading a few hundred lines of decompiled C, renaming a function, getting one more piece of the puzzle, then realizing the function you just named is called from another function you also haven’t named yet. The flashy parts — unpacking the protected binary, bypassing the anti-debug, finding the bug — are the small minority of what you spend time on.

The skill that actually improves over time is the habit of stopping to rename and annotate as you go. It feels like overhead and it makes the second pass through any binary three times faster than the first. That’s the part of RE I’d tell my past self to take more seriously.

References

#

• Ghidra Official Site
• x64dbg Debugger
• GEF - GDB Enhanced Features
• Frida Dynamic Instrumentation
• Practical Malware Analysis - No Starch Press
• Reverse Engineering for Beginners - Dennis Yurichev
• Crackmes.one
• Intel x86/x64 Instruction Set Reference