Advanced Malware Analysis: Disassembly Techniques

Disassembling malicious code, converting the binary back into something you can actually read, is one of the core skills in malware analysis. Malware authors count on their code being opaque; disassembly is how you take that advantage away from them.

This post walks through the static and dynamic disassembly techniques a malware analyst actually reaches for: manual disassembly, the major automated tools, decompilers, debuggers, and dynamic binary instrumentation.

What is disassembly?

#

Disassembly is the process of converting binary code into human-readable assembly language instructions. It’s essential in malware analysis because malware authors routinely obfuscate their code specifically to make it hard to analyze. Disassembling malware can reveal its functionality, identify the system calls and functions it uses, and show whether the code is packed or otherwise obfuscated.

There are two main approaches: static disassembly, which analyzes the binary without running it, and dynamic disassembly, which executes the code and analyzes it as it runs.

Static disassembly techniques

#

Static disassembly is usually the first pass on a new sample, since it doesn’t require detonating anything.

Manual disassembly

#

Manual disassembly means translating machine code into assembly instructions by hand rather than relying on a tool to do it for you. It’s slower, but it’s the only option when the code is complex or obfuscated enough that automated tools choke on it. It requires a solid grasp of the target architecture’s assembly language, calling conventions, and executable file format.

There are a few steps involved:

Identify the entry point

#

The entry point is the first instruction executed when the program loads into memory. You find it by examining the executable’s header. For example, objdump will report it directly:

$ objdump -f malware.bin

malware.bin:     file format elf64-x86-64
architecture: i386:x86-64, flags 0x00000112:
EXEC_P, HAS_SYMS, D_PAGED
start address 0x400440

The entry point here is 0x400440.

Trace the program’s execution path

#

From the entry point, you follow the flow of control through the functions and subroutines the program calls. This is the slow, tedious part, and it requires annotating as you go to keep track of program state and flag anything interesting. objdump -d gives you the disassembly to trace through:

$ objdump -d malware.bin

...

400440:       48 83 ec 08             sub    $0x8,%rsp
400444:       c7 44 24 04 00 00 00    movl   $0x0,0x4(%rsp)
40044b:       00
40044c:       48 8d 44 24 04          lea    0x4(%rsp),%rax
400451:       48 89 04 24             mov    %rax,(%rsp)
400455:       48 8d 45 f8             lea    -0x8(%rbp),%rax
400459:       be 08 00 00 00          mov    $0x8,%esi
40045e:       ba 02 00 00 00          mov    $0x2,%edx
400463:       48 89 c7                mov    %rax,%rdi
400466:       e8 c5 fe ff ff          callq  400330 <memcpy@plt>
...

The malware starts at 0x400440 and, among other operations, calls memcpy.

Identify key functions and system calls

#

File I/O, network communication, encryption, and process injection are the calls worth flagging first, since they’re what actually tell you what the malware does. Check the parameters going in and the values coming back:

$ objdump -d malware.bin

...

4004a3:       bf 01 00 00 00          mov    $0x1,%edi
4004a8:       e8 93 fe ff ff          callq  400340 <close@plt>
4004ad:       48 8d 45 f8             lea    -0x8(%rbp),%rax
4004b1:       ba 08 00 00 00          mov    $0x8,%edx
4004b6:       be 01 00 00 00          mov    $0x1,%esi
4004bb:       48 89 c7                mov    %rax,%rdi
4004be:       e8 6d fe ff ff          callq  400330 <write@plt>
...

Here the malware calls close and write, both file I/O operations.

Reconstruct high-level code

#

Once you’ve identified the key functions and calls, you can start turning the disassembly into pseudocode that describes the program’s behavior at a level a human can actually follow. Say you’re looking at a sample that encrypts files on an infected system; after tracing the encryption routine and the surrounding file I/O, you might end up with something like:

for each file in the target directory:
    open the file for reading
    read the contents of the file
    encrypt the contents of the file using the encryption algorithm
    close the file
    open the file for writing
    write the encrypted contents to the file
    close the file

That pseudocode is the payoff: it’s what you’d hand to someone else on the team without making them read raw disassembly first.

Automated disassembly tools

#

Manual disassembly doesn’t scale to large or heavily obfuscated binaries, which is where IDA Pro, Ghidra, and Binary Ninja come in. All three disassemble the binary automatically and layer on cross-referencing, call graph analysis, and (in varying degrees) decompilation.

It’s worth being precise about what each tool shows you by default, since they don’t agree. IDA Pro’s default view is the raw disassembly listing, assembly mnemonics, one instruction per line. Its decompiler (Hex-Rays, more on that below) is a separate view you switch to. Ghidra and Binary Ninja both default to showing decompiled, C-like pseudocode alongside the raw listing, which is why their output looks noticeably different from IDA’s even when you’re looking at the exact same function.

IDA Pro

#

IDA Pro supports Windows PE, Linux ELF, and macOS Mach-O, among other formats, and pairs static analysis with debugging and scripting. Loading a binary and looking at the disassembly listing for main gives you something like this:

.text:0000000000401000 ; =============== S U B R O U T I N E =======================================
.text:0000000000401000
.text:0000000000401000 ; int __cdecl main(int argc, const char **argv, const char **envp)
.text:0000000000401000 _main           proc near               ; CODE XREF: __libc_start_main+23↑p
.text:0000000000401000
.text:0000000000401000 var_50          = qword ptr [-50h]
.text:0000000000401000 var_48          = qword ptr [-48h]
.text:0000000000401000 var_40          = qword ptr [-40h]
.text:0000000000401000 var_38          = qword ptr [-38h]
.text:0000000000401000 var_30          = qword ptr [-30h]
.text:0000000000401000 var_20          = qword ptr [-20h]
.text:0000000000401000 var_18          = qword ptr [-18h]
.text:0000000000401000 var_10          = qword ptr [-10h]
.text:0000000000401000 var_8           = qword ptr [-8h]
.text:0000000000401000 argc            = dword ptr  8
.text:0000000000401000 argv            = qword ptr  10h
.text:0000000000401000 envp            = qword ptr  18h
.text:0000000000401000
.text:0000000000401000                 push    rbp
.text:0000000000401001                 mov     rbp, rsp
.text:0000000000401004                 mov     [rbp+var_8], rdi
.text:0000000000401008                 mov     [rbp+var_10], rsi
.text:000000000040100c                 mov     eax, 0
.text:0000000000401011                 pop     rbp
.text:0000000000401012                 retn
.text:0000000000401012 _main           endp

You can see the entry point at 0x401000, the main function’s stack layout (all those var_XX locals), and its parameters. There’s no obfuscation getting in the way here, so it’s a clean read; real malware rarely looks this tidy.

Ghidra

#

Ghidra, NSA’s free and open source reverse engineering suite, covers the same range of formats as IDA Pro. Its decompiler runs automatically, so opening main gets you pseudocode straight away rather than a raw listing:

entry:
undefined8 main(void)

{
  int32_t iVar1;

  iVar1 = puts("Hello, world!");
  return CONCAT71((int7)(iVar1 >> 8),1);
}

That CONCAT71 call is Ghidra being pedantic about how main’s 8-byte return value gets built from a smaller puts return, not a bug in the code. It’s a good example of decompiler output needing a little translation of its own before it reads naturally.

Binary Ninja

#

Binary Ninja is a newer commercial disassembler with a strong API for scripting analysis, and like Ghidra it decompiles by default:

int main(void)

{
  puts("Hello, world!");
  return 0;
}

Binary Ninja’s decompiler tends to produce cleaner, more idiomatic-looking C than Ghidra’s for straightforward code like this; the gap narrows fast once you’re looking at real obfuscated malware instead of a toy example.

Decompilers

#

A decompiler goes a step further than disassembly, reconstructing something closer to the original source language (C, C++, or occasionally Java) instead of raw assembly. That higher-level view is a real speed boost when you’re hunting for vulnerabilities or trying to understand program logic quickly, though it’s an approximation, not a perfect reversal, and it can mislabel or misrepresent code the compiler optimized aggressively.

IDA Pro’s decompiler is a separate product called Hex-Rays (historically an add-on, though it now ships bundled for most supported architectures). Its output looks structurally similar to Ghidra’s, generally recognizable, but with sub_XXXXXX naming for anything it can’t identify, since Hex-Rays doesn’t always propagate string references into the pseudocode the way Ghidra’s decompiler does:

int __cdecl main(int argc, const char **argv, const char **envp)
{
  int result; // eax

  sub_401B00("Hello, World!");
  result = 0;
  return result;
}

Here Hex-Rays couldn’t resolve sub_401B00 to a known function name (it’s actually puts), so it left the raw address-based label in place. Ghidra’s decompiler is a little more aggressive about resolving those references, which is part of why its earlier example already showed puts by name.

Java decompilers

#

Java decompilers turn compiled .class bytecode back into Java source, useful when you only have the compiled artifact. JD-GUI, JAD, and Fernflower are the common choices. Loading a class file into JD-GUI might get you something like:

public static void main(String[] args) throws Exception {
    String message = "Hello, World!";
    Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5PADDING");
    byte[] ivBytes = new byte[16];
    Arrays.fill(ivBytes, (byte) 0);
    IvParameterSpec iv = new IvParameterSpec(ivBytes);
    SecretKeySpec key = new SecretKeySpec("mysecretpassword".getBytes("UTF-8"), "AES");
    cipher.init(Cipher.ENCRYPT_MODE, key, iv);
    byte[] encrypted = cipher.doFinal(message.getBytes());
    System.out.println(new String(encrypted));
}

That last line, printing raw encrypted bytes through new String(encrypted), is a real anti-pattern (it mangles binary data through a lossy charset decode), but it’s exactly the kind of sloppy code you actually find in the wild. Malware authors aren’t necessarily good programmers.

Python decompilers

#

Python decompilers turn .pyc bytecode back into Python source. uncompyle6 covers Python 1.0 through 3.8 but is no longer actively maintained past that range; decompyle3, a fork by the same author, handles 3.7/3.8 more reliably. pycdc is a separate C++-based decompiler that reaches into some newer bytecode versions the other two can’t touch. None of the classic tools handle Python 3.9+ well; that gap is still an open problem in the tooling. Loading bytecode into uncompyle6 for something in its supported range might produce:

def main():
    message = "Hello, World!"
    key = b"mysecretpassword"
    iv = b'\x00' * 16
    aes = AES.new(key, AES.MODE_CBC, iv)
    ciphertext = aes.encrypt(pad(message.encode(), AES.block_size))
    print(ciphertext)

Dynamic disassembly techniques

#

Static analysis only gets you so far against packed or heavily obfuscated malware. At some point you need to actually run the sample and watch what it does, ideally somewhere it can’t do any damage.

Debuggers

#

Debuggers let you step through a running program, set breakpoints, and inspect memory as execution happens, which is how you catch behavior that static analysis alone would miss.

gdb

#

gdb is the standard debugger on Linux and is built around ELF binaries; that’s its native habitat and where it’s genuinely strong. It can be cross-compiled for other formats, but its macOS support has effectively been dead since Apple dropped it from Xcode in favor of LLDB (which is also the only realistic option for debugging native Apple Silicon binaries). For Linux malware, though, gdb is still the workhorse:

(gdb) file malware
Reading symbols from malware...
(gdb) break main
Breakpoint 1 at 0x40113c
(gdb) run
Starting program: /home/user/malware

Breakpoint 1, 0x000000000040113c in main ()

From here you can step through the code and inspect memory as the program runs.

OllyDbg

#

OllyDbg was the standard Windows debugger for years, built for 32-bit PE binaries with a solid disassembly view built in. It hasn’t been meaningfully updated in a long time and never gained 64-bit support, so most Windows malware analysts have moved to x64dbg, its actively maintained, open source, 32- and 64-bit-capable successor. The workflow is the same either way:

1. File > Open
2. Select malware.exe
3. Press F2 to open the breakpoint window
4. Right click on the empty area of the window
5. Click on “New breakpoint”
6. Enter the address of the entry point of the program
7. Click on “OK”
8. Press F9 to run the program

WinDbg

#

WinDbg is Microsoft’s own debugger, built for Windows PE and COFF binaries and still the standard choice for kernel-level debugging on Windows:

1. File > Open executable
2. Select malware.exe
3. Type bp main to set a breakpoint at the entry point of the program
4. Type g to run the program

Once the breakpoint hits, you can step through the code and examine memory the same way you would with gdb or x64dbg.

Dynamic binary instrumentation (DBI)

#

DBI lets you observe and manipulate a running program’s behavior directly, hooking system calls, API calls, and arbitrary functions in real time. It’s a step beyond a debugger: instead of pausing execution at breakpoints, DBI rewrites or instruments the code as it runs.

PIN

#

Pin is Intel’s DBI framework for IA-32 and x86-64, and it’s also what underlies several of Intel’s own performance tools (VTune, Advisor, SDE). Building and running a Pintool against a sample looks like:

source/tools/MyPinTool/obj-intel64/MyPinTool.so -- /path/to/malware

That loads the tool and starts instrumenting the target as it runs, letting you monitor system calls and intercept API calls in real time.

DynamoRIO

#

DynamoRIO started as a research collaboration between MIT and Hewlett-Packard in the early 2000s, was briefly commercialized by Determina, and has been maintained by VMware since 2007. Running a sample under one of its instrumentation tools:

> drrun -c /path/to/dynamorio/samples/simple/instr.dll -- malware.exe

Frida

#

Frida was created by Ole André Vadla Ravnås, a security researcher formerly at NowSecure, and remains an independent open source project. It covers Windows, Linux, Android, iOS, and macOS, which makes it the obvious choice for mobile malware in particular. Attaching to a running Android process and hooking system() calls:

import frida

def on_message(message, data):
    print(message)

process = frida.get_usb_device().attach('com.example.malware')
script = process.create_script("""
Interceptor.attach(Module.findExportByName(null, "system"), {
    onEnter: function(args) {
        console.log("[*] system(" + args[0].readUtf8String() + ")");
    }
});
""")
script.on('message', on_message)
script.load()

Virtual machines (VMs)

#

Running any of this against live malware means doing it somewhere disposable. VirtualBox, VMware, and QEMU are the usual choices for isolating a sample from the host system, snapshotting before detonation, and reverting cleanly afterward.

Examples

#

Two well-documented cases show how these techniques come together in practice.

Stuxnet worm

#

Stuxnet targeted industrial control systems, chained together four Windows zero-day exploits, and was built to attack very specific hardware configurations. The sample was packed and layered with obfuscation, so analysts leaned on automated tools first and fell back to manual disassembly for the parts that resisted them.

IDA Pro was central to identifying Stuxnet’s main functions: analysts found it spread via infected USB drives and network shares using a Windows shortcut (.LNK) vulnerability, then used additional exploits to reach the specific Siemens PLC hardware it was targeting. Debugging with WinDbg surfaced the custom protocol Stuxnet used to talk to its command and control infrastructure, along with its ability to modify code on infected systems to stay hidden.

WannaCry ransomware

#

WannaCry hit hundreds of thousands of computers worldwide in May 2017, spreading through a Windows SMB vulnerability (EternalBlue, an exploit developed by the NSA and leaked by the Shadow Brokers) and encrypting files on every system it reached. Like Stuxnet, the sample was packed and obfuscated, so the analysis combined automated and manual disassembly.

IDA Pro exposed the encryption routine and the key system calls driving the worm’s spread. WannaCry’s command and control traffic ran over Tor, using a bundled Tor client routed through a local SOCKS5 proxy, which was itself a useful signal for defenders once identified. Debugging with gdb showed just how fast the self-propagation was: infected systems scanned for and compromised new vulnerable hosts in seconds, which is what let it spread as far as it did before anyone could react.

Conclusion

#

Disassembly techniques are what let you turn opaque binary code into something you can actually reason about, and identify the specific functions and system calls a piece of malware relies on. Combining automated tools with manual disassembly is how analysts worked out what Stuxnet and WannaCry were actually doing under the packing and obfuscation.

Whether you’re on a red team, doing pen testing, or working malware analysis full time, these are foundational skills. IDA Pro, Ghidra, WinDbg, and gdb (or their modern equivalents like x64dbg) cover most of what you’ll need to get started.