Dynamic malware analysis: unpacking, memory forensics, and the anti-analysis arms race

Malware analysis is the work of figuring out what a piece of someone else’s code does, when that code was deliberately written to make figuring it out hard. The defenders’ job is to extract IOCs and TTPs the rest of the organization can detect and respond to; the operators’ job (on the red team side) is to understand what the defenders are seeing so the next implant doesn’t look like the last one. Both roles need the same skills.

The two-track approach is universal: static analysis examines the file without running it (PE headers, strings, imports, disassembled code in Ghidra or IDA), and dynamic analysis runs the sample in a controlled environment and observes its behavior (debugger, sandbox, memory dump, packet capture). Static is faster and safer; dynamic is more conclusive and survives obfuscation that static analysis chokes on. The work in 2026 is almost always a back-and-forth: triage statically, run dynamically to confirm or extract the next stage, return to static to read the unpacked payload, repeat.

This walkthrough covers the dynamic side: PE triage as a prerequisite, x86_64 assembly fundamentals for the analyst who doesn’t write assembly, debugging with x64dbg, unpacking via the ESP trick and IAT reconstruction with Scylla, memory forensics with Volatility, malicious-document analysis with olevba, network monitoring with Wireshark, and the anti-analysis techniques the malware uses to detect that it’s being analyzed and shut down. Case studies pull from WannaCry, TrickBot, and Emotet.

Static triage: PE headers and assembly fundamentals

#

Before running anything, look at it. Static triage tells you whether the sample is worth detonating, what it might do when it runs, and what to watch for in the dynamic phase.

Reading the PE header

#

Every Windows executable follows the Portable Executable format. PEStudio (Marc Ochsenmeier) and Detect It Easy (DIE) parse the structure and surface the fields that matter to an analyst:

• Imports. What APIs does the binary call? InternetOpenUrl and WinHttpOpen mean network activity. CryptEncrypt or BCryptEncrypt in combination with FindFirstFileW suggests ransomware. VirtualAllocEx + WriteProcessMemory + CreateRemoteThread is the canonical Windows code-injection trio. IsDebuggerPresent flags anti-analysis intent before the binary even runs.
• Section table. Look for sections with entropy above 7.0 (out of 8.0). That’s compressed or encrypted data, which usually means the executable is packed. Unusual section names (.upx0, .themida, .aspack) name-and-shame the packer directly.
• Strings. Run strings on the binary first. URLs, IPv4 patterns, hardcoded keys, PDB paths (C:\Users\dev\source\repos\TrickBot\Release\loader.pdb is exactly as informative as it looks), error messages in Russian or Chinese. Any of these can identify the malware family in seconds without running it.
• Imphash. A hash of the import table, useful as a family signature. Two samples with the same imphash are usually built from the same source tree even if everything else is obfuscated.

x86_64 assembly fundamentals

#

You don’t need to write assembly. You need to read it.

• Registers worth knowing.
  • RAX: return values from function calls; the most-watched register during debugging.
  • RCX, RDX, R8, R9: the first four arguments to a function under the Windows x64 calling convention. System V (Linux/macOS) uses RDI, RSI, RDX, RCX, R8, R9 instead; different OS, different convention.
  • RSP: stack pointer; arguments past the first four spill onto the stack.
  • RIP: instruction pointer; what executes next.
• Common instructions.
  • MOV dst, src: move data between registers, memory, or immediates.
  • XOR rax, rax: zero a register; faster than MOV rax, 0 and a smaller encoding, so the compiler emits it constantly.
  • JNZ / JZ: jump if not zero / jump if zero, based on the zero flag set by the prior comparison. Control flow lives in these conditional jumps.
  • CALL: invoke a function; pushes the return address and jumps.
  • RET: pop the return address and jump back. Functions begin with a prologue (PUSH RBP; MOV RBP, RSP; SUB RSP, ...) and end with an epilogue and RET.

Debugging with x64dbg

#

OllyDbg v1 was the reference debugger for the better part of a decade, but v1.10 (2013) was the final release and it’s 32-bit-only. The v2.01 alpha shipped the same year and went nowhere. The modern equivalent is x64dbg (the umbrella project containing x32dbg and x64dbg), led by Duncan Ogilvie (Mr. Exodia) and actively maintained by a contributor community. It’s open-source, supports both x86 and x64 binaries, and has the plugin ecosystem (Scylla, ScyllaHide, xAnalyzer) that the older debuggers don’t.

Unpacking with the ESP trick

#

A packed binary contains a small unpacker stub plus the encrypted real payload. At runtime, the stub allocates memory, decrypts the payload into it, fixes up the imports, and jumps to the original entry point (OEP) of the unpacked code. The analyst’s job is to let that decryption happen, then dump the unpacked image before the malicious code actually runs.

The classic technique is the ESP trick, which exploits the fact that most packer stubs begin by saving every register with PUSHAD and restore them with POPAD right before jumping to the OEP. Catching the moment POPAD runs puts you one instruction away from the unpacked entry point.

1. Open the sample in x64dbg.
2. Run until execution stops at the entry point. Step through the first few instructions until you see a PUSHAD (or PUSH RAX; PUSH RCX; ... series on x64). Step over it with F8.
3. Right-click the ESP (or RSP) register, choose Follow in Dump.
4. In the dump pane, set a hardware breakpoint on access on the first four bytes. This catches the future POPAD that reads those bytes back.
5. Run with F9. The packer stub does its decryption work while you wait.
6. The hardware breakpoint fires at the POPAD instruction.
7. Single-step until you see a large unconditional jump like JMP 0x00401000. That target is the OEP.
8. Open the Scylla plugin, click IAT Autosearch, then Get Imports, then Dump to write the unpacked image to a new file. Scylla rewrites the PE header so the result is a clean executable that opens in IDA or Ghidra for static analysis.

Rebuilding the IAT

#

Aggressive packers destroy or hide the Import Address Table, so the unpacked image can’t be loaded directly even after dumping. Scylla’s IAT reconstruction handles the common case: it walks the running process’s memory looking for resolved API addresses, matches them against known DLL exports, and writes a fresh IAT into the dumped image. If Scylla’s autosearch misses some imports, the manual flow is to set a breakpoint on GetProcAddress, log every call, and reconstruct the table by hand from the resulting list.

Case study: the WannaCry kill switch

#

WannaCry is the canonical worked example for analyst-side dynamic analysis because the story includes both a working exploit chain (EternalBlue + DoublePulsar, leaked from the NSA by the Shadow Brokers) and a deliberate kill switch the malware authors built in, and because Marcus Hutchins (MalwareTech) found that kill switch in something close to real-time on May 12, 2017.

The hook: WannaCry’s bootstrap code attempted to connect to a long random-looking domain (iuqerfsodp9ifjaposdfjhgosurijfaewrwergwea.com) before doing anything else. If the connection succeeded, the malware exited. If it failed (because the domain didn’t resolve), the malware proceeded to spread. The authors likely included this as a sandbox-detection check, since some sandboxes resolve every DNS query to a fake IP and pretend the connection succeeded.

Hutchins, watching live traffic from infected hosts, noticed the unresolved domain in his sinkhole feed. He registered it for about $10.69 (£8 at the time). Every new WannaCry infection immediately resolved the domain, got a successful connection, and exited before reaching the propagation code. The single registration halted the worm’s spread across Europe and most of the US that afternoon.

The dynamic analysis path that recovers this kill-switch logic with x64dbg or any modern debugger:

1. Attach to a WannaCry sample running in an isolated VM (no network bridge, disposable snapshot).
2. Set a breakpoint on InternetOpenA (or WinHttpOpen, depending on the sample variant).
3. Run the sample. When the breakpoint hits, the arguments on the stack show the URL the malware is about to fetch.
4. Examine the string. That’s your kill switch domain.
5. Continue execution, set a breakpoint on the function that handles the HTTP response, and watch the comparison: success means exit, failure means propagate.

Two operator-relevant lessons:

• Kill switches are common. Plenty of malware families include them, either as sandbox checks or as a deliberate “off switch” the authors want to retain. Finding them statically or in a debugger is one of the highest-value first findings in any analysis.
• Connections from a debugger or sandbox don’t behave like connections from the real world. Many sandboxes fake DNS resolution to keep samples from talking to actual C2; that fakery is exactly what WannaCry’s kill switch was designed to detect. Modern analysis VMs use INetSim or FakeNet-NG to simulate the internet locally, but the response patterns those tools produce are themselves detectable. The arms race goes both ways.

Memory forensics with Volatility

#

Memory forensics catches what static and disk-based analysis miss. Process-hollowed implants, reflectively loaded DLLs, fileless PowerShell payloads, decrypted strings sitting in heap, network connections in the kernel socket tables, all of these live in memory and disappear when the system is powered down. A snapshot of RAM (via winpmem, DumpIt, or VM-level memory capture from VMware or VirtualBox) preserves the evidence; Volatility parses it.

There are two versions in the wild that you need to know which one you’re talking to:

• Volatility 2 (Python 2, profile-based). Commands look like vol.py -f memory.dmp --profile=Win10x64_18362 netscan. The profile is the OS build, and you find it with imageinfo. Volatility 2 is in maintenance mode but still widely used because Vol3 doesn’t yet have feature parity for every plugin.
• Volatility 3 (Python 3, auto-detected symbol tables). First public release was 2019. No more profiles; the framework figures out the OS version from the kernel signatures in the dump. Commands use namespaced plugin names: vol3 -f memory.dmp windows.info, vol3 -f memory.dmp windows.netscan, vol3 -f memory.dmp windows.malfind.

Case study: TrickBot in memory

#

TrickBot is the modular banking trojan that ran from 2016 through December 2021, when the operators wound it down and moved to BazarLoader/Conti. Its memory footprint is one of the most-analyzed in the field, and it’s a good worked example because it ships almost every memory-evasion trick at once: process hollowing into svchost.exe, encrypted module strings, in-memory module loading, and credential theft via injected hooks in browsers.

A standard Volatility 3 triage on a TrickBot-infected system:

# What is this image?
vol3 -f trickbot.dmp windows.info

# Walk the process tree, looking for hollowed children of legit parents
vol3 -f trickbot.dmp windows.pstree

# Find injected code (executable + private memory regions not backed by a file)
vol3 -f trickbot.dmp windows.malfind

# Active network connections, where is the C2?
vol3 -f trickbot.dmp windows.netscan

# Hidden processes the rootkit cross-view detects
vol3 -f trickbot.dmp windows.psscan

# Dump a specific process for static analysis
vol3 -f trickbot.dmp windows.dumpfiles --pid 1234

malfind is the workhorse plugin for finding TrickBot’s injected modules: it walks every process’s VAD tree looking for memory regions that are marked PAGE_EXECUTE_READWRITE, are private (not file-backed), and contain what looks like executable code (MZ header, common opcodes at the start). TrickBot’s hollowed payloads light up malfind immediately. netscan shows the C2 callbacks, TrickBot typically used HTTPS to a rotating set of IPs registered to known bulletproof hosting; the IPs in any given dump can be cross-referenced against public TrickBot IOC feeds (abuse.ch’s Feodo Tracker is the standard source).

Malicious documents

#

Most ransomware in 2026 still arrives via email, and most of those emails carry an Office document with a malicious payload. Microsoft disabled macros from the internet by default in 2022, which moved the attacker tradecraft toward container-bypass tricks (ISO and IMG mounts that strip the Mark-of-the-Web), OneNote attachments (which still execute embedded scripts), HTML smuggling, and LNK-in-archive delivery. The VBA-macro flow is still relevant, though, because legacy .doc and .xls files don’t carry MOTW the way OOXML does, and because internal corporate email frequently exempts macro-bearing attachments from the global block.

Document format basics

#

Legacy Office files (.doc, .xls, .ppt) use the OLE (Object Linking and Embedding) compound-document format. Modern Office files (.docx, .xlsx, .docm, .xlsm) are ZIP archives of XML and embedded streams. The .docm and .xlsm macro-bearing variants are the most common phishing payloads.

The two reference tools:

• oledump.py by Didier Stevens. Walks the OLE stream structure, lists every stream and its size, and lets you extract specific streams for analysis. Useful for finding the VBA project inside an OLE file and pulling out individual macros.
• olevba by Philippe Lagadec, part of the oletools suite. Targets VBA macros directly: extracts them, decodes the common obfuscation tricks (hex, base64, chr-concatenation), and flags suspicious keywords (AutoOpen, Document_Open, Shell, CreateObject, Win32_Process.Create) with severity ratings.

Extracting macros with olevba

#

olevba invoice.docm

Output shows every VBA macro in the document, the deobfuscated strings, and a summary of suspicious behaviors detected. For most commodity phishing, this is enough: you’ll see the PowerShell one-liner that downloads the second stage, the C2 URL it calls, and the dropper logic without ever opening the file in Word.

When deobfuscation isn’t enough: debug it

#

Attackers obfuscate macros to defeat static signature matching:

' What olevba sees
Str = "po" & "wer" & "sh" & "ell.exe"
Shell(Str)

Heavier obfuscation uses character math, environment-variable indirection, and runtime string assembly that olevba can’t fully resolve. The reliable move is to debug the macro itself:

1. Open the document in Word inside an isolated VM (Microsoft Defender, network disconnected, snapshot ready).
2. Alt+F11 opens the VBA editor.
3. Find the entry point (AutoOpen, Document_Open, AutoExec, or a Workbook_Open event).
4. Set a breakpoint at the function entry, press F5 to run.
5. Step through with F8. Watch the Locals window, each variable assignment shows the runtime value, including fully-decoded strings that the static analysis couldn’t unwind.

The technique works because the malware’s deobfuscation routine is doing the work for you; you just need to be watching the variable contents at the right moment. Most macros decrypt their payload string immediately before passing it to Shell or CreateObject, so a breakpoint on those calls catches the final command line.

Network monitoring

#

Detonating a sample on the wire shows you what the malware actually talks to: C2 IPs and domains, exfiltration patterns, secondary payload downloads, kill-switch DNS queries. The standard setup is a VM with a simulated internet (INetSim or FakeNet-NG on a separate host) plus Wireshark capturing on the analysis subnet.

Wireshark essentials for malware traffic

#

The display filter syntax in Wireshark is its own thing; the few patterns worth knowing for malware work:

ip.src == 192.168.x.y || ip.dst == 192.168.x.y     # all traffic to/from the infected host
http.request                                        # HTTP requests only
tls.handshake.extensions_server_name                # SNI values from TLS connections
dns.qry.name                                        # DNS queries
tcp.port == 443 && !tls.handshake                   # encrypted TCP that isn't a TLS handshake

The first move with any capture: pull out every domain the sample resolved (dns filter, then File → Export Specified Packets → CSV). Cross-reference against AlienVault OTX, VirusTotal Passive DNS, or AbuseIPDB. Most commodity malware uses C2 infrastructure that’s been catalogued before.

Case study: Emotet on the wire

#

Emotet is the modular loader that ran from 2014 through Operation Ladybird (Europol plus the Netherlands, Germany, US, UK, France, Lithuania, Canada, and Ukraine) on January 27, 2021. It returned on November 14, 2021 via a TrickBot push and stayed active through early 2023, with sporadic activity since. The C2 protocol changed over its lifetime:

• Pre-March 2019. HTTP GET requests with the encoded payload in the Cookie header.
• Post-March 2019. HTTP POST with the encrypted blob in the request body. The User-Agent is a long forged IE/Trident string designed to blend in with real browser traffic; Emotet definitely never used something like User-Agent: Emotet/1.0, which would have been the dumbest possible choice and would have made the malware self-signing on every IDS in the world.

A representative Emotet POST in the post-2019 era, as it appears in a Wireshark HTTP follow:

POST /<random-path>/ HTTP/1.1
Host: <c2-ip-or-domain>
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 10.0; WOW64; Trident/7.0; ...)
Content-Type: multipart/form-data; boundary=<random>
Content-Length: <variable>

<encrypted blob: AES key wrapped with an RSA public key embedded in the loader>

The encryption is RSA-wrapped AES; the loader carries the RSA public key statically, so you can extract it during unpacking and identify Emotet samples by it. The C2 IP rotates frequently; the standard sources (Feodo Tracker, the Cryptolaemus team’s daily Emotet feed before the takedowns) maintained live IP lists for the duration of the campaign.

Anti-analysis: when the malware notices you watching

#

Anything sophisticated enough to land in a serious incident response has anti-analysis tradecraft baked in. The malware checks whether it’s being debugged, whether it’s in a VM, whether the host looks like a real user environment. If any check fails, the sample exits, sleeps for hours, or pivots to a benign payload to waste the analyst’s time. Recognizing these checks and bypassing them is the difference between a one-day analysis and a one-week analysis.

Anti-debugging

#

• IsDebuggerPresent() and CheckRemoteDebuggerPresent(). Both ultimately read the BeingDebugged flag in the Process Environment Block (PEB). Bypass: set the flag to 0 in the PEB before the call returns. The ScyllaHide plugin for x64dbg automates this and a dozen related checks (NtGlobalFlag, ProcessHeap flags, NtQueryInformationProcess with ProcessDebugPort or ProcessDebugFlags).
• RDTSC timing checks. The malware reads the CPU timestamp counter (RDTSC) before and after a code region. If the elapsed cycle count is much larger than the region’s normal execution time (because you single-stepped through it in a debugger), the sample concludes it’s being analyzed and exits. Bypass: patch the timing check to always pass, or use ScyllaHide’s RDTSC hook.
• Exception-based detection. Throwing an int3 (breakpoint instruction) inside an SEH handler, if a debugger is attached, the debugger catches the int3; if not, the SEH handler does. Inverting the malware’s logic on this catches it.
• Hardware breakpoints. The malware reads the debug registers (DR0-DR7) via GetThreadContext and bails if anything is set. Some packers actively clear them mid-execution to defeat your breakpoints. ScyllaHide can hide these too.

Anti-VM

#

• MAC address prefix checks. VMware’s default OUIs are 00:0C:29 (auto-assigned), 00:50:56 (manual/vCenter), and 00:05:69 (older ESX). VirtualBox uses 08:00:27 (Cadmus/Innotek). Bypass: change the VM’s MAC address to something residential before detonation.
• CPUID hypervisor bit. Bit 31 of ECX in the result of CPUID leaf 1 is the “hypervisor present” indicator. It’s always 0 on bare metal and set by any hypervisor that exposes itself honestly. VMware also exposes a “VMware” string in CPUID leaves 0x40000000 through 0x40000006. Bypass: most hypervisors let you mask the CPUID bit; VMware’s cpuid.1.ecx configuration option toggles it.
• Disk and BIOS strings. Querying the registry for HKLM\HARDWARE\DESCRIPTION\System\BIOS returns vendor strings like “VBoxBIOS” or “Phoenix Technologies LTD” with telltale fields. Querying disk size, RAM, or CPU core counts to detect “small” sandbox configurations is the natural extension.
• Wait for user activity. A sample sleeps until it detects mouse movement, scroll events, or window focus changes (MITRE T1497.002). Automated sandboxes often skip user-input simulation, so a malware that waits 30 minutes for a click never executes its payload during analysis. The canonical example is the “Upclicker” trojan (2012); Ursnif/Gozi and FIN7’s tooling are well-documented modern users.

The general bypass strategy is to identify the check (usually visible as a suspicious call early in execution), patch the result, and continue. ScyllaHide handles the common cases without manual work; custom checks require reading the disassembly and writing the patch yourself.

What this comes down to

#

Dynamic malware analysis is a craft, and the craft is mostly patience and pattern recognition. Every sample is somebody else’s puzzle, deliberately constructed to slow you down. The tools (x64dbg, Scylla, Volatility, olevba, Wireshark, ScyllaHide) are the same handful that every other analyst is using, which means the differentiator isn’t tooling, it’s how quickly you recognize a packer, how fast you find the kill switch, how readily you spot a process hollowed into svchost.exe instead of running where it should.

The case studies in this post (WannaCry’s kill switch, TrickBot’s memory footprint, Emotet’s C2 protocol) are documented examples because they were analyzed publicly by analysts whose work is on GitHub, on blogs, and in conference talks. The skill builds by reading those analyses, then doing your own on the next sample. There’s no shortcut and there’s no LLM that can replace the reading.

References

#

• Practical Malware Analysis (Book)
• Malware Analyst’s Cookbook
• x64dbg Documentation
• Volatility Foundation
• OALabs (Open Analysis)
• ScyllaHide
• PEStudio