Malware analysis for red teamers

Red Team operators who can reverse-engineer their own tools end up with better tradecraft than the ones who can’t. If you compile a payload, throw it at a target, and it gets blocked, the question is why, and “it got caught” doesn’t help anyone. “It got caught because the Rich Header was stripped and Defender’s heuristics flag stripped-header binaries from non-Microsoft signers” does, because now you have something to change for the next attempt.

This post is about the self-audit loop. Run the same triage on your payload that a defender’s automation runs on it five minutes after it lands. Most of the time on the offensive side you don’t get to see what the test grader will look at. Here you do. Use it.

Static analysis: what the file looks like at rest

#

Static analysis examines the file without executing it. AV and EDR do it first, so you should too.

PE header and metadata

#

The Portable Executable header tells the OS loader how to map the binary. It also tells an analyst a fair amount about who built it.

• TimeDateStamp in 1992 or 2099? Anomalous timestamps get flagged. A lot of malware authors zero this field; that’s also anomalous.
• The Rich Header is XOR-encrypted compiler and linker metadata (Visual Studio version, build IDs). Attackers often strip it, but absence of a Rich Header on a binary that otherwise looks like an MSVC build is itself suspicious; it’s the dog that didn’t bark.
• Standard sections are .text (code, RX), .data (variables, RW), .rsrc (resources, R). Anything with RWX is almost always either a packer’s unpacking stub or runtime-allocated shellcode storage. Either way it lights up.

Imports and the IAT

#

The Import Address Table is the list of OS API functions your binary needs at load time. It’s also a short description of what the binary intends to do, and EDR engines read it that way.

• A small “Notepad” clone has no business importing VirtualAllocEx, WriteProcessMemory, and CreateRemoteThread. That’s the canonical cross-process injection trio (MITRE T1055.002) and it’s the first thing any static engine looks for.
• A complex binary with zero imports is usually packed, or it’s a stage 1 loader that resolves everything at runtime. Either pattern is suspicious by itself.

Strings

#

The strings utility does not lie. If your payload contains plain-text IPs (192.168.1.5), URLs (http://evil.com/beacon), or error messages (Error injecting shellcode), an analyst has the C2 endpoint and a rough capability summary in under a minute. Static signature engines have the same information even faster.

PDB paths are the most common self-own. The linker happily embeds C:\Users\xX_Hacker_Xx\Projects\BestMalware\Release\beacon.pdb into your release build by default. Strip them, or use linker /PDBALTPATH: to write something neutral.

Entropy

#

Shannon entropy on a byte sequence runs 0 to 8. Standard compiled code reads roughly 4.0–6.0. Packed or encrypted regions read 7.0–8.0. EDRs treat high-entropy sections as “I can’t see inside, so I’ll assume the worst.” UPX-packing a binary in the hope of evading static signatures is, in 2026, a guarantee of detection: the stock UPX stub has a well-known signature, and the high-entropy section adds a second flag on top of the first.

Tools worth having installed:

• PEStudio . Initial triage. Flags suspicious imports, sections, and indicators automatically.
• capa . Mandiant FLARE’s capability detection (e.g., “contains code to parse HTTP,” “matches process-injection pattern”). Mandiant is part of Google Cloud now but the repo stayed at mandiant/capa.
• Strings . Sysinternals classic.

Dynamic analysis: what the file does when it runs

#

Dynamic analysis runs the sample in an instrumented environment and watches behavior. This is what sandboxes do, and it’s what an EDR is doing in real time on the endpoint.

Telemetry sources

#

A 2018-era picture of EDR was “they hook ntdll in userland and watch your API calls.” That’s still partly true, but it’s no longer the main signal. Modern commercial EDR (CrowdStrike Falcon, SentinelOne, Defender for Endpoint) leans heavily on:

• Kernel callbacks. PsSetCreateProcessNotifyRoutineEx, PsSetCreateThreadNotifyRoutine, ObRegisterCallbacks, and the minifilter framework give the EDR a ringside view of process creation, handle access, and file I/O from a place you can’t unhook from userland.
• ETW (Event Tracing for Windows). The Threat Intelligence ETW provider in particular surfaces things like NtAllocateVirtualMemory calls with PAGE_EXECUTE_READWRITE, which is what your shellcode injection looks like.
• Userland inline hooks. Still present in some products, but increasingly the weakest of the three. Unhooking ntdll is well-documented and isn’t the trick it was five years ago.

If your evasion plan is “I bypass the userland hooks,” your plan handles the least important telemetry source. The kernel callbacks and ETW providers do not care.

Network indicators

#

Sandboxes (and EDR network telemetry) look for:

• DNS lookups to DGA-looking or recently registered domains.
• HTTP/HTTPS beaconing with regular intervals. Jitter helps; perfectly periodic beacons are a giveaway.
• Traffic on uncommon ports like 4444, 8080, or 8443 from hosts that have no business serving them.

Persistence

#

The same registry, service, and scheduled-task locations have been on every IR playbook for fifteen years. Sandboxes check them automatically:

• Run keys (HKCU\Software\Microsoft\Windows\CurrentVersion\Run and the HKLM equivalent).
• Newly created services.
• Newly created scheduled tasks, especially under \Microsoft\Windows\ where they try to blend in.

Tools for dynamic analysis:

• Process Monitor . Real-time file, registry, process, and network activity. Indispensable.
• Wireshark . Packet capture for the network side.
• x64dbg . When you need to single-step through the assembly because the high-level analysis isn’t enough.

The self-audit loop

#

For a Red Team payload, the development cycle that catches the obvious problems looks like this:

1. Develop. Write the tool or payload.
2. Static audit. Run PEStudio and capa. Check the IAT. Check entropy of every section. Strip PDB paths. Encrypt or runtime-resolve any string that names an IP, domain, or capability.
3. Dynamic audit. Detonate in a local VM with ProcMon and Wireshark running. Compare what the process actually does against what you intended it to do. Unexpected file writes, unexpected child processes, and unexpected outbound connections are all things you’d rather find here than in a customer’s SIEM.
4. EDR test. Run against a lab instance of whatever the target environment runs. The license cost of one EDR seat is much smaller than the cost of burning an operation.
5. Deploy. Only now, on an engagement, on a target.

Most operators skip step 2. PEStudio takes a minute to install and a few hours to actually get useful with, and a few hours is hard to spend when there’s a payload sitting on disk ready to send. Send it later. The defender’s automation will run step 2 on your binary whether you bothered to or not, and they’ll have an opinion about the result.

Closing thought

#

Malware analysis is reverse engineering pointed at malicious code. For an operator, getting comfortable with it is mostly a way to stop being surprised by detections you could have predicted. Pretty much anything on GitHub with “C2” or “loader” in the readme is already on the major vendors’ signature lists. The vendors watch GitHub the same way you do, sometimes a little more carefully. Running one of those tools straight out of the box means a defender has likely seen the exact bytes before, knows what they do, and has a rule ready. Audit your own work before that becomes your problem.