Sometimes you’re the villain, and sometimes you’re the one who has to catch him. The same operator who spends Monday building a phishing pretext spends Tuesday reading the alerts it would have tripped. Blue team work is red team work run backwards: you already know how the intrusion happens, so you build the exercise to see whether anyone would actually notice.
That’s the useful frame for the two disciplines this post covers, threat hunting and incident response. Everything below is the defender’s side of techniques you already use offensively.
What blue team exercises are#
A blue team exercise simulates a real intrusion to test whether your detection and response hold up. You run the attack, or someone runs it against you, and you watch to see what fires, what gets missed, and how long it takes anyone to react. The point is to find the gaps before an actual attacker does.
The work splits into two halves. Threat hunting is the proactive half: going looking for compromise that no alert caught, hunting for indicators of compromise (IOCs) and attacker behavior already sitting in your network. Incident response is the reactive half: once something is found, containing it and cleaning up with minimum damage.
Why they’re worth running#
Running exercises tells you three things a compliance checkbox never will. You learn where your posture is actually weak, not where the audit says it should be strong. You get to rehearse the incident response plan while nobody’s career is on the line, which is the only time you’ll find out that the on-call runbook points at a Slack channel that was archived last year. And over time the habit builds a team that goes looking for trouble instead of waiting for a vendor alert to tell them they’ve had it for six months.
What three real breaches taught defenders#
The best exercises are grounded in how real intrusions actually played out. Three are worth studying because each one failed at a different point in the detection-and-response chain.
Target, 2013#
Attackers stole payment card data and personal records for up to 110 million Target customers over the 2013 holiday season. The initial access had nothing to do with Target’s payment systems. The attackers phished credentials from Fazio Mechanical, a third-party HVAC vendor with network access to Target’s environment, pivoted inward, and pushed memory-scraping malware onto point-of-sale terminals to grab card data as it was swiped.
The part worth sitting with: Target’s tooling worked. FireEye sensors flagged the malware, and the alerts went to a security team that didn’t act on them. Detection without response is just an expensive log. The breach became public when the data showed up for sale and a journalist connected it back to Target, not because anyone inside caught it in time.
Equifax, 2017#
Equifax exposed Social Security numbers, birth dates, and other records for 143 million U.S. consumers, a figure later revised to about 147.9 million. The root cause was an unpatched Apache Struts vulnerability, CVE-2017-5638, in a public-facing web application. Apache had shipped the patch on March 7, 2017, and the attackers walked in through the hole it was meant to close about two months later.
This one is a patch-management story wearing a breach’s clothing. The vulnerability was known, the fix existed, and the window between disclosure and exploitation was measured in weeks. Attackers then sat in the environment for months, partly because an expired certificate had blinded the traffic-inspection tooling that might have caught the exfiltration.
NotPetya, 2017#
NotPetya is filed under ransomware, but it was a wiper wearing a ransom note. It encrypted machines with no working recovery path, so paying got you nothing. It spread through a poisoned software update for M.E.Doc, a Ukrainian accounting package, then tore laterally through networks using EternalBlue and stolen credentials.
The blast radius is the lesson. A payload aimed at Ukraine took down Maersk’s global shipping operation, Merck, and FedEx’s TNT Express subsidiary, with total damage estimated near ten billion dollars. Flat networks and shared local admin credentials turned a regional attack into a worldwide one. Segmentation and credential hygiene, the boring controls, were the difference between a bad day and a company-wide outage.
Threat hunting techniques#
Threat hunting is the assumption that something already got in and nothing alerted on it. You go looking anyway. A few approaches carry most of the weight.
Behavioral analysis#
Behavioral analysis watches how users and systems normally act, then flags the deviations. A login at 3 a.m. from an account that only ever works nine-to-five, a workstation suddenly talking to a host in a country it’s never touched, a service account pulling gigabytes it has no reason to pull. None of these are signatures. They’re patterns that only look wrong against a baseline of normal.
The strength is that it catches the unknown. Signature detection needs to have seen the threat before; behavioral analysis just needs the behavior to stand out. Beaconing malware that phones a C2 server on a regular interval will surface as an odd traffic rhythm even if no product has a rule for that specific family yet.
Two tools do the heavy lifting here. User and Entity Behavior Analytics (UEBA) applies machine learning to activity across accounts and devices, scoring how far each action sits from the norm, and it’s particularly good at insider threats where the credentials are legitimate but the behavior isn’t. Log analysis is the low-tech cousin: system and application logs already record login times, access patterns, and connections, and a lot of intrusions are visible in them if anyone reads them. EDR, network traffic analysis, and SIEM feed behavioral hunting too, and each gets its own treatment below.
Endpoint detection and response#
Endpoints are where attackers land and where they do most of their work, which makes them the richest place to hunt. EDR agents sit on laptops, desktops, and servers collecting process trees, network connections, and file and registry activity, then look for trouble in that stream.
They find it several ways. Signature matching catches known-bad by hash or pattern. Behavioral analysis on the endpoint baselines normal process and application activity and flags the deviations, which is how you catch a Word process spawning PowerShell spawning a network connection. Machine learning generalizes from large corpora of known malware to catch variants no signature covers. And threat intelligence feeds keep the agent current on the IOCs and techniques seen elsewhere in the wild.
Detection is only half the product. Once an endpoint trips, EDR can isolate the box from the network to stop lateral movement, quarantine the offending file or process, block traffic to known-bad infrastructure, and preserve forensic data (process history, memory, logs) so you can reconstruct what happened. That containment-plus-evidence combination is why EDR sits at the center of most modern SOC workflows.
Network traffic analysis#
Where EDR watches the host, network traffic analysis watches the wire. It inspects packets and flows for the things that don’t fit: unusual transfer volumes, odd ports, protocol behavior that doesn’t match what the protocol is supposed to do.
The detection methods mirror the ones above. Signature matching flags traffic that matches known attack patterns. Anomaly detection compares current traffic against a baseline per segment and surfaces the deviations. Protocol analysis digs into how a protocol is behaving to catch abuse like scanning or malformed-request attacks. Flow analysis looks at who’s talking to whom, from which address to which, in what volume, which is where data exfiltration and internal reconnaissance tend to show themselves.
When something lands, the response options are network-shaped: block traffic to malicious hosts, throttle or cut a compromised device’s access to slow the spread, and capture packets and flow records for the investigation.
Threat intelligence#
Threat intelligence is context. It’s the difference between “an unknown IP hit our web server” and “a known Cobalt Strike C2 node associated with a group currently targeting our sector hit our web server.” Gathered and applied well, it turns raw alerts into decisions.
It comes in a few flavors. Open source intelligence (OSINT) pulls from public reporting, vendor writeups, and social media. Technical intelligence works from artifacts: file hashes, C2 domains, malware behavior. Human intelligence comes from people, insight into who’s operating and why. Cyber threat intelligence (CTI) is the security-specific blend of these, focused on malware campaigns, zero-days, and APT activity.
You consume it through the same tooling you use for everything else. SIEM platforms correlate intel feeds against your own logs so a known-bad indicator lights up the moment it appears. Threat intelligence platforms (TIPs) aggregate feeds from multiple sources and push indicators out to your detection stack automatically. Malware analysis tools take a suspicious sample and tell you what family it belongs to and what to hunt for next.
Vulnerability management#
Hunting for attacker activity is half the job; the other half is closing the doors before anyone uses them. Vulnerability management is the loop of finding weaknesses, ranking them by how much damage they’d do, and fixing them in that order.
Finding them takes a few forms. Vulnerability scanners sweep systems automatically for known issues in operating systems, applications, and network gear. Penetration testing goes deeper on the things scanners miss, chaining findings the way an attacker would. Red team engagements go deeper still, running a full simulated adversary against people, process, and technology at once.
Then you triage. Anything that leads to data theft or system compromise on a reachable, public-facing asset jumps the queue. Equifax is the cautionary tale here: the vulnerability was known and the patch existed, and none of that mattered because it wasn’t applied in time. Remediation is usually patch management (apply the vendor fix), hardening (secure configuration, kill unnecessary services, fix weak credentials), or segmentation (wall systems off so a compromise in one place doesn’t spread to the rest).
Incident response#
Threat hunting finds the intrusion. Incident response is what you do next, and doing it well under pressure is a rehearsed skill, not an instinct. Hunting doesn’t stop when the incident starts, either: the same techniques run again during response to scope the damage and find anything the first pass missed. The rest of the work moves through a predictable loop.
flowchart LR A[Triage] --> B[Containment] B --> C[Analysis] C --> D[Reporting] C -.->|new leads| A
Triage#
Triage is the first thirty minutes: figure out what you’re actually looking at and how badly it matters. An alert comes in, from a control, a user report, or a partner tip-off. You do an initial read of logs and traffic to size the scope. Then you prioritize, because a compromised domain controller and a single phished mailbox are not the same emergency and can’t wait in the same queue.
SIEM platforms drive most of this by pulling alerts into one place and, increasingly, kicking off automated first steps. Incident response platforms give the process structure and a shared workspace, and threat intelligence platforms add the context that tells you whether the indicator you’re staring at is background noise or the leading edge of something worse.
Containment#
Containment stops the bleeding before you understand the whole wound. First, isolate: disable the compromised accounts, pull affected hosts off the network, block the traffic tied to the incident. Then identify the source and nature of what you’re dealing with from logs and captured traffic, so containment doesn’t turn into blind whack-a-mole. Remediate the root cause, whether that’s a patch, a config change, or a policy fix. And communicate throughout, because an incident that IT, security, leadership, and affected customers learn about at different times through different channels is an incident you’re now managing twice.
The tooling is what you already have. Firewalls and network segmentation isolate the affected systems, EDR isolates and inspects individual endpoints, and an incident response platform keeps the whole effort coordinated instead of scattered across five people’s terminals.
Analysis#
Analysis is the reconstruction: how did they get in, what did they touch, what did they take. You collect the evidence first, logs, packet captures, and disk and memory artifacts from every system in scope. You analyze it to pin down the root cause and the full set of IOCs. You rebuild the timeline, tying together which systems and accounts were involved and in what order. And you cross-reference against threat intelligence to see whether the indicators match a known group or campaign, which often tells you what else to go looking for.
Reporting#
Reporting turns the incident into something the organization can act on. Document what happened accurately: the timeline, the systems and data affected, and what you did about it. Identify and brief the stakeholders who need to know, which for a breach touching customer data now includes legal, regulators, and sometimes the customers themselves. Keep the communication clear enough that non-technical readers can follow it. And write the final report so the details survive past the adrenaline of the incident, because the next team to hit something similar will read it.
Tools you’ll actually run#
Plenty of tools claim a place in blue team work. A handful earn it. These four cover most open-source detection and response, and the commercial section covers what you buy when open source runs out of room.
The ELK stack#
ELK, now usually called the Elastic Stack, is Elasticsearch, Logstash, and Kibana. Logstash collects and normalizes log data from wherever it lives, Elasticsearch stores and searches it at volume, and Kibana visualizes it. Together they give you a place to put every log you generate and a fast way to ask questions of it.
Elasticsearch is a distributed search and analytics engine that stores data as JSON documents in indexes and searches them in near real time; it scales horizontally, which is what lets it swallow the log volume a real network produces. Logstash is the pipeline that feeds it, using input and filter plugins to pull from syslog, files, and application logs and clean the data into a consistent shape on the way in. Kibana is the front end, with dashboards, alerting, and a query interface that supports full-text, boolean, and regex search.
For a defender that means three things in one stack: security monitoring through dashboards and alerts on suspicious events, threat hunting through fast search across all your logs at once, and incident response through the log history you need to reconstruct what happened.
Zeek#
Zeek, formerly Bro, is a network security monitor that watches traffic passively from a tap or SPAN port and turns it into structured logs. It doesn’t try to be an inline blocker. It’s a framework for understanding what’s happening on your network, and it’s very good at that.
Zeek parses protocols rather than just matching bytes. It pulls metadata out of TCP, UDP, ICMP, HTTP, DNS, TLS, and more, logging connections, file transfers, certificate details, and DNS queries in a form you can correlate against endpoint and SIEM data. That protocol awareness is the whole point: Zeek tells you a host made a TLS connection with a self-signed certificate to a domain registered yesterday, not just that packets moved.
On top of the protocol logs, Zeek runs its own event-driven scripting language, a domain-specific language built for network analysis rather than a general-purpose one bolted on. You write policy scripts that fire on connection events, detect the behavior you care about, and generate alerts or new logs. That scriptability is why Zeek shows up in so many SOCs: you can teach it to recognize things no vendor rule covers, and wire its output into whatever else you run.
Sysmon#
Sysmon is Microsoft’s Sysinternals system monitor for Windows. It installs a driver and service that log activity the default Windows event log doesn’t capture in useful detail, and it’s the closest thing to free EDR-grade endpoint telemetry you can deploy.
It logs the events that matter for hunting: process creation with full command lines and hashes, network connections,
file creation, and registry changes. The process-creation logging is the standout, because it records the parent
process too. That’s how you catch process-injection and living-off-the-land tradecraft, the winword.exe that spawned
powershell.exe that opened a network connection, which is the signature of a phishing payload detonating.
Sysmon runs off an XML configuration file, and the config is where the tool lives or dies. A good config (the SwiftOnSecurity and Olaf Hartong baselines are the common starting points) filters out the noise and keeps the signal, and you deploy it by group policy so every box logs the same way. Feed that stream into a SIEM and you have real endpoint visibility across the fleet for the cost of some tuning.
VirusTotal#
VirusTotal is the fast first look at a suspicious file or URL. Upload it, or search for its hash, and you get results from dozens of antivirus engines plus behavioral analysis in one report.
For a defender it does two jobs. As malware triage, it tells you whether a sample is known-bad, what family it likely belongs to, and what it does when it runs, with hashes, file details, and detonation behavior in the report. As a hunting resource, it lets you pivot on an indicator, searching hashes, domains, and IP addresses to find related samples and infrastructure. The API is the part that makes it operational: wire it into your SIEM or EDR and indicators get enriched automatically instead of one analyst pasting hashes into a web form.
One caution worth stating: a public VirusTotal upload is public. Don’t submit files that might contain sensitive internal data, because you’re handing a copy to a third party and everyone who queries the hash later.
Commercial tools#
Open source gets you a long way, but at scale you usually end up paying for correlation, coverage, and support. Four categories matter.
SIEM platforms centralize and correlate security data from across the environment and alert on it, and the enterprise tiers add response automation on top. Splunk Enterprise Security, IBM QRadar, and Elastic Security are the common names. EDR platforms are the commercial version of the endpoint monitoring above, with cloud-backed detection and response; CrowdStrike Falcon, Carbon Black Cloud, and Microsoft Defender for Endpoint (formerly Defender ATP) lead here. Network security monitoring platforms watch traffic at enterprise scale with heavier analytics than Zeek alone, including Darktrace, Cisco Secure Network Analytics (formerly Stealthwatch), and RSA NetWitness. And cloud security tools cover what the on-prem stack can’t see, monitoring AWS, Azure, and GCP for misconfiguration and intrusion; Microsoft Defender for Cloud, AWS Security Hub, and Google Security Command Center are the native options.
Buy for the gap you actually have, not the logo. The categories overlap, and a well-tuned open-source stack beats an un-tuned expensive one every time.
The part tools can’t fix#
No single tool catches everything, and the three breaches above each broke at a different link in the chain. Target’s sensors caught the malware and nobody acted on the alert. Equifax left a known patch unapplied for two months. NotPetya only spread worldwide because the networks it hit were flat. The tooling in this post handles detection, but an alert nobody reads, a patch nobody deploys, or a segment nobody builds will sink you regardless of what you spent on the SIEM.
The exercises are the point. Run the attack against yourself, watch what fires and what doesn’t, close the gap, and run it again. You already know how the break-in works. Blue team work is pointing that knowledge at your own door and checking whether anyone’s watching it.
