Cyber Threat Hunting - Techniques and Best Practices

Welcome back, fellow hackers and red teamers! In this article, we’ll dive deep into the world of cyber threat hunting. As a follow-up to our previous articles about threat analysis, we’ll be exploring the techniques and best practices you need to know to become a master cyber threat hunter. Are you ready to learn from real-world examples and get your hands dirty with some code samples? Let’s get started!

Introduction

Cyber threat hunting is a proactive approach to identifying and mitigating threats before they can cause damage to an organization’s assets. It’s a crucial part of any robust cybersecurity strategy. As threat hunters, we go beyond traditional security measures such as firewalls, intrusion detection systems (IDS), and antivirus software to actively search for signs of malicious activity that might have slipped through the cracks.

In this article, we will explore the different techniques and methodologies used by expert threat hunters, with a focus on providing actionable advice and real-world examples. By the end of this article, you’ll have a solid understanding of how to hunt for threats, along with the tools and techniques you need to stay ahead of attackers.

The Hunting Process

Before we delve into the techniques and best practices, it’s important to understand the hunting process. The cyber threat hunting process can be divided into four main stages:

1. Hypothesis Generation: Formulating a hypothesis or theory about what an attacker might be doing or attempting to do within the environment.
2. Investigation: Gathering and analyzing data to validate or disprove the hypothesis.
3. Remediation: Taking appropriate action to mitigate identified threats.
4. Improvement: Learning from the hunt and adjusting your tactics to improve future hunting efforts.

These stages are not always linear, and you may find yourself iterating between them as you uncover new information or face challenges.

Hunting Techniques

There are various techniques that can be employed when hunting for cyber threats. In this section, we’ll cover four main categories of hunting techniques:

1. Network-based Hunting
2. Endpoint-based Hunting
3. Log-based Hunting
4. Threat Intelligence-based Hunting

Network-based Hunting

Network-based hunting involves analyzing network traffic to identify anomalies and signs of malicious activity. This can be done using a variety of tools and techniques, including deep packet inspection (DPI), flow analysis, and network behavior analysis.

Deep Packet Inspection (DPI)

DPI involves inspecting the content of network packets to identify patterns that may indicate malicious activity. This can include looking for known signatures of malware or examining the contents of specific packets for indications of command-and-control (C2) communications.

One example of a tool that can be used for DPI is the open-source network intrusion detection system (NIDS) Suricata. Suricata can inspect packets in real-time and apply rules to detect known malicious activity. Here’s an example of a Suricata rule that detects the C2 traffic of the notorious Emotet malware:

alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"ET TROJAN Emotet CnC Checkin"; flow:established,to_server; content:"|17 03 03|"; depth:3; content:"|00 2e|"; offset:3; depth:2; byte_test:2,>,256,1,relative; content:"|00 01 00 00 00|"; within:5; distance:1; byte_test:1,=,5,5,relative; metadata:former_category MALWARE; reference:url,www.volexity.com/blog/?p=62; classtype:trojan-activity; sid:2020178; rev:3; metadata:affected_product Windows_XP_Vista_7_8_10_Server_32_64_Bit, attack_target Client_Endpoint, deployment Perimeter, tag Emotet, signature_severity Major, created_at 2014_11_20, updated_at 2020_01_30;)

Flow Analysis

Flow analysis involves examining the flow of network traffic between different hosts and devices, looking for patterns or anomalies that may indicate malicious activity. NetFlow, a network protocol developed by Cisco, can be used to collect and analyze flow data.

An open-source tool that can be used to process and analyze NetFlow data is SiLK (System for Internet-Level Knowledge). Using SiLK, you can analyze flow data to identify unusual or suspicious traffic patterns, such as connections to known malicious IP addresses, excessive data transfers, or unusual communication patterns.

Here’s an example of using SiLK’s rwfilter command to filter out flows involving a known malicious IP address ( 192.0.2.1):

rwfilter --type=in,inweb --start-date=2020/01/01:00 --end-date=2020/01/31:23 --sip=192.0.2.1 --pass=stdout | rwstats --fields=sip --count=10

Network Behavior Analysis

Network behavior analysis involves monitoring the behavior of network devices and hosts to identify unusual activity that may indicate a compromise. This can include looking for changes in network traffic patterns, unusual login attempts, or other signs of potential compromise.

One example of a network behavior analysis tool is the open-source Bro/Zeek network security monitor. Zeek can analyze network traffic in real-time, generate logs of network events, and detect suspicious activity based on built-in or custom scripts.

Here’s a simple example of a Zeek script that alerts when a host makes more than ten failed SSH login attempts within a five-minute window:

module Notice;

export {
    redef enum Notice::Type += { SSHBruteForce };
}

global ssh_threshold: count = 10;
global ssh_window: interval = 5min;

global ssh_attempts: table[addr] of count &create_expire=ssh_window &expire_func=clear_attempts;

event ssh_attempted_login(c: connection, msg: string)
    {
    local orig_h = c$id$orig_h;
    ssh_attempts[orig_h] += 1;

    if ( ssh_attempts[orig_h] >= ssh_threshold )
        {
        NOTICE([$note=SSHBruteForce, $src=orig_h, $msg=fmt("Possible SSH brute force from %s", orig_h)]);
        ssh_attempts[orig_h] = 0;
        }
    }

function clear_attempts(attempts: table [addr] of count, idx: addr, count: count)
{
    delete ssh_attempts[idx];
}

This script leverages Zeek’s built-in ssh_attempted_login event to keep track of failed SSH login attempts per source IP address. If the number of failed attempts from a specific IP address reaches the defined threshold within the specified time window, a SSHBruteForce notice is generated.

Endpoint-based Hunting

Endpoint-based hunting involves monitoring and analyzing the behavior of individual devices within an organization’s network, such as workstations, servers, and other endpoints. This can include searching for signs of malware, examining system logs, and looking for indicators of compromise (IoCs) on the filesystem or in memory.

Filesystem Analysis

Filesystem analysis involves examining the contents of an endpoint’s filesystem for signs of malicious activity, such as the presence of known malware, unauthorized changes to system files, or suspicious file modifications.

One popular open-source tool for filesystem analysis is The Sleuth Kit (TSK). TSK can be used to analyze disk images, recover deleted files, and search for specific file types or content. Here’s an example of using TSK’s fls command to list the contents of a directory, including deleted files:

fls -r -m / image.dd

Memory Analysis

Memory analysis involves examining the contents of an endpoint’s memory (RAM) for signs of malicious activity, such as running malware processes, injected code, or other IoCs.

Volatility is a powerful open-source memory forensics framework that can be used to analyze memory dumps from various operating systems. For example, you can use Volatility to list the running processes in a memory dump and identify any suspicious or unexpected processes:

volatility -f memory.dmp --profile=Win7SP1x64 pslist

System Log Analysis

System log analysis involves examining the logs generated by an endpoint’s operating system and applications for signs of malicious activity. This can include looking for unusual login attempts, unauthorized changes to system settings, or other indicators of compromise.

For example, you can use the open-source log analysis tool Logwatch to generate a report of log events from a Linux system:

logwatch --detail High --range "between -7 days and -1 days"

Log-based Hunting

Log-based hunting involves analyzing logs generated by various systems, devices, and applications within an organization’s network. This can include network devices (such as firewalls, routers, and switches), security devices ( such as IDS/IPS and antivirus systems), and application logs (such as web server logs and database logs).

Log analysis can be performed using a variety of tools, such as the open-source log management system Graylog. Graylog allows you to collect, store, and analyze logs from various sources, making it easier to identify patterns and anomalies that may indicate malicious activity.

Here’s an example of a Graylog query that searches for failed login attempts to a web application:

source:webserver AND message:"Failed login attempt" AND response:401

Threat Intelligence-based Hunting

Threat intelligence-based hunting involves using information about known threats, such as IoCs, tactics, techniques, and procedures (TTPs), and threat actor profiles, to proactively search for signs of compromise within an organization’s network.

Threat intelligence can be obtained from a variety of sources, such as commercial threat intelligence providers, open-source intelligence (OSINT), and industry-specific information sharing and analysis centers (ISACs). By incorporating threat intelligence into your hunting process, you can better prioritize your efforts and focus on the threats most relevant to your organization.

Here are some examples of how to use threat intelligence in your hunting process:

Indicator of Compromise (IoC) Hunting

IoCs are specific artifacts or patterns that can be used to identify a compromise, such as IP addresses, domain names, file hashes, or registry keys. You can use IoCs to search for signs of compromise within your environment, such as scanning for malicious files, looking for connections to known malicious IPs or domains, or searching for specific registry keys associated with malware.

For example, you can use the open-source IOC Scanner tool to scan a system for known IoCs, such as file hashes or registry keys associated with a specific malware family:

ioc-scanner -i indicators.csv -t file_hash,registry_key -o results.csv

Tactic, Technique, and Procedure (TTP) Hunting

TTPs are the specific methods used by threat actors to carry out their attacks. By understanding the TTPs used by specific threat actors or malware families, you can better focus your hunting efforts on the techniques most likely to be used against your organization.

For example, you can use the MITRE ATT&CK framework to identify the TTPs associated with a specific threat actor or malware family, and then use that information to guide your hunting efforts. You might search for specific command-line arguments used by a known malware, or look for signs of lateral movement techniques commonly used by a specific threat actor.

Threat Actor Profile Hunting

Threat actor profiles are collections of information about specific threat actors or groups, such as their motivations, goals, target industries, and known TTPs. By understanding the threat actors most likely to target your organization, you can prioritize your hunting efforts to focus on the techniques and indicators most relevant to your specific threat landscape.

For example, you might focus your hunting efforts on the TTPs used by an Advanced Persistent Threat (APT) group known to target your industry, or search for IoCs associated with a specific ransomware family that has been targeting organizations similar to yours.

Best Practices

Now that we’ve covered the main techniques used in cyber threat hunting, let’s discuss some best practices to help you get the most out of your hunting efforts:

1. Develop a Hunting Plan: Before starting a hunt, it’s important to have a clear plan that outlines your objectives, the techniques and tools you plan to use, and the scope of your hunting efforts. This will help ensure that your hunting efforts are focused and effective.
2. Leverage Automation: Automating as much of the hunting process as possible can help you save time and resources, allowing you to focus on more complex tasks that require human analysis. This can include using tools and scripts to automatically collect and analyze data, as well as incorporating machine learning and artificial intelligence to help identify anomalies and patterns.
3. Continuously Update Your Knowledge: The threat landscape is constantly evolving, and it’s important to stay up-to-date on the latest threats, TTPs, and threat intelligence. This can include attending conferences, participating in online forums and mailing lists, and regularly reviewing reports and articles from industry experts.
4. Collaborate and Share Information: Collaborating with other threat hunters, both within your organization and across the broader security community, can help you gain valuable insights and stay ahead of emerging threats. This can include participating in industry-specific information sharing and analysis centers (ISACs) or joining online forums and discussion groups dedicated to threat hunting.
5. Learn from Each Hunt: After each hunting engagement, take the time to review your findings, identify lessons learned, and adjust your tactics and techniques accordingly. This will help you continually improve your hunting skills and ensure that your future hunts are even more effective.
6. Maintain a Hunting Toolkit: As a threat hunter, it’s essential to have a collection of tools and resources at your disposal to help you quickly and efficiently hunt for threats. This can include both commercial and open-source tools, as well as custom scripts and utilities that you’ve developed to suit your specific needs.
7. Document Your Findings: Thoroughly documenting your hunting findings can provide valuable insights for future hunts, as well as help to demonstrate the value of your hunting efforts to stakeholders within your organization. Be sure to include information about the techniques and tools used, the results of your analysis, and any recommendations for remediation or further investigation.
8. Implement Threat Hunting Metrics: Measuring the effectiveness of your threat hunting efforts can help you identify areas for improvement and demonstrate the value of your hunting program to stakeholders. Some useful metrics to consider include the number of threats identified, the time taken to detect and remediate threats, and the overall impact of your hunting efforts on your organization’s security posture.

Tools and Frameworks

In this section, we’ll provide a brief overview of some popular tools and frameworks that can help you in your threat hunting efforts:

• Elastic Stack (ELK): Elastic Stack, commonly known as the ELK Stack, is a powerful open-source log management and analytics platform that can be used to collect, store, and analyze logs from various sources. It consists of Elasticsearch (a search and analytics engine), Logstash (a data processing pipeline), and Kibana (a data visualization platform).
• MISP (Malware Information Sharing Platform): MISP is an open-source platform for sharing, storing, and correlating indicators of compromise (IoCs). It can be used to import and export threat intelligence in various formats, such as STIX, OpenIOC, and CSV, and supports integration with other security tools and platforms.
• YARA: YARA is a versatile open-source tool for identifying and classifying malware samples based on textual or binary patterns. It can be used to create custom rules for detecting specific malware families, as well as to scan files, memory dumps, or network traffic for signs of malicious activity.
• ThreatHunting: ThreatHunting is a Splunk app designed to facilitate the proactive hunting of threats by providing pre-built dashboards, visualizations, and analytics based on the MITRE ATT&CK framework. It can be used in conjunction with other Splunk apps and add-ons to analyze logs and data from various sources.
• GRR Rapid Response: GRR Rapid Response is an open-source remote live forensics framework that can be used to perform incident response and threat hunting on remote endpoints. It supports a wide range of live forensic capabilities, such as memory analysis, file system analysis, and registry analysis, as well as the ability to execute custom scripts and tools on remote endpoints.

Real-World Examples

To help illustrate the concepts and techniques discussed in this article, let’s take a look at some real-world examples of successful threat hunting engagements:

1. APT29 Hunting Campaign: In 2016, a major cybersecurity firm launched a proactive hunting campaign targeting the activities of APT29, a sophisticated threat actor believed to be associated with the Russian government. By leveraging threat intelligence and focusing on the TTPs used by the group, the threat hunters were able to identify previously unknown instances of compromise within their clients’ networks, leading to the discovery of several previously unknown malware samples and command-and-control infrastructure.
2. Dridex Banking Trojan Hunt: In 2015, a multinational cybersecurity company conducted a threat hunting engagement focused on the Dridex banking Trojan, which was responsible for stealing millions of dollars from individuals and businesses worldwide. By analyzing the Trojan’s infrastructure and TTPs, the threat hunters were able to identify several new command-and-control servers, as well as uncover a previously unknown version of the malware.
3. ShadowBrokers Leak Hunting: In 2017, a group calling themselves the ShadowBrokers leaked a large collection of exploits and tools believed to be associated with the Equation Group, a sophisticated cyber-espionage group with suspected ties to the NSA. In response, cybersecurity firms and researchers around the world launched threat hunting campaigns to identify and mitigate any potential compromises resulting from the leaked tools. These efforts led to the discovery of several new vulnerabilities and the development of patches and mitigations to protect against the leaked exploits.
4. WannaCry Ransomware Hunting: In May 2017, the WannaCry ransomware attack caused widespread disruption, affecting hundreds of thousands of systems worldwide. Following the initial outbreak, threat hunters from various organizations worked tirelessly to analyze the ransomware, identify its propagation methods, and develop mitigations to prevent further infections. Their efforts led to the discovery of a “kill switch” that effectively halted the spread of the ransomware, as well as the development of several decryption tools to help victims recover their encrypted files.
5. NotPetya Ransomware Hunting: In June 2017, the NotPetya ransomware attack caused significant disruption to businesses and organizations worldwide, particularly in Ukraine. Following the attack, threat hunters from various organizations collaborated to analyze the ransomware, identify its methods of propagation, and develop mitigations to protect against future infections. Their efforts led to the discovery of several previously unknown vulnerabilities and the development of patches and mitigations to protect against the ransomware’s propagation methods.

These real-world examples demonstrate the value of proactive threat hunting in identifying and mitigating emerging threats before they can cause widespread damage. By leveraging the techniques and best practices discussed in this article, you can improve your organization’s security posture and stay one step ahead of the ever-evolving cyber threat landscape.

Conclusion

In this article, we have explored the concept of cyber threat hunting, its importance in today’s cybersecurity landscape, various techniques used in threat hunting, best practices, and some real-world examples. By implementing a proactive threat hunting program and leveraging the techniques and best practices discussed in this article, you can better protect your organization from emerging threats and improve your overall security posture.

Remember, as a threat hunter, it is crucial to stay up-to-date on the latest threats, tactics, and techniques, and continuously hone your skills to stay ahead of the adversaries. Happy hunting!