Unsupervised learning

As explained in Cyber Threat Hunting

Unsupervised machine learning is a machine-learning paradigm that discovers patterns and structures in data without predefined labels or specific instructions.

Unsupervised learning analyzes unlabeled data to find hidden structure: similar records can be grouped, and unusual records can stand apart, without being told which records are benign or malicious. In threat hunting, it shifts the starting point from a fixed indicator or threshold to patterns present in the data. section 8.1 section 8.1.8

Use it after understanding and preparing the data, not as a replacement for that work. Extract useful features from raw events, encode non-numeric values appropriately, inspect correlations, and remove highly correlated fields before applying an algorithm. Data preprocessing can determine whether resulting clusters are meaningful. section 8.1.2 section 8.1.8 section 8.8

A practical unsupervised threat-hunting workflow is:

• Collect connection events from a data store or file, retaining fields that may describe behavior. section 8.1.2
• Calculate behavioral features such as connection timing, counts, variance, and inbound-versus-outbound byte differences. section 8.1.2
• Apply a clustering algorithm such as k-means to group data points, then examine small or unusual clusters for anomalies. section 8.0 section 8.3.2
• Investigate cluster members, excluding known normal traffic carefully rather than treating cluster assignment as a verdict. section 8.3.2 section 8.8

Finding jittered command-and-control traffic

A hunt can start with a hypothesis that compromised internal hosts beacon to a command-and-control server with an unknown interval, jitter level, port, or destination. After k-means assigns connection records to four clusters, inspecting two small clusters can expose malicious beaconing without setting a standard-deviation threshold first. Clustering remains imperfect: malicious connections may span multiple clusters. section 8.1.8 section 8.3.2

Figure 8.1 Threat hunting with unsupervised machine process

Follow the end-to-end threat-hunting process from unlabeled event data through preparation, modeling, and anomaly investigation. section 8.1

Treat outputs as leads, not proof. Algorithms have different rules and tunable parameters; compare approaches when needed, seek specialist help for deeper mathematical questions, and use findings to help create labeled data for later supervised learning. section 8.8

How Cyber Threat Hunting covers it

How deep it goes

Defines unlabeled pattern discovery and walks through preprocessing, feature selection, k-means clustering, and cluster investigation while deferring deeper mathematics.

Examples

Uses network connection events and jittered command-and-control beaconing, including clusters containing suspicious internal-to-external connections.

What it assumes

Assumes threat-hunting familiarity but introduces core ML concepts and uses existing libraries rather than requiring new algorithm code.

What you can do after

Apply it via existing Python libraries to prepare connection data, cluster it, and investigate anomalous groups as threat-hunting leads.

How it connects

Connects feature engineering, one-hot encoding, correlation analysis, k-means, clustering, anomaly detection, and supervised learning.

liveBookmanning.com

Read Cyber Threat Hunting on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Data Without LabelsAn unsupervised deep learning algorithm trains a multilayer neural model to discover structure or representations without target labels.
• 

  liveBookmanning.com
  Grokking Machine LearningA machine-learning approach in which a model discovers structure or patterns from data without target outputs.
• 

  liveBookmanning.com
  Financial AI in PracticeUnsupervised learning is a machine-learning approach that identifies structure or patterns in data without relying on labeled outcomes.
• 

  liveBookmanning.com
  Optimization Algorithms: AI techniques for design, planning, and control problemsUnsupervised learning is a machine-learning paradigm in which a model discovers structure or patterns in data without labeled target outputs.
• 

  liveBookmanning.com
  Inside AIUnsupervised learning is a machine-learning technique in which an algorithm analyzes unlabeled data to discover patterns or groupings.
• 

  liveBookmanning.com
  Statistics Slam DunkUnsupervised learning is a machine-learning approach that discovers structure or patterns in data without labeled target values.
• 

  liveBookmanning.com
  Investing for ProgrammersUnsupervised learning is a machine-learning approach that identifies structure or patterns in data without predefined output labels.
• 

  liveBookmanning.com
  Outlier Detection in PythonMachine learning that extracts structure, information, or patterns from data without labeled target values.
• 

  liveBookmanning.com
  The AI Pocket BookA machine learning approach that seeks patterns in unlabeled data without a target output.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthUnsupervised learning is a machine learning paradigm that learns structure or representations from data without labeled target outputs.
• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionUnsupervised learning is a machine-learning approach that derives patterns or representations directly from data without human-provided target labels.
• 

  liveBookmanning.com
  Grokking AI ApplicationsUnsupervised learning is a machine learning approach that discovers patterns or structure in data without labeled examples.
• 

  liveBookmanning.com
  Math and Architectures of Deep LearningA learning setup that uses training inputs without manually provided target outputs.
• 

  liveBookmanning.com
  Acing the CCNA Exam: Volume 2 Advanced Networking and SecurityUnsupervised learning is a machine-learning approach that discovers patterns or structure in data without predefined labels.
• 

  liveBookmanning.com
  Introduction to Generative AI, Second EditionUnsupervised learning is machine learning in which training data are provided without labels.
• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA learning setting in which models are trained without human-provided labels, often by exploiting structure in the data.
• 

  liveBookmanning.com
  The Art of AI Product DevelopmentUnsupervised learning is machine learning in which a model learns patterns from data without explicitly provided target labels.
• 

  liveBookmanning.com
  Advanced Algorithms and Data StructuresUnsupervised learning is machine learning that analyzes unlabeled data to discover structure or patterns without predefined categories.
• 

  liveBookmanning.com
  Software Testing with Generative AIA machine-learning approach that trains a model on unlabeled data to discover implicit patterns or structure.

Compare how 20 books explain unsupervised learning →

Related concepts

• supervised learningcontrasts with · Supervised learning is a machine-learning approach that learns to map inputs to target outputs from examples paired with known labels or values.
• K-means clusteringincludes · K-means clustering partitions data into a specified number of groups by assigning observations to representative centers and iteratively updating those centers to reduce within-group distances.
• density estimatorincludes · A density estimator is a statistical method or model that estimates the probability density underlying observed data.
• hierarchical clusteringincludes · Hierarchical clustering is a clustering method that organizes observations into nested groups by successively merging or splitting clusters.
• machine learningkind of · Machine learning is a field of computational methods in which systems learn patterns or relationships from data to make predictions, decisions, or other outputs.
• clusteringincludes · Clustering is the technique of grouping data items according to similarity or shared characteristics.

This explainer was generated by Manning's knowledge graph from the text of Cyber Threat Hunting. The section links point to the parts of the book it draws on.