Anomaly

As explained in Cyber Threat Hunting

An outlier is an observation that lies unusually far from the central or typical range of a data set.

An outlier is an observation unusually far from a data set’s central or typical range. In threat hunting, outliers can represent noise, skew, or activity worth investigating. section 7.1.2 section 7.0

The interquartile range (IQR) identifies the middle portion of an ordered distribution between the lower and upper quartiles. Removing values outside that range reduces noise before calculating variance and standard deviation, especially when data is skewed or nonnormal. section 7.1.2

A practical outlier-handling workflow is:

• Group network events by source IP, destination IP, and destination port. section 7.1.2
• Calculate lower and upper quartiles for the timing values. section 7.1.2
• Drop rows whose time differences fall outside the IQR. section 7.1.2
• Calculate standard deviation, variance, and connection counts on the remaining data. section 7.1.2
• Remove duplicate connection groups and exclude known benign destinations before reviewing low-variance candidates. section 7.1.2

Filtering connection timing outliers

A pandas workflow drops rows below the lower quartile or above the upper quartile of time_diff_sec, then retains connection groups with low standard deviation. The resulting candidates include repeated connections to 162.125.2.14:443 and 40.87.160.0:23456. section 7.1.2

Figure 7.4 Quartiles for a nonnormal distribution

Use the quartile diagram to compare the middle range with values that lie unusually far from it in a nonnormal distribution. section 7.1.2

How Cyber Threat Hunting covers it

How deep it goes

Explains quartiles, skew, and IQR filtering with pandas listings, then stops short of fully interpreting each candidate connection.

Examples

Demonstrates outlier filtering on network timing data grouped by source IP, destination IP, and destination port.

What it assumes

Assumes familiarity with standard deviation and variance while teaching quartiles, medians, skew, and IQR from basic definitions.

What you can do after

Diagnose noisy network timing data with pandas and produce lower-variance connection candidates for threat investigation.

How it connects

Connects outliers with quartiles, interquartile range, standard deviation, variance, skewed distributions, beaconing, and anomaly detection.

liveBookmanning.com

Read Cyber Threat Hunting on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Outlier Detection in PythonAn extreme value is an observation far from the typical level of a time series, either unusually small or unusually large.
• 

  liveBookmanning.com
  Statistics Slam DunkAn outlier is an observation that lies unusually far from the other observations in a distribution or dataset.
• 

  liveBookmanning.com
  The Well-Grounded Data AnalystAn outlier is an observation whose value differs substantially from the typical pattern of the other observations.
• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIAn extreme value is an observation or value located unusually far toward an edge of an observed distribution.
• 

  liveBookmanning.com
  Data Without LabelsIn DBSCAN, an outlier or noise point is an observation that is neither a core point nor reachable from a core point and therefore receives no cluster assignment.
• 

  liveBookmanning.com
  Starting Data Analytics with Generative AI and PythonAn outlier is an observation that is unusually distant from the other values in a dataset or from an expected pattern.
• 

  liveBookmanning.com
  Pandas Workout: 200 exercises to make you a stronger data analystAn outlier is an observation unusually far below or above the central range of a dataset according to a chosen criterion.
• 

  liveBookmanning.com
  Experimentation for EngineersAn observed data point that lies unusually far from the other observations in a dataset or fitted relationship.
• 

  liveBookmanning.com
  Statistics Every Programmer NeedsAn outlier is an observation that is unusually distant from the pattern represented by the rest of a dataset.
• 

  liveBookmanning.com
  Data-Driven CybersecurityAn outlier is an observation that lies unusually far from the expected distribution or reference pattern.
• 

  liveBookmanning.com
  The Art of AI Product DevelopmentAn outlier is a data point whose behavior or value is anomalous relative to the dominant patterns in a dataset.
• 

  liveBookmanning.com
  Financial AI in PracticeAn outlier is a data value that is unusually distant from other observations in a dataset or distribution.
• 

  liveBookmanning.com
  Grokking StatisticsAn outlier is an observation that differs markedly from most other observations in a dataset.

Compare how 14 books explain anomaly →

Related concepts

• out-of-distributionincludes · Out-of-distribution describes data or inputs whose generating distribution differs materially from the distribution represented in a model’s training data.
• local outlierincludes · A local outlier is a data item that differs from other items in its nearby or comparable cluster while belonging to a common overall type.
• winsorizationaddressed by · Winsorization is a data transformation that replaces extreme observations beyond selected thresholds with less-extreme boundary values rather than removing them.
• interquartile rangeuses · The interquartile range is the difference between the third and first quartiles of a distribution, measuring the spread of its middle half.
• box-and-whisker plotused by · A box-and-whisker plot is a statistical graphic that summarizes a numerical distribution using quartiles, a median, and whiskers or markers indicating spread and potential outliers.
• contextual anomalyincludes · A contextual anomaly is an observation that is anomalous relative to its surrounding temporal, situational, or other contextual conditions.

This explainer was generated by Manning's knowledge graph from the text of Cyber Threat Hunting. The section links point to the parts of the book it draws on.