Reference dataset

As explained in Outlier Detection in Python

A reference dataset is a collection of observations used as a basis for comparing other observations or assessing what is typical.

A reference dataset is a collection of observations used as a basis for comparing new observations or assessing what is typical. Its usefulness depends on whether it represents the relevant normal state: older data may be obsolete, while recent data may itself be atypical. section 4.14

Choose the reference data according to the monitoring situation:

• Use an established reference set to detect data drift, support frequent arrivals, handle streaming data, or compensate for slowly arriving observations. section 4.14
• Prefer verified high-quality periods when historical data is known to contain no defects, such as a stable industrial process. section 4.14
• Compare against multiple reference points when no single period reliably represents the expected behavior; use a rolling window when conditions change over time. section 4.14
• Balance reference-set size against computation: larger sets can improve distance-based prediction but make methods such as LOF and KNN slower. section 10.2.1

In subspace outlier detection, a reference set is the neighborhood of k points used to identify the features that explain the greatest variance around each observation. Shared nearest neighbors can make this comparison more robust in high-dimensional data with irrelevant features. section 6.10

A high-dimensional reference set

A synthetic dataset contains 35 features, with correlated features 8 and 9 and one inserted outlier. SOD uses a reference set of three neighbors and flags the point because its relationship between those features is unusual, although its other features are typical. section 6.10

Figure 6.14 The left pane shows point A in a 2D dataspace. The point is unusual considering feature A but not if using Euclidean distances in the full 2D dataspace. Using additional features can be counterproductive. In the middle and right panes, we see another point, point B, which is an outlier in the C–D subspace but not in the E–F subspace. We need only features C and D to identify this outlier.

Shows why a reference set or selected subspace can reveal an outlier that appears ordinary when all features and Euclidean distance are considered together. section 6.10

How Outlier Detection in Python covers it

How deep it goes

Explains selection trade-offs, temporal representativeness, quality checks, neighborhood construction, and prediction cost without resolving a universal choice.

Examples

Uses click histories, a defect-free industrial period, rolling monitoring data, and a 35-feature synthetic SOD dataset.

What it assumes

Assumes familiarity with outlier detection, distance-based methods, neighborhoods, and feature spaces.

What you can do after

Evaluate and configure reference data for anomaly comparisons while balancing representativeness, quality, detection reliability, and prediction cost.

How it connects

Connects reference datasets to outlier detection, data drift, streaming data, rolling windows, KNN, LOF, and subspace outlier detection.

liveBookmanning.com

Read Outlier Detection in Python on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Machine Learning Platform EngineeringA reference dataset is a feature dataset containing the features associated with users across the training and testing populations.

Compare how 2 books explain reference dataset →

Related concepts

• Evidently AIused by · Evidently AI is an open-source Python tool for evaluating, testing, monitoring, and reporting on machine-learning data, models, predictions, and behavior.
• shared neighbors distanceproduced by · Shared neighbors distance measures dissimilarity between observations according to the neighbors they have in common.
• local outlier factorused by · Local Outlier Factor is a density-based anomaly-detection algorithm that scores an observation by comparing its local density with those of its neighbors.
• k-nearest neighborsused by · K-nearest neighbors is a method that predicts a query’s class or value from the labels or values of its k closest training examples, as measured by a specified distance or similarity.
• data driftused by · Data drift is a change over time in the statistical distribution or characteristics of data relative to a reference distribution or set of characteristics.
• subspace outlier detectionused by · Subspace outlier detection is an outlier detection method that identifies anomalies within feature subspaces selected from local reference sets.

This explainer was generated by Manning's knowledge graph from the text of Outlier Detection in Python. The section links point to the parts of the book it draws on.