liveBook · Manning

Curse of dimensionality

As explained in Build AI Drug Discovery Pipelines

The curse of dimensionality is the phenomenon in which statistical estimation, geometric coverage, and related computational tasks become increasingly difficult as the number of dimensions grows.

The curse of dimensionality describes how statistical estimation, geometric coverage, and related computational tasks become harder as the number of dimensions grows. In chemical machine learning, a fixed number of molecules occupies high-dimensional descriptor space sparsely, weakening the meaning of distance and similarity. section 6.2.2 section 7.1

The difficulty has several practical consequences:

  • Distance-based methods such as Kennard–Stone selection and clustering require pairwise distances, increasing computational cost with the number of samples and dimensions. section 6.2.2
  • Data become increasingly sparse because the number of points needed to cover the space at a given density scales exponentially with dimensionality. section 6.2.3
  • Sparse coverage can make distances less informative, while dimensions beyond three become difficult to visualize directly. section 6.2.2

A practical response is to exploit the blessing of non-uniformity: not every dimension is equally important. Principal component analysis compresses descriptors into orthogonal components ordered by explained variance, retaining a smaller representation while preserving important structure. Representative splitting can then select diverse training molecules that better align train and test distributions. section 6.5 section 6.2.3

RNA-target ligand modeling

A dataset containing 48 ligands represented by 191 descriptors sparsely populates its feature space. Kennard–Stone selection chooses samples that are maximally distant from those already selected, seeking a diverse subset that captures dataset variability and broadens the QSAR model’s applicability domain. section 6.2.2 section 6.2.3

Figure 6.8 Dimension x2 carries almost no information since the x2-values for each point barely deviate relative to the values along dimension x1. Conversely, the x1 dimension carries a lot of information and removing it would result in a compressed form of the data that is very dissimilar from the original data.

Use the figure to see how PCA preserves an informative direction while removing a dimension with little variation. section 6.2.3

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains sparsity, distance degradation, computational scaling, and PCA-based mitigation, while stopping short of a full statistical derivation.
Examples
Uses 191-dimensional ligand descriptors, PCA projections, Kennard–Stone selection, and high-throughput screening data for drug repurposing.
What it assumes
Assumes basic descriptors, distances, QSAR, and datasets, while defining sparsity, dimensionality reduction, and PCA in context.
What you can do after
Evaluate high-dimensional chemical representations and apply PCA or representative sampling to improve visualization and dataset coverage.
How it connects
Connects dimensionality reduction with PCA, clustering, similarity methods, QSAR, drug repurposing, and representative data splitting.

Also explained in

Compare how 10 books explain curse of dimensionality →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.