liveBook · Manning

Curse of dimensionality

As explained in LLM Evaluation and Alignment, The Foundational Ideas

The curse of dimensionality is the set of difficulties that arise as the number of dimensions increases, including sparse data and less-discriminatory distances.

The curse of dimensionality describes why increasing dimensions makes data sparse and distances less discriminatory: many points become far apart, yet their distances become increasingly similar. section 3.2.3

In embedding spaces, direction can preserve relationships better than magnitude. Cosine similarity therefore compares vector angles instead of Euclidean distance or vector length. section 3.2.3

Semantic similarity despite different wording

“The weather is freezing” and “it is cold today” share no lexical overlap, while “the attorney filed the brief” and “the lawyer submitted the document” can occupy nearby embedding regions. These contrasts motivate semantic metrics and angular comparison.

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains sparsity, distance concentration, and the shift toward angular comparison, without developing higher-dimensional mathematics.
Examples
Contrasts paraphrased sentences with no lexical overlap and vectors that differ in magnitude but share direction.
What you can do after
Evaluate/understand embedding similarity by prioritizing vector orientation when high-dimensional distances lose discrimination.
How it connects
Connects embeddings with Euclidean distance, cosine similarity, lexical metrics, and semantic metrics.

Also explained in

Compare how 10 books explain curse of dimensionality →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.