Curse of dimensionality

As explained in How Large Language Models Work

The curse of dimensionality is the phenomenon in which increasing the number of dimensions causes the space, data requirements, and computational costs to grow substantially.

Adding dimensions lets embeddings represent subtler relationships, but it also increases memory, computation, semantic-space size, and training requirements. section 3.2.1

From one number to several

Representing words with one number quickly makes relationships nonsensical. Two or more numbers provide room to place similar meanings near one another, while increasing dimensions makes the space harder and costlier to learn. section 3.2.1

Figure 3.4 Adding another dimension to our token representation allows us to represent a more diverse arrangement of semantic relationships. Here we see how two dimensions can capture relationships for multiple meanings of the same word.

Use the figure to see how an additional dimension accommodates more diverse semantic relationships and multiple meanings. section 3.2.1

How How Large Language Models Work covers it

How deep it goes

Defines the phenomenon and traces its main costs, but stops short of mathematical derivation or quantitative analysis.

Examples

Uses one- and two-number word representations to show why added dimensions improve semantic relationships while expanding the space.

What it assumes

Introduces embeddings, vectors, dimensions, and semantic space while assuming only basic familiarity with token representations.

What you can do after

Evaluate embedding choices by weighing richer relationships against storage, computation, and data requirements.

How it connects

Connects embeddings with vectors, dimensions, semantic space, latent space, and machine learning models.

liveBookmanning.com

Read How Large Language Models Work on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionThe curse of dimensionality is the difficulty of analyzing data as the number of dimensions grows, including sparse representations and increased computational cost.
• 

  liveBookmanning.com
  Advanced Algorithms and Data StructuresThe curse of dimensionality is the set of effects that make data analysis and algorithmic search increasingly difficult as the number of dimensions grows.
• 

  liveBookmanning.com
  Outlier Detection in PythonThe curse of dimensionality is a set of difficulties caused by increasing dimensionality, including data sparsity, less informative distances, and rapidly increasing computational demands.
• 

  liveBookmanning.com
  Data Without LabelsThe curse of dimensionality is the set of difficulties that arise when analyzing data in spaces with many dimensions, including degraded distance behavior, sparsity, increased computation, and overfitting.
• 

  liveBookmanning.com
  Bayesian Optimization in ActionThe curse of dimensionality is the phenomenon in which computational or statistical costs grow exponentially as the number of dimensions increases.
• 

  liveBookmanning.com
  Build a Robo-Advisor with Python (From Scratch)The curse of dimensionality is the rapid growth in computational and data requirements as the number of states, variables, or dimensions in a problem increases.

Compare how 7 books explain curse of dimensionality →

Related concepts

• dimensionality reductionaddressed by · Dimensionality reduction is the transformation of data into a representation with fewer dimensions while retaining selected information, structure, or relationships.
• feature selectionaddressed by · Feature selection is the process of choosing a subset of input features for use in a modeling, analysis, or prediction task.
• distance-based algorithmcontrasts with · A distance-based algorithm analyzes or identifies observations using distances between data points.
• univariate statistical testaddressed by · A univariate statistical test is a statistical procedure that analyzes one variable at a time to evaluate a statistical property, association, or unusualness.
• kernel density estimatecontrasts with · A kernel density estimate is a nonparametric probability-density estimate formed by combining localized kernel functions centered on observed data points.

This explainer was generated by Manning's knowledge graph from the text of How Large Language Models Work. The section links point to the parts of the book it draws on.