liveBook · Manning

Dimensionality reduction

As explained in Retrieval Augmented Generation, The Foundational Ideas

Dimensionality reduction is the transformation of high-dimensional data into a lower-dimensional representation while retaining selected structure or information.

Dimensionality reduction transforms high-dimensional embeddings into a compact representation before clustering. It addresses the curse of dimensionality: as dimensions increase, points become sparse and nearest and farthest distances converge, leaving algorithms such as K-Means with little separation to exploit. section 10.3.1

UMAP performs this compression through manifold learning. It builds a graph in the original space and optimizes a lower-dimensional graph that preserves nearby points and broad cluster topology. Its n_neighbors parameter controls the tradeoff: small values emphasize local subtopics, while large values preserve broader themes. section 10.3.1

Use the reduced representation as part of a hierarchical retrieval pipeline:

  • Compress 1536-dimensional text-embedding-3-small vectors before clustering. section 10.3.1
  • Feed the reduced vectors into soft clustering so text can inform multiple higher-level summaries. section 10.0
  • Adjust n_neighbors by level: favor local detail for leaf chunks and broader structure for higher-level themes. section 10.3.1

RAPTOR reduction pipeline

A Python implementation chunks documents into roughly 100-token leaves, embeds them, reduces the embeddings with UMAP, fits a Gaussian mixture model, summarizes each cluster, and repeats the loop until a root summary remains. section 10.4

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains distance failure, graph-based compression, structure preservation, and n_neighbors tradeoffs, then demonstrates implementation.
Examples
Uses text-embedding-3-small vectors, leaf chunks, UMAP, Gaussian mixture models, and recursive cluster summaries.
What it assumes
Assumes familiarity with embeddings and clustering while introducing the dimensionality problem and UMAP mechanism.
What you can do after
Build it from scratch with Python, UMAP, scikit-learn, LangChain, and umap-learn.
How it connects
Connects UMAP and manifold learning to embeddings, clustering, hierarchical retrieval, and summarization.

Also explained in

Compare how 15 books explain dimensionality reduction →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.