Dimensionality reduction transforms high-dimensional embeddings into a compact representation before clustering. It addresses the curse of dimensionality: as dimensions increase, points become sparse and nearest and farthest distances converge, leaving algorithms such as K-Means with little separation to exploit. section 10.3.1
UMAP performs this compression through manifold learning. It builds a graph in the original space and optimizes a lower-dimensional graph that preserves nearby points and broad cluster topology. Its n_neighbors parameter controls the tradeoff: small values emphasize local subtopics, while large values preserve broader themes. section 10.3.1
Use the reduced representation as part of a hierarchical retrieval pipeline:
- Compress 1536-dimensional text-embedding-3-small vectors before clustering. section 10.3.1
- Feed the reduced vectors into soft clustering so text can inform multiple higher-level summaries. section 10.0
- Adjust n_neighbors by level: favor local detail for leaf chunks and broader structure for higher-level themes. section 10.3.1
RAPTOR reduction pipeline
A Python implementation chunks documents into roughly 100-token leaves, embeds them, reduces the embeddings with UMAP, fits a Gaussian mixture model, summarizes each cluster, and repeats the loop until a root summary remains. section 10.4














