liveBook · Manning

ClusteringEngine

As explained in Retrieval Augmented Generation, The Foundational Ideas

ClusteringEngine is the companion notebook component that combines dimensionality reduction, cluster selection, and probabilistic clustering for RAPTOR tree construction.

ClusteringEngine groups embedded text for RAPTOR tree construction. It combines Gaussian mixture modeling, Bayesian Information Criterion (BIC) selection of cluster count, and probabilistic assignments rather than requiring an arbitrary K. section 10.4 section 10.4.2

At each level, the engine receives node embeddings and returns cluster IDs for every node. A posterior threshold controls overlap: a chunk can join multiple clusters, while an argmax fallback keeps every leaf assigned when no probability clears the threshold. section 10.4.4 section 10.4.2

The main configuration and reliability choices are:

  • Use a low threshold such as 0.1 to favor recall and overlap, or 0.3–0.5 to tighten categorization when summaries bridge unrelated material. section 10.4.2
  • Fix the random seed because UMAP optimization and GMM initialization are stochastic and otherwise produce different trees. section 10.4.2
  • Pass a covariance regularization floor; without it, the BIC sweep can return one cluster per node and prevent the tree from narrowing. section 10.4.2

Overlapping thematic summaries

On Cinderella leaves, an assignment such as chunk 3 -> clusters [0, 2] lets the same passage inform two parent summaries. An insolvency paragraph can therefore reach both termination and financial-risk summaries, making the resulting structure a directed acyclic graph rather than a strict tree. section 10.4.2

Listing 10.2 Choosing K by Bayesian Information Criterion.
class ClusteringOptimizer:
    def __init__(self, max_clusters: int = 50,
                 random_state: int = 42,
                 reg_covar: float = 1e-3):  #A
        self.max_clusters = max_clusters
        self.random_state = random_state
        self.reg_covar = reg_covar
    def get_optimal_k(self, data: np.ndarray) -> int:
        """Determine K by taking the lowest BIC over a sweep."""
        max_k = min(self.max_clusters, data.shape[0])
        n_range = range(1, max_k + 1)
        bics = []
        for n in n_range:
            gm = GaussianMixture(
                n_components=n,
                random_state=self.random_state,
                n_init=1,
                reg_covar=self.reg_covar)  #B
            gm.fit(data)
            bics.append(gm.bic(data))
        return n_range[np.argmin(bics)]  #C

The BIC implementation shows the bounded sweep over candidate Gaussian-mixture component counts, fixed random state, and covariance regularization. section 10.4.2

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains GMM assignments, BIC selection, thresholds, seeds, and regularization, but defers dimensionality reduction and summarization internals.
Examples
Uses Cinderella leaves, overlapping insolvency assignments, printed cluster IDs, and a BIC implementation listing.
What it assumes
Assumes embedded text vectors and introduces clustering as one stage alongside dimensionality reduction and summarization.
What you can do after
Operate/configure a clustering stage with posterior thresholds, reproducible seeds, BIC selection, and covariance regularization.
How it connects
Connects Gaussian mixture models, Bayesian Information Criterion, UMAP, embeddings, soft clustering, and RAPTOR tree construction.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.