ClusteringEngine groups embedded text for RAPTOR tree construction. It combines Gaussian mixture modeling, Bayesian Information Criterion (BIC) selection of cluster count, and probabilistic assignments rather than requiring an arbitrary K. section 10.4 section 10.4.2
At each level, the engine receives node embeddings and returns cluster IDs for every node. A posterior threshold controls overlap: a chunk can join multiple clusters, while an argmax fallback keeps every leaf assigned when no probability clears the threshold. section 10.4.4 section 10.4.2
The main configuration and reliability choices are:
- Use a low threshold such as 0.1 to favor recall and overlap, or 0.3–0.5 to tighten categorization when summaries bridge unrelated material. section 10.4.2
- Fix the random seed because UMAP optimization and GMM initialization are stochastic and otherwise produce different trees. section 10.4.2
- Pass a covariance regularization floor; without it, the BIC sweep can return one cluster per node and prevent the tree from narrowing. section 10.4.2
Overlapping thematic summaries
On Cinderella leaves, an assignment such as chunk 3 -> clusters [0, 2] lets the same passage inform two parent summaries. An insolvency paragraph can therefore reach both termination and financial-risk summaries, making the resulting structure a directed acyclic graph rather than a strict tree. section 10.4.2
Listing 10.2 Choosing K by Bayesian Information Criterion.
class ClusteringOptimizer:
def __init__(self, max_clusters: int = 50,
random_state: int = 42,
reg_covar: float = 1e-3): #A
self.max_clusters = max_clusters
self.random_state = random_state
self.reg_covar = reg_covar
def get_optimal_k(self, data: np.ndarray) -> int:
"""Determine K by taking the lowest BIC over a sweep."""
max_k = min(self.max_clusters, data.shape[0])
n_range = range(1, max_k + 1)
bics = []
for n in n_range:
gm = GaussianMixture(
n_components=n,
random_state=self.random_state,
n_init=1,
reg_covar=self.reg_covar) #B
gm.fit(data)
bics.append(gm.bic(data))
return n_range[np.argmin(bics)] #CThe BIC implementation shows the bounded sweep over candidate Gaussian-mixture component counts, fixed random state, and covariance regularization. section 10.4.2
