liveBook · Manning

BIC

As explained in Retrieval Augmented Generation, The Foundational Ideas

The Bayesian Information Criterion is a model-selection metric that balances likelihood-based goodness of fit against a penalty for model complexity.

BIC selects among statistical models by balancing likelihood-based fit against model complexity. In clustering, minimizing BIC favors enough Gaussian mixture components to capture structure without adding unsupported detail. section 10.3.3

BIC = p ln(n) − 2 ln(L)

The criterion combines the maximized likelihood L, parameter count p, and sample count n: BIC = p ln(n) − 2 ln(L). Lower values indicate the preferred model. section 10.3.3

Use BIC to choose the number of clusters when labels are unavailable:

  • Sweep candidate values of K, fit a Gaussian Mixture Model for each value, and choose the K with the lowest BIC. section 10.4.2
  • Prefer simpler models unless an added cluster improves likelihood enough to offset its additional parameters. section 10.3.3
  • Bound the sweep and regularize covariance estimates; otherwise a component can claim one data point, shrink covariance toward zero, and drive likelihood upward without bound. section 10.3.3 section 10.4.2

Choosing clusters for a retrieval tree

For ten points arranged in three separated blobs, an unrestricted sweep can make BIC fall from 110.8 at K=1 to −57.6 at K=10. A bounded, regularized sweep instead supports adaptive cluster counts for diverse or focused corpora. section 10.3.3 section 10.4.2

Listing 10.2 Choosing K by Bayesian Information Criterion.
class ClusteringOptimizer:
    def __init__(self, max_clusters: int = 50,
                 random_state: int = 42,
                 reg_covar: float = 1e-3):  #A
        self.max_clusters = max_clusters
        self.random_state = random_state
        self.reg_covar = reg_covar
    def get_optimal_k(self, data: np.ndarray) -> int:
        """Determine K by taking the lowest BIC over a sweep."""
        max_k = min(self.max_clusters, data.shape[0])
        n_range = range(1, max_k + 1)
        bics = []
        for n in n_range:
            gm = GaussianMixture(
                n_components=n,
                random_state=self.random_state,
                n_init=1,
                reg_covar=self.reg_covar)  #B
            gm.fit(data)
            bics.append(gm.bic(data))
        return n_range[np.argmin(bics)]  #C

The implementation caps K at the number of observations, fits GaussianMixture models across the candidate range, and returns the index of the minimum BIC. section 10.4.2

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Defines the criterion, explains its fit-complexity terms, demonstrates a degeneracy, and implements a bounded Gaussian-mixture sweep.
Examples
Uses ten points in three blobs, retrieval-tree clustering, and assignments where one chunk belongs to multiple clusters.
What it assumes
Assumes embedding vectors and unsupervised clustering, while defining likelihood, parameters, and complexity penalties.
What you can do after
Apply it via GaussianMixture to automate K selection while constraining candidate counts and covariance regularization.
How it connects
Connects model selection with likelihood, Gaussian mixture models, soft clustering, embeddings, and dimensionality reduction.

Also explained in

Compare how 5 books explain BIC →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.