liveBook · Manning

Inverse document frequency

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Inverse document frequency is a corpus-based statistic that assigns lower weights to terms occurring in many documents and higher weights to rarer terms.

Inverse document frequency (IDF) weights terms by how unusual they are across a corpus: common terms receive lower weights, while rare terms receive higher weights. It is calculated as log(total documents / documents containing the term). section 3.2.4

In BERTScore, IDF provides optional importance weighting after token embeddings are compared with cosine similarity. Matching “the” contributes little, while matching a distinctive term such as “bureaucracy” contributes more. section 3.2.4

Apply IDF weighting in BERTScore by:

  • Computing the maximum semantic similarity between each reference token and candidate tokens. section 3.2.4

Weighted matching

A reference token “the” with maximum similarity 0.713 and IDF 1.27 produces weighted similarity 0.906; repeating this across tokens yields an importance-weighted BERTScore recall of 0.753.

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines the corpus formula and works through weighted matching, but does not develop IDF beyond BERTScore.
Examples
Uses “the” and “bureaucracy,” then calculates weighted similarity for “the” and an importance-weighted recall score.
What it assumes
Assumes token similarity and embeddings, while defining IDF and its weighting intuition directly.
What you can do after
Apply IDF weighting through BERTScore to emphasize distinctive words when comparing candidate and reference text.
How it connects
Connects corpus statistics with BERTScore, embeddings, cosine similarity, semantic similarity, and greedy matching.

Also explained in

Compare how 6 books explain inverse document frequency →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.