Inverse document frequency (IDF) measures how much information a term adds by comparing the total corpus size with the number of documents containing that term. Common terms such as “intelligence” appear across many AI books and distinguish little; rarer terms such as “neural network” can identify documents more usefully. section 3.6
Compute and interpret IDF in a small corpus:
- Count the documents containing the term, rather than counting every occurrence. section 3.6
- Multiply term frequency by IDF to obtain a TF–IDF weight, combining within-document prominence with corpus-wide distinctiveness. section 3.0
- Use logarithms to reduce extreme raw differences in large collections; one rare document among a million otherwise produces an outsized score. section 3.6.1
Comparing “bias” and “and”
Two documents from a corpus about algorithmic racial discrimination contain “and” at roughly 0.0292 and 0.0303 term frequency, while “bias” occurs at 0.0167 and 0.0022. Because both terms occur in both documents, their raw IDF is 1; TF–IDF therefore preserves the difference in document-level usage without pretending that either term distinguishes the documents corpus-wide.
Equation 3.5
Inspect the formal TF and IDF equations behind the practical counting examples. section 3.6.1
