Inverse document frequency (IDF) weights query terms by how rarely they appear across the document collection. Rare terms such as “hat” and “cat” contribute more to relevance than common terms such as “the” and “in”. section 3.1.5
The weighting process follows three steps:
- Count document frequency: measure how many indexed documents contain the term. section 3.1.5
- Combine IDF with term frequency (TF) to balance how important a term is with how strongly it describes a document. section 3.1.6
The cat in the hat
With 10,000 documents, the example assigns IDF values of 1.0513 to “the”, 1.1053 to “in”, 5.5953 to “cat”, and 6.2786 to “hat”. A TF-IDF calculation therefore ranks doc2 highest because it matches the most important terms most strongly, followed by doc3 and doc1.
