Document length normalization adjusts term-frequency relevance scores so longer documents are not favored merely because they contain more words. A common calculation divides dampened term frequency by document length, making scores more comparable across documents. section 3.1.4
A normalized term-frequency score typically:
- Counts occurrences of a query term in the document. section 3.1.4
- Dampens repeated occurrences with a square root or logarithm, so each additional match contributes less. section 3.1.4
A short query and three documents
For the query “the cat in the hat,” raw term counts rank a verbose document first because it repeats “the” and “in” frequently. Using tf(term, doc) = sqrt(term_count / document_length) changes the scores: doc1 = 1.2456, doc2 = 1.3842, and doc3 = 0.6696. The better matching doc2 therefore moves above doc1, although term importance still requires a separate signal such as inverse document frequency.
BM25 generalizes the idea with k for term-frequency saturation and b for normalization strength. Its term-frequency component uses document length relative to average document length; increasing b increases the normalization effect. Lucene-based search systems use related document-length norms, and BM25 exposes these controls directly. section 3.2.1
Listing 3.3 Cosine similarity of raw TF vectors
query_vector = [1, 1] #[apple:1, juice:1] doc1_tf_vector = [1, 1] #[apple:1, juice:1] doc2_tf_vector = [3, 4] #[apple:3, juice:4] doc1_score = cosine_similarity(query_vector, doc1_tf_vector) doc2_score = cosine_similarity(query_vector, doc2_tf_vector) print_scores([doc1_score, doc2_score])
Compare raw term vectors with cosine similarity before applying frequency and length normalization. section 3.1.4
