Term frequency (TF) counts how often a term occurs in a document. Replacing binary presence values with counts distinguishes documents that contain the same query terms but represent them with different frequency. Higher frequency can indicate a stronger match, although the scoring function determines how that evidence affects ranking. section 3.1.4
Raw counts versus cosine similarity
For the query “apple juice,” doc1 has the vector [1, 1] and doc2 has [3, 4]. Cosine similarity scores them 1.0 and 0.9899, so it favors doc1 because its proportions exactly match the query. A dot product scores them 2 and 7, so it favors doc2, where both terms occur more often. section 3.1.4
Raw counts can overvalue long documents or repeated terms. A normalized TF uses the square root of the term count divided by document length: sqrt(count / document length). The square root dampens additional occurrences, while the denominator makes documents of different lengths more comparable. In a small “the cat in the hat” collection, this normalization changes the ranking to doc2, doc1, then doc3? No: the calculated scores are doc2 1.3842, doc1 1.2456, and doc3 0.6696. section 3.1.4
TF becomes more useful when combined with inverse document frequency (IDF). TF measures how well a term describes a document, while IDF measures how important the term is across the collection. TF-IDF therefore rewards frequent occurrences of informative terms rather than abundant occurrences of common terms. BM25 retains simplified TF-IDF weighting and adds controls for term-frequency saturation and document-length normalization: k controls the saturation point, and b controls normalization strength. section 3.1.6 section 3.2.1
Listing 3.3 Cosine similarity of raw TF vectors
query_vector = [1, 1] #[apple:1, juice:1] doc1_tf_vector = [1, 1] #[apple:1, juice:1] doc2_tf_vector = [3, 4] #[apple:3, juice:4] doc1_score = cosine_similarity(query_vector, doc1_tf_vector) doc2_score = cosine_similarity(query_vector, doc2_tf_vector) print_scores([doc1_score, doc2_score])
Compare the raw TF vectors directly before switching to dot-product scoring; the paired examples make the effect of repeated term occurrences visible. section 3.1.4
