TF-IDF weights terms for text-based relevance ranking. Term frequency (TF) measures how well a term describes a document, while inverse document frequency (IDF) measures how important that term is across the collection. Combining them gives keyword matches different weights instead of treating every occurrence as equally informative. section 3.1.6
A search pipeline can represent the query and each document as numerical vectors, then compare those vectors to produce a relevance score. In the worked calculation, the query “the cat in the hat” becomes repeated term entries, each document receives TF-IDF values for those terms, and the scores are combined with a dot product. section 3.1.6 section 3.4
Ranking “the cat in the hat”
Three documents produce TF-IDF relevance scores of 1.4215 for doc1, 18.9633 for doc2, and 14.2099 for doc3. Doc2 ranks first because it matches the important terms most strongly; doc3 contains all the words but less often, while doc1 contains mostly insignificant words. A Python implementation can tokenize the query, create a vector of ones, calculate each document’s TF-IDF vector, and take the dot product. section 3.1.6
Listing 3.8 Calculating TF-IDF for the query the cat in the hat
def tf_idf(term, doc): return TF(term, doc) * IDF(term)**2 query = "the cat in the hat" terms = tokenize(query) query_vector = list(numpy.repeat(1, len(terms))) doc_vectors = [[tf_idf(doc, term) for term in terms] for doc in docs] doc_scores = [[dot(query_vector, dv)] for dv in doc_vectors] print_tf_idf_scores(terms, doc_vectors, doc_scores)
Use the executable listing to inspect the TF-IDF function, query vector, document vectors, and score calculation. section 3.1.6
TF-IDF is a foundation for keyword-based search, but many engines use BM25, which retains TF-IDF at its core and adds term-frequency saturation and document-length normalization. These controls often improve ranking over standard TF-IDF cosine similarity. section 3.2.1 section 3.4
