Term frequency
Term frequency is the raw or normalized count of a term’s occurrences within a document or other text unit.
Natural Language Processing in Action, Second Edition
Normalized term frequency is term frequency divided by the number of terms in the document.
- How it is used
- Measures a term’s share of a document for bag-of-words representations and document comparison.
- How deep it goes
- Works through Python token counts and length normalization, then extends the calculation toward inverse document frequency and logarithmic scaling.
- Examples
- Compares “justice” with common tokens in a sentence and computes “bias” frequencies in two documents about algorithmic bias.
- What you can do after
- Build it from scratch with token counts and document lengths, then compare normalized frequencies across documents.
Read how Natural Language Processing in Action, Second Edition explains term frequency →
Elasticsearch in Action, Second Edition
Term frequency is the count or normalized frequency of a term's occurrences within a document or field.
- How it is used
- Supports Elasticsearch relevance ranking by measuring how often a query term occurs in a matching document or field.
- How deep it goes
- Defines TF, compares TF-IDF with BM25, and shows score inspection while deferring custom similarity algorithms.
- Examples
- Uses Java book titles and an explained movie-title match query for “Lord.”
- What you can do after
- Diagnose/debug relevance scores with explain or _explain by inspecting term-frequency and related scoring details.
Read how Elasticsearch in Action, Second Edition explains term frequency →
AI-Powered Search
Term frequency is the count or within-document frequency of a term's occurrences, used to measure how strongly a document matches that term.
- How it is used
- Supports keyword relevance ranking by estimating how strongly a document matches a query term.
- How deep it goes
- Explains raw counts, cosine and dot-product effects, normalized TF, TF-IDF, and BM25 controls without deriving a complete search engine.
- Examples
- Uses apple-juice vectors and a three-document cat-in-the-hat collection to compare counts, normalized TF, TF-IDF, and BM25 rankings.
- What you can do after
- Apply it via Python to build term vectors, calculate normalized TF, and compare ranking behavior with dot product or BM25.
AI Agents in Action, Second Edition
Term frequency is the frequency of a term in a document, commonly normalized by the document's total number of words.
- How it is used
- Supports a simple starting point for document vectorization and vector similarity search before semantic retrieval.
- How deep it goes
- Defines frequency and places it within TF-IDF, but defers detailed formula derivation and semantic representation internals.
- Examples
- Uses a sentence about a blue sky and a collection of sentences about skies, animals, and breakfast foods.
- What you can do after
- Apply it via scikit-learn’s TfidfVectorizer to convert documents into sparse vectors for similarity comparison.
Read how AI Agents in Action, Second Edition explains term frequency →
Just Use Postgres!
Term frequency is the number of occurrences of a term in a document or text representation.
- How it is used
- Ranks PostgreSQL full-text search results for a movie recommendation service.
- How deep it goes
- Explains occurrence counts and connects them to ts_rank, positions, density, weights, and document-length normalization.
- Examples
- Compares 'ghost' occurrences in two movie lexeme vectors and ranks matching movies with ts_rank.
- What you can do after
- Apply it via PostgreSQL by inspecting lexemes and calculating search rankings with ts_rank.
Related concepts
- term frequency-inverse document frequencypart of · Term frequency-inverse document frequency is a text-weighting scheme that combines a term’s frequency in a document with its inverse frequency across a document collection.
- relevanceuses · Relevance is the degree to which information, a result, or a response pertains to a query, task, purpose, or information need.
- document length normalizationconfigured by · Document length normalization adjusts retrieval relevance scores to account for differences in document length or term frequency.
- token countmeasures · A token count is a measure of the number of tokens present in text or processed or generated during a model or corpus operation.
- BM25uses · BM25 is a probabilistic information-retrieval ranking function that scores documents for a query using term frequency, inverse document frequency, and document-length normalization.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.