liveBook · Manning

Term frequency

As explained in Build an Advanced RAG Application (From Scratch)

Term frequency is the count or normalized proportion of times a term occurs within a document.

Term frequency (TF) measures how often a term occurs in a document. It can use a raw count or normalize that count by the document's total number of terms, providing the frequency component of TF-IDF keyword ranking. section 3.2 section 3.2.1

TF = (Number of times the term "fox" appears in the document) / (Total number of terms in the document)

The normalized form divides a term's occurrence count by the document length. section 3.2

Counting a term

If "fox" appears twice in an 18-term document, TF is 2 / 18, approximately 0.1111. The raw-count form is simply 2. section 3.2

A small keyword-search implementation uses TF as follows:

  • Tokenize each sentence into words and build an inverted index mapping terms to document identifiers. section 3.2.1
  • Count a term with the document token list's count operation. section 3.2.1
  • Multiply TF by inverse document frequency, then add matching query-term weights to rank documents. section 3.2.1

How Build an Advanced RAG Application (From Scratch) covers it

How deep it goes
Defines raw and normalized frequency, then implements counting inside TF-IDF without developing broader weighting theory.
Examples
Uses a fox-count equation and a six-sentence Python corpus containing terms such as learning and networks.
What it assumes
Uses documents and tokenization as inputs while demonstrating tokenization and inverted-index construction from sentences.
What you can do after
Build a basic TF-IDF keyword ranker by counting query terms in tokenized documents.
How it connects
Connects term frequency to TF-IDF, inverse document frequency, inverted indexes, tokenization, and keyword search.

Also explained in

Compare how 6 books explain term frequency →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build an Advanced RAG Application (From Scratch). The section links point to the parts of the book it draws on.