Document frequency counts how many documents in a collection contain a specified word or token. It counts documents, not repeated occurrences within one document: a token appearing five times in one document still contributes one document to its frequency. section 3.1
Raw token counts often elevate punctuation and common words such as “of,” “to,” and “and.” Document frequency provides the across-document signal needed to distinguish broadly shared tokens from words that help identify particular documents. section 3.1
Splitting an article into documents
Take a Wikipedia article summary and split it into sentences or paragraphs. Count each token within each smaller document, then record the number of documents in which each token appears. That document-level count can help normalize within-document counts and support more useful bag-of-words representations. section 3.1
Document frequency is one ingredient in term frequency–inverse document frequency (TF–IDF). Combined with how often a token appears in a particular document, it helps assign importance across a corpus and retrieve relevant documents rather than treating every occurrence as equally informative. section 3.0
Document frequency supports several natural-language processing tasks:
- Finding relevant documents by comparing keyword importance within a collection. section 3.0
- Reducing the influence of words that occur broadly across documents when constructing TF–IDF features. section 3.1
- Supporting text classification and topic analysis through numerical representations of word importance. section 3.0
