Bag-of-words model

As explained in Data Without Labels

Bag-of-words is a text representation that uses word occurrences or frequencies as features while disregarding word order and context.

The bag-of-words model represents each sentence or document as a fixed-length vector of word frequencies. It tokenizes the corpus, builds a vocabulary of unique words, and ignores grammar, word order, and context. Optional frequency thresholds can restrict the vocabulary. section 7.9

A practical construction workflow is:

• Tokenize every document into individual words. section 7.9
• Collect the unique tokens into a corpus vocabulary, optionally applying upper and lower frequency limits. section 7.9
• Encode each document with one vector position per vocabulary word, using zero when a word is absent. section 7.9
• Use the resulting vectors for fast, computationally inexpensive document classification, while recognizing that frequency alone can reduce accuracy. section 7.9 section 7.20

Two-sentence vocabulary

For “It is raining heavily” and “We should eat fruits,” treat the words in both sentences as the vocabulary. The first sentence receives its word counts and zeros for words appearing only in the second sentence. section 7.9

Figure 7.5 The frequency of each word has been calculated. In this example, we have two sentences.

Inspect the token frequencies that precede vector construction. section 7.9

Figure 7.6 We are assuming that in the vocabulary only two sentences are present and the first sentence will be represented as shown.

Inspect how the first sentence becomes a vocabulary-sized vector with zero values for absent words. section 7.9

How Data Without Labels covers it

How deep it goes

Explains tokenization, vocabulary creation, frequency vectors, and thresholds, but stops short of implementation code and contextual modeling.

Examples

Uses two short sentences to demonstrate token frequencies, vocabulary-sized vectors, and zero values for absent words.

What it assumes

Assumes a text corpus and basic tokenization, while defining vocabulary construction and frequency-based vectorization.

What you can do after

Apply the model to convert sentences or documents into frequency vectors for fast document classification.

How it connects

Contrasts frequency-only representation with TF-IDF, n-grams, language models, and contextual word representations.

liveBookmanning.com

Read Data Without Labels on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionBag-of-words is a text representation that records word occurrences while disregarding the order and syntactic relationships of the words.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA text model that ignores token order and represents a document by which words occur in it, often with per-word weights.
• 

  liveBookmanning.com
  Machine Learning System DesignA bag-of-words representation describes text using counts or indicators of its words while disregarding word order.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionA bag-of-words model represents text as an unordered collection of tokens and assigns learned weights to token presence indicators.

Compare how 5 books explain bag-of-words model →

Related concepts

• sequence modelcontrasts with · A sequence model processes ordered data while accounting for dependencies among its positions to produce predictions or representations.
• n-gram modeladdresses · An n-gram model is a probabilistic language model that estimates sequence probabilities using contiguous sequences of a fixed number of tokens, typically to predict a token from preceding context.
• latent Dirichlet allocationused by · Latent Dirichlet allocation is a generative probabilistic topic model representing documents as mixtures of latent topics and topics as distributions over words.
• parameteruses · A model parameter is a numerical value, such as a weight or bias, that is learned or set to determine how a model computes outputs.
• term frequency-inverse document frequencycontrasts with · Term frequency-inverse document frequency is a text-weighting scheme that combines a term’s frequency in a document with its inverse frequency across a document collection.

This explainer was generated by Manning's knowledge graph from the text of Data Without Labels. The section links point to the parts of the book it draws on.