How deeply does Deep Learning with R, Third Edition cover bag-of-words model?

Explains tokenization, set conversion, multi-hot encoding, and learned word weights, but stops short of detailed model architecture.

Bag-of-words model

As explained in Deep Learning with R, Third Edition

A bag-of-words model represents text as an unordered collection of tokens and assigns learned weights to token presence indicators.

A bag-of-words model discards word order, represents a review as an unordered set of tokens, and learns a weight for each token. Token presence can then serve as input for text classification. section 14.3 section 14.4

From a review to a feature vector

For “this movie made me cry,” tokenization produces token IDs such as [0, 1, 3, 0, 5]. Removing order gives {0, 1, 3, 5}; multi-hot encoding turns that set into [1, 1, 0, 1, 0, 1]. The resulting fixed-size vector records which vocabulary items are present, such as “cry,” but not their positions. section 14.4

Use the representation as a simple baseline for text classification:

Assign learned weights to words whose presence suggests a class, such as “terrible” for negative sentiment or “riveting” for positive sentiment. section 14.4
Prefer sequence models when meaning depends on word order; recurrent models and Transformers preserve or inject positional information. section 14.3

"this movie made me cry"

c("[UNK]" = 0, "movie" = 1, "film" = 2, "made" = 3, "laugh" = 4, "cry" = 5)

Inspect the toy sentence and vocabulary used to build the bag-of-words representation. section 14.4

c(1, 1, 0, 1, 0, 1)

Inspect the final multi-hot vector that records token presence and absence. section 14.4

How Deep Learning with R, Third Edition covers it

Cover of Deep Learning with R, Third Edition

liveBook manning.com

Read Deep Learning with R, Third Edition on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Natural Language Processing in Action, Second EditionBag-of-words is a text representation that records word occurrences while disregarding the order and syntactic relationships of the words.
liveBook manning.com

Deep Learning with Python, Third EditionA text model that ignores token order and represents a document by which words occur in it, often with per-word weights.
liveBook manning.com

Data Without LabelsBag-of-words is a text representation that uses word occurrences or frequencies as features while disregarding word order and context.
liveBook manning.com

Machine Learning System DesignA bag-of-words representation describes text using counts or indicators of its words while disregarding word order.

Compare how 5 books explain bag-of-words model →

Related concepts

sequence modelcontrasts with · A sequence model processes ordered data while accounting for dependencies among its positions to produce predictions or representations.
n-gram modeladdresses · An n-gram model is a probabilistic language model that estimates sequence probabilities using contiguous sequences of a fixed number of tokens, typically to predict a token from preceding context.
latent Dirichlet allocationused by · Latent Dirichlet allocation is a generative probabilistic topic model representing documents as mixtures of latent topics and topics as distributions over words.
parameteruses · A model parameter is a numerical value, such as a weight or bias, that is learned or set to determine how a model computes outputs.
term frequency-inverse document frequencycontrasts with · Term frequency-inverse document frequency is a text-weighting scheme that combines a term’s frequency in a document with its inverse frequency across a document collection.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with R, Third Edition. The section links point to the parts of the book it draws on.