Continuous Bag of Words

As explained in Natural Language Processing in Action, Second Edition

Continuous bag-of-words is a word-embedding training approach that predicts a target word from nearby context words in a sliding window.

Continuous bag-of-words (CBOW) trains word embeddings by predicting a target word from nearby context words. Unlike document-level bag-of-words, it uses a continuously sliding window, producing many small training examples from each document. section 6.3.2

A CBOW training set is built by:

• Taking raw domain text as the corpus. section 6.3.2
• Moving a window through each document to form many tiny synthetic documents. section 6.3.2
• Using nearby words as inputs and the central target word as output. section 6.3.2

Choosing a corpus

A medical application can learn from medical journals, while a general-purpose embedding can use Wikipedia or online news. Each corpus shapes the word representations available to the model. section 6.3.2

Figure 6.4 CBOW neural network architecture

Inspect the CBOW neural-network architecture alongside the sliding-window prediction process. section 6.3.2

How Natural Language Processing in Action, Second Edition covers it

How deep it goes

Explains the prediction task and sliding-window dataset construction, but does not detail optimization or implementation code.

Examples

Uses medical journals, Wikipedia, online news, and sliding-window synthetic documents as training examples.

What it assumes

Assumes familiarity with documents and words while distinguishing document-level bag-of-words from sliding-window CBOW vectors.

What you can do after

Build it from scratch by turning a corpus into sliding-window context-target examples for embedding training.

How it connects

Connects word embeddings with bag-of-words and contrasts CBOW with skip-gram.

liveBookmanning.com

Read Natural Language Processing in Action, Second Edition on liveBook Get the book at manning.com

Related concepts

• Word2Vecpart of · Word2Vec is a family of shallow neural models that learns vector representations of words from their contexts in text.
• embeddingproduces · A numerical vector representation of an item or input that encodes features or relationships useful for computational tasks.
• context windowuses · A context window is the bounded sequence of tokens or other input information a model can access when processing or generating an output.
• language modelingkind of · Language modeling is the task of learning or applying statistical structure or probabilities over language sequences to predict or generate language units from context.
• language modelkind of · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
• skip-gramcontrasts with · Skip-gram is a word-embedding training architecture that learns to predict surrounding context items from a target item.

This explainer was generated by Manning's knowledge graph from the text of Natural Language Processing in Action, Second Edition. The section links point to the parts of the book it draws on.