liveBook · Manning

Continuous Bag of Words

As explained in Retrieval Augmented Generation, The Foundational Ideas

CBOW is a Word2Vec architecture that predicts a center word from multiple surrounding context words.

Continuous Bag of Words (CBOW) is a Word2Vec architecture that predicts a missing center word from multiple surrounding context words. It learns dense word embeddings from raw text through self-supervised prediction. section 2.1.2

CBOW’s architecture separates prediction from the representation it retains:

  • Context-word embeddings are combined by summing them in a projection layer. section 2.1.2
  • A vocabulary-wide output layer scores candidate center words. section 2.1.2
  • After training, the prediction layer is discarded; the projection rows become the word embeddings. section 2.1.2

Fill-in-the-blank prediction

Given several surrounding words with one center position missing, CBOW combines the surrounding representations and predicts which vocabulary word belongs in the blank. section 2.1.2

Figure 2.1 CBOW vs skip-gram architectures.

Compare the CBOW and Skip-gram information flow in the architecture diagram. section 2.1.2

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains context summation, vocabulary-wide scoring, and retained embeddings, but does not detail optimization procedures.
Examples
Uses fill-in-the-blank prediction and a CBOW-versus-Skip-gram architecture diagram.
What you can do after
Evaluate/understand how CBOW converts context-prediction training into reusable word embeddings.
How it connects
Connects Word2Vec with Skip-gram, distributional hypothesis, self-supervised learning, and dense embeddings.

Also explained in

Compare how 2 books explain Continuous Bag of Words →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.