liveBook · Manning

Continuous Bag of Words

Also called CBOW

Continuous Bag of Words is a Word2Vec training architecture that predicts a target word from its surrounding context words.

How 2 Manning books explain it

Natural Language Processing in Action, Second Edition

Continuous bag-of-words is a word-embedding training approach that predicts a target word from nearby context words in a sliding window.

How it is used
Trains word embeddings from raw domain text by converting nearby context and target words into prediction examples.
How deep it goes
Explains the prediction task and sliding-window dataset construction, but does not detail optimization or implementation code.
Examples
Uses medical journals, Wikipedia, online news, and sliding-window synthetic documents as training examples.
What you can do after
Build it from scratch by turning a corpus into sliding-window context-target examples for embedding training.

Read how Natural Language Processing in Action, Second Edition explains Continuous Bag of Words →

Retrieval Augmented Generation, The Foundational Ideas

CBOW is a Word2Vec architecture that predicts a center word from multiple surrounding context words.

How it is used
Uses surrounding context to learn dense word representations from raw text.
How deep it goes
Explains context summation, vocabulary-wide scoring, and retained embeddings, but does not detail optimization procedures.
Examples
Uses fill-in-the-blank prediction and a CBOW-versus-Skip-gram architecture diagram.
What you can do after
Evaluate/understand how CBOW converts context-prediction training into reusable word embeddings.

Read how Retrieval Augmented Generation, The Foundational Ideas explains Continuous Bag of Words →

Also covered in

What to know first

Related concepts

Questions about Continuous Bag of Words

What is Continuous Bag of Words?

Continuous Bag of Words is a Word2Vec training architecture that predicts a target word from its surrounding context words.

Which Manning books explain Continuous Bag of Words?

Natural Language Processing in Action, Second Edition and Retrieval Augmented Generation, The Foundational Ideas.

What should I know before learning Continuous Bag of Words?

Manning books that teach Continuous Bag of Words usually introduce accuracy, bag-of-words model, language model and LSTM first.

What is the difference between Continuous Bag of Words and skip-gram?

Continuous Bag of Words is a Word2Vec training architecture that predicts a target word from its surrounding context words. Skip-gram is a word-embedding training method that learns representations by predicting context items from a target item.

What is Continuous Bag of Words a kind of?

Continuous Bag of Words is a kind of language modeling. Language modeling is the task of learning or applying probability distributions over language sequences to estimate or predict units from context.

What is Continuous Bag of Words part of?

Continuous Bag of Words is part of Word2Vec. Word2Vec is a family of shallow neural models that learns vector representations of words from their contexts in text.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.