Continuous Bag of Words
Continuous Bag of Words is a Word2Vec training architecture that predicts a target word from its surrounding context words.
Natural Language Processing in Action, Second Edition
Continuous bag-of-words is a word-embedding training approach that predicts a target word from nearby context words in a sliding window.
- How it is used
- Trains word embeddings from raw domain text by converting nearby context and target words into prediction examples.
- How deep it goes
- Explains the prediction task and sliding-window dataset construction, but does not detail optimization or implementation code.
- Examples
- Uses medical journals, Wikipedia, online news, and sliding-window synthetic documents as training examples.
- What you can do after
- Build it from scratch by turning a corpus into sliding-window context-target examples for embedding training.
Read how Natural Language Processing in Action, Second Edition explains Continuous Bag of Words →
Retrieval Augmented Generation, The Foundational Ideas
CBOW is a Word2Vec architecture that predicts a center word from multiple surrounding context words.
- How it is used
- Uses surrounding context to learn dense word representations from raw text.
- How deep it goes
- Explains context summation, vocabulary-wide scoring, and retained embeddings, but does not detail optimization procedures.
- Examples
- Uses fill-in-the-blank prediction and a CBOW-versus-Skip-gram architecture diagram.
- What you can do after
- Evaluate/understand how CBOW converts context-prediction training into reusable word embeddings.
Read how Retrieval Augmented Generation, The Foundational Ideas explains Continuous Bag of Words →
Also covered in
- LLMs in Production
- Deep Learning with Python, Third Edition
- Deep Learning with R, Third Edition
- Data Without Labels
- Machines that ThinkIn early access (MEAP)
What to know first
- accuracyAccuracy is the degree to which a result agrees with a specified truth, reference, or criterion, commonly measured as the proportion of evaluated predictions that are correct.
- bag-of-words modelA bag-of-words model represents text using word or token occurrences, counts, or weights while disregarding their order and syntactic relationships.
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- LSTMA long short-term memory (LSTM) network is a recurrent neural-network architecture with gated memory cells that control information flow and model dependencies in sequential data.
Related concepts
- skip-gramcontrasts with · Skip-gram is a word-embedding training method that learns representations by predicting context items from a target item.
- Word2Vecpart of · Word2Vec is a family of shallow neural models that learns vector representations of words from their contexts in text.
- embeddingproduces · An embedding is a numerical vector representation of an item in a feature space, encoding selected properties or relationships for computational tasks such as comparison or retrieval.
- context windowuses · A context window is a bounded span of sequence elements or contextual information that a model or processing operation can consider together.
- language modelingkind of · Language modeling is the task of learning or applying probability distributions over language sequences to estimate or predict units from context.
- language modelkind of · A language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
Questions about Continuous Bag of Words
What is Continuous Bag of Words?
Continuous Bag of Words is a Word2Vec training architecture that predicts a target word from its surrounding context words.
Which Manning books explain Continuous Bag of Words?
Natural Language Processing in Action, Second Edition and Retrieval Augmented Generation, The Foundational Ideas.
What should I know before learning Continuous Bag of Words?
Manning books that teach Continuous Bag of Words usually introduce accuracy, bag-of-words model, language model and LSTM first.
What is the difference between Continuous Bag of Words and skip-gram?
Continuous Bag of Words is a Word2Vec training architecture that predicts a target word from its surrounding context words. Skip-gram is a word-embedding training method that learns representations by predicting context items from a target item.
What is Continuous Bag of Words a kind of?
Continuous Bag of Words is a kind of language modeling. Language modeling is the task of learning or applying probability distributions over language sequences to estimate or predict units from context.
What is Continuous Bag of Words part of?
Continuous Bag of Words is part of Word2Vec. Word2Vec is a family of shallow neural models that learns vector representations of words from their contexts in text.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

