liveBook · Manning

Skip-gram

Skip-gram is a word-embedding training method that learns representations by predicting context items from a target item.

How 2 Manning books explain it

Natural Language Processing in Action, Second Edition

Skip-gram is a word-embedding training approach that predicts surrounding or skipped words from a context word.

How it is used
Trains word embeddings from raw domain text by converting local word relationships into supervised prediction pairs.
How deep it goes
Explains prediction direction, window construction, vocabulary encoding, model layers, batching, and device selection, but does not detail optimization theory.
Examples
Uses WikiText2, lowercased token sequences, a ten-token window, vocabulary IDs, and 100-dimensional Word2Vec vectors.
What you can do after
Builds skip-gram embeddings from scratch with Python and PyTorch using tokenization, windowing, batching, and a Word2Vec model.

Read how Natural Language Processing in Action, Second Edition explains skip-gram →

Retrieval Augmented Generation, The Foundational Ideas

Skip-gram is a Word2Vec architecture that predicts surrounding context words from a target word.

How it is used
Learns dense word representations from raw text by predicting nearby context words.
How deep it goes
Explains the three-layer architecture and discarded output, but does not detail optimization or training accelerations.
Examples
Uses CBOW-versus-Skip-gram architecture diagrams and learned vector relationships as concrete illustrations.
What you can do after
Understand how Skip-gram converts context-prediction training into reusable word embeddings.

Read how Retrieval Augmented Generation, The Foundational Ideas explains skip-gram →

Also covered in

What to know first

Related concepts

Questions about skip-gram

What is skip-gram?

Skip-gram is a word-embedding training method that learns representations by predicting context items from a target item.

Which Manning books explain skip-gram?

Natural Language Processing in Action, Second Edition and Retrieval Augmented Generation, The Foundational Ideas.

What should I know before learning skip-gram?

Manning books that teach skip-gram usually introduce accuracy first.

What is the difference between skip-gram and Continuous Bag of Words?

Skip-gram is a word-embedding training method that learns representations by predicting context items from a target item. Continuous Bag of Words is a Word2Vec training architecture that predicts a target word from its surrounding context words.

What is skip-gram part of?

Skip-gram is part of skip-bigram. A skip-bigram is an ordered pair of words separated by zero or more intervening words in a sequence.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.