liveBook · Manning

Skip-gram

As explained in Retrieval Augmented Generation, The Foundational Ideas

Skip-gram is a Word2Vec architecture that predicts surrounding context words from a target word.

Skip-gram is a Word2Vec architecture that learns dense word embeddings by predicting surrounding context words from a target word. It operationalizes the distributional hypothesis: words used in similar contexts tend to receive nearby representations. section 2.1.2

Its shared network follows a compact pipeline:

  • Receives a vocabulary-wide one-hot vector for the target word. section 2.1.2
  • Maps the target through a typically 100–300-dimensional projection with no activation function; one row per word becomes the learned embedding. section 2.1.2
  • Scores every vocabulary word at the output layer for its likelihood of appearing near the target, then discards that prediction layer. section 2.1.2
Figure 2.1 CBOW vs skip-gram architectures.

Compare Skip-gram’s target-to-context direction with CBOW’s context-to-target direction. section 2.1.2

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains the three-layer architecture and discarded output, but does not detail optimization or training accelerations.
Examples
Uses CBOW-versus-Skip-gram architecture diagrams and learned vector relationships as concrete illustrations.
What it assumes
Assumes vocabulary, context, raw text, and basic neural-network layers while introducing their embedding-learning roles.
What you can do after
Understand how Skip-gram converts context-prediction training into reusable word embeddings.
How it connects
Connects distributional hypothesis, word embeddings, CBOW, self-supervised learning, and semantic similarity.

Also explained in

Compare how 2 books explain skip-gram →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.